Introduction
At its simplest, creating a server cluster provides the ability to have multiple servers work together to provide an automatic fail-over capability so that, should one server fail, the remaining servers take over and all existing encoders/viewing clients continue to operate without user intervention.
Within EdgeVis Server this is known as Multi-site Resilience (MSR).
How does it work?
Multiple servers join together to create a Server Cluster.
NOTE: The maximum number of servers in a cluster is FOUR. Larger deployments are possible - please raise a ticket with support to request further information.
Each machine in the cluster automatically synchronises accounts and rules and provides a connection point for encoders and viewing clients to connect to. On initial connection to any of the servers an encoder/viewer will then be supplied the address of all available servers to ensure the latest cluster information is shared:
When a viewing client asks for a video stream, it is seamlessly directed to the appropriate server that hosts that particular encoder – this can mean that viewers are simultaneously connected to multiple servers within the cluster at the same time.
In the event of an issue with one of the servers within the server cluster, the encoder already knows the location of the remaining servers and will automatically connect to another server. This failover can take up to 10 seconds to occur, during which time viewers will not receive video for encoders connected to the lost server.
Viewers will also automatically connect to another server within the cluster, and will learn the new location of all encoders from the lost server. This allows video playback to resume automatically, without intervention of the user.
Auto load-balancing
To avoid overloading any one server in the server cluster, the cluster will automatically move encoders from one server to another to roughly create an equal number of encoders across each server. All of this happens transparently, and encoders/viewers will be directed to the correct location automatically. Encoders are only moved when they are not in use to avoid any user disruption.
After a lost server returns to the server cluster, the cluster will load balance (after a period of stability) to ensure load is again evenly spread across all servers.
It is possible to override the server’s default balancing behaviour by using the Server Pool/Preference capability. This will force the encoders to ignore auto load-balancing and default to a supplied server (or collection of servers).
What problems does a server cluster not solve?
There are many different forms of redundancy and failover. Server clustering does not:
Provide a hot-failover capability
All servers in a server cluster are online and operational at all times.
Allow routing between servers
Video is always sent from the server that hosts the encoder – there is no server-to-server routing of video traffic.
Dynamic scaling of server clusters
All servers are expected to be online for normal operation. Servers should be ‘offline’ from the cluster for as little time as possible, and assume that any changes made to accounts on an ‘offline’ server could be potentially lost.
System Requirements
There is a separate article that outlines a sizing guide and recommendations on the system resources a server may require, which includes guidance on how to scale up a server cluster.
However, there are additional requirements you must consider:
Your cluster's internet connection must support the combined bandwidth needs of the encoders and viewers across all servers.
An accurate synchronised system clock across all machines in your cluster (Windows will do this automatically; Linux may require an NTP client)
Each of the machines in the cluster must be accessible to the other machines without using port forwarding.
For example, they must all be on the same local network or on separate networks connected using a VPN.Specifically, these ports must be accessible on each server machine (from every other server machine):
Port 1527 (TCP)
Port 47100 (TCP)
Port 47500 (TCP)
Port 9443 (TCP)
The connections between all the server PCs must meet these requirements:
Maximum 100ms round trip time
Minimum 10mbps bandwidth, 50mbps recommended
Slower links will result in a longer recovery period after a failure.
Multiple servers may share one internet connection with port-remapping.
Steps to create a server cluster
You can create a server cluster at any point, either while installing a clean new server, or by upgrading an existing standalone server that already has encoders and viewers.
Step 1: Check your setup meets the system requirements
There are a number of additional network requirements listed above you should check on your machines before proceeding:
Every machine gas an accurate and synchronised clock
Your machines can all see each other (without port forwarding)
The listed ports are all accessible between machines
The link between the machines satisfies the bandwidth and round-trip criteria
Additionally, these instructions assume you are using EdgeVis Server version 8.0 or above. If not, you should upgrade your server, encoder, and clients to the latest release. Additionally, Milestone/Genetec users should be using a release no older than two years old.
Contact support for guidance if your infrastructure predates version 8.0.
Step 2: Enable ‘Multi-Site Resilience’ mode
If this is a new installation, you should first install your server instance using the standard installation instructions for a Standard Server. This will allow you to enable the options for MSR.
Log into the web interface of your new or existing EdgeVis Server.
From the server homepage click the All server settings link at the bottom of the page. Then click the Multi-site resilience option.
Click the Enable multi-site resilience button on the right.
Once the operation is complete, this EdgeVis Server will still be operational and is, in effect, a server cluster containing just one server.
For long-running servers, you may receive a warning message that you can not upgrade your server due to legacy issues. Please contact support for instructions on how to resolve.
Step 3: Install EdgeVis Server on the remaining server machines
The process of joining additional servers to your new cluster is performed during the installation of EdgeVis Server on to your additional machines.
Once installation is complete on any machine in the cluster, you should be able to log into the web interface - select the All server settings link at the bottom of the page. Then select the Multi-site resilience option. This should now show that your new machine has joined the cluster:
Step 4: License your new server cluster
With all servers now connected the final stage is to request a licence for the servers within the cluster.
If unfamiliar with the process of how to request a server licence, follow the Installing licences on EdgeVis Server section from the EdgeVis Server Setup Guide to request a licence, as you would for a single standalone server. It is only necessary to make one request – the hardware signature for every PC in the cluster is included in the request.
Once installed the licence is automatically transferred to every PC in the cluster, and it will now be possible to log in to the web interface on each PC in the cluster.
How does server clustering affect users?
For both encoders and viewing clients, users should be unaware of the complexity of Multi-site Resilience and need only supply the address of any of the PCs in the server cluster.
Encoders
When configuring the encoder use the external address (either IP Address or DNS name) of ANY of the PCs in the server cluster.
Encoders will be automatically provided the address of all servers within the cluster when they first connect
Encoders will be automatically directed to the correct server to aid with load-balancing
Encoders will attempt to connect to a different server within the cluster should its original server fail
All of the above happens behind the scenes and the end-user should only notice a short period of disruption while the encoders/clients switch over to another PC in the cluster.
The choice of which server the encoder joins is decided when the encoder connects to the server (and will normally be the lightest-loaded server). When the encoder is connected, but not streaming, the server may choose to move an encoder to another server as this will balance the load between servers.
Viewing clients / VMS Gateway / Third-party VMS
Provide the user the external address (either IP Address or DNS name) of ANY of the PCs in the server cluster.
Clients will be automatically provided the address of all servers within the cluster when they first connect.
When viewing an encoder, clients will be automatically directed to the correct server hosting that encoder.
Clients will attempt to connect to a different server within the cluster should its original server fail.
All of the above happens behind the scenes and the end-user should only notice a short period of disruption while the encoders/clients switch over to another PC in the cluster.
Server Web Management interface
Administrators are free to log in to any of the servers, using the standard web interface running on port 9443. Each server retrieves the account information from the same database, providing a consistent set of domains/accounts/alarms regardless of which PC in the cluster they are logged into.
To view the current status of a server cluster click the All server settings link at the bottom of the server’s homepage. Then click the Multi-site resilience option. Clicking on each server in the cluster will allow the user to view which encoders/viewers are connected to each respective server.
In the event of a server failure, an administrator attempting to log in to the web interface must manually enter the IP address of another PC in the cluster within their web browser.
Next steps...