At its simplest, Multi-site Resilience (MSR) is the ability to have multiple servers work together to provide an automatic fail-over capability so that, should one server fail, the remaining servers take over and all existing encoders/viewing clients continue to operate without user intervention.

Multiple servers join together to create a Server Cluster.

NOTE: The maximum number of servers in a cluster is FOUR. If you require a larger deployment, more sophisticated (but complex) solutions are available - please contact Digital Barriers for further details.

Each machine in the cluster automatically synchronises accounts and rules and provides a connection point for encoders and viewing clients to connect to. On initial connection to any of the servers an encoder/viewer will then be supplied the address of all available servers to ensure the latest cluster information is shared:

When a viewing client asks for a video stream, it is seamlessly directed to the appropriate server that hosts that particular encoder – this can mean that viewers are simultaneously connected to multiple servers within the cluster at the same time.

In the event of an issue with one of the servers within the server cluster, the encoder already knows the location of the remaining servers and will automatically connect to another server. This failover can take up to 10 seconds to occur, during which time viewers will not receive video for encoders connected to the lost server.

Viewers will also automatically connect to another server within the cluster, and will learn the new location of all encoders from the lost server. This allows video playback to resume automatically, without intervention of the user.

To avoid overloading any one server in the server cluster, the cluster will automatically move encoders from one server to another to roughly create an equal number of encoders across each server. All of this happens transparently, and encoders/viewers will be directed to the correct location automatically. Encoders are only moved when they are not in use to avoid any user disruption.

After a lost server returns to the server cluster, the cluster will load balance (after a period of stability) to ensure load is again evenly spread across all servers.

It is possible to override the server’s default balancing behaviour by using the Server Pool/Preference capability. This will force the encoders to ignore auto load-balancing and default to a supplied server (or collection of servers).

When deciding the bandwidth and processing power required for each server within the cluster, it is important to calculate based on worst case situations. For example, should one server go offline in a two-server cluster, then all encoders/viewers will be hosted on one server, which must be able to handle the load and bandwidth throughput of all attached encoders.

MSR is designed to:

There are many different forms of redundancy and failover. MSR does not:

Multi-site resilience (MSR) is an enterprise level feature that requires careful planning. As part of the preparation for enabling Server Redundancy please review the following checklist to determine if there are any additional steps that should be performed, and whether your current encoder/viewer installations are compatible with MSR:

The following sections outline each of the items in more detail…

Only if each item has been checked (and any required actions de-risked and planned) should you proceed onto the installation steps in the next chapter.

Failure to prepare for MSR can create a situation where encoders will attempt to connect to the wrong backup server, and older viewing clients will no longer be able to connect.

The following hardware specifications will support up to 500 encoders and 500 video streams simultaneously. Users looking to support more than 500 encoders should consider utilising multiple servers – please refer to the Installation Steps section for instructions for installing a multi-server installation using MSR.

All installation types of EdgeVis Server require:

Running EdgeVis Server in a multiple server configuration requires the following:

Running in a virtual machine/cloud environment is supported, however it should be noted that it is not uncommon for virtual machines to share resources between other users and performance can suffer in an unpredictable manner.

To counter this, it is recommend to over-specify any virtual machine. For example, Amazon users should consider a m5.xlarge instance for systems running > 20 encoders.

Linux users

This article primarily deals with installation of MSR on Microsoft Windows, however Linux installations can also utilise MSR. To enable MSR on Linux follow these instructions, and when instructed to install EdgeVis Server, refer to the document EdgeVis Server v8.X - Linux Installation and Upgrading for Linux guidance.

Each installation of EdgeVis Server contains a database that holds all accounts, permissions, alarm rules and events. One database will, during normal operation, be designated the master and will service requests from every server within the cluster. Each other database within the cluster will contain a synchronised copy of the all data, to allow another to take over if required. To enable this synchronisation the following requirements MUST be met:

Where the servers exist in different locations it may be necessary to create a VPN connection between sites to allow all servers to communicate with each other.

All existing encoder firmware are compatible with servers using multi-site resilience (MSR). However older firmware will work in a compatibility mode, the behaviour of which will depend on the configuration of the encoder.

Review the table below and decide if it would be prudent to upgrade all encoders to 7.1+ firmware IN ADVANCE of upgrading the server.

While it is recommended to upgrade encoder firmware prior to enabling MSR, upgrading encoder firmware post server installation is also possible. Encoders upgraded to 7.1+ will automatically learn the list of servers within the cluster and reconnect using the PRIMARY server.

Mobile Encoders

Older Mobile Encoders do not support the PRIMARY/BACKUP scheme and will only connect to the server address provided. Users should be encouraged to upgrade to the latest version available within the respective app store to enable MSR compatibility once the server upgrade is complete.

Older viewing clients will not function properly on a server with multi-site resilience (MSR) enabled – they do not understand that encoders may be moved between servers, and users would incorrectly be told encoders are off-line.

There are three categories that a viewing client will fall into:

Fully-Compatible Clients

Partially-Compatible Clients

By default, the server will treat these versions as incompatible. However, it is possible to enable a compatibility mode on the server that will allow these versions to work in a limited compatibility mode – please refer to Appendix A for further details.

Not-Compatible Clients

These clients are incompatible with MSR and will not be updated to make them work. Customers must either upgrade to a compatible client or contact their VMS provider to discuss their upgrade options.

What will happen if an incompatible client connects to a server with MSR enabled?

EdgeVis Server will detect incompatible clients and provide a warning when the user attempts to view a list of available video streams. For example, users of TVI Viewer would observe the following:

The installation steps are slightly different for new installations, or for customers wishing to upgrade an existing server installation – Step 1 is different for both situations. Please follow the checklists below as appropriate:

Each of the steps is outlined in the following sections.

Each of the steps is outlined in the following sections.

Servers that were originally installed using EdgeVis Server 7.0.X and below need to convert their databases to operate in MSR mode.

Refer to the EdgeVis Server Setup Guide and follow the instructions in the earlier section Installing EdgeVis Server on Microsoft Windows to install EdgeVis Server. Linux installation instructions are also available on the support site.

Normally the next step would be to obtain a licence from Digital Barriers for the server – however in this instance it is possible to wait until all servers are set up and working before requesting the licence. Any encoders that require firmware upgrading in the next step can be upgraded using USB flash drives if necessary.

While reviewing the readiness checklist some of the actions required creating lists of upgrades required:

At this stage, the first server in the cluster is still running in single server mode and so all encoders/clients should operate as normal regardless of firmware/version. Take this opportunity to upgrade all of the items in the above checklists, as the upgraded components should continue to operate as before, while minimising down-time/issues during later steps.

By default, an EdgeVis Server runs in stand-alone mode and must be converted to MSR mode to allow the creation (and licensing) of a server cluster.

From the server homepage click the Advanced server settings link at the bottom of the page. Then click the Multi-site resilience option.

Click the Enable multi-site resilience button on the right.

Once the operation is complete, this EdgeVis Server will still be operational and is, in effect, a server cluster containing just one server. Any viewing clients that are in the ‘Not-compatible’ category will no longer be able to connect at this point.

Run the EdgeVis Server installer on any other PC that will join the server cluster. Follow the standard installation steps until the setup wizard starts.

Logging onto the web interface of the second server, at this stage, will present a message that should confirm the server has joined the server cluster, and is ready to be licensed.

To check the installation of the second server, and clustering support is enabled return to the web interface of the first server and refresh the Multi-site resilience settings page. This should now show the new server as a member of the cluster (with a warning that it is unlicensed).

With both servers now connected the final stage is to request a licence for the servers within the cluster.

If unfamiliar with the process of how to request a server licence, follow the Installing licences on EdgeVis Server section from the EdgeVis Server Setup Guide to request a licence, as you would for a single standalone server. It is only necessary to make one request – the hardware signature for every PC in the cluster is included in the request.

Once installed the licence is automatically transferred to every PC in the cluster, and it will now be possible to log in to the web interface on each PC in the cluster.

For both encoders and viewing clients, end-users should be unaware of the complexity of Multi-site Resilience, and need only supply the address of any of the PCs in the server cluster.

Provide the encoder the external address (either IP Address or DNS name) of ANY of the PCs in the server cluster.

All of the above happens behind the scenes and the end-user should only notice a short period of disruption while the encoders/clients switch over to another PC in the cluster.

The choice of which server the encoder joins is decided when the encoder connects to the server (and will normally be the lightest-loaded server). When the encoder is connected, but not streaming, the server may choose to move an encoder to another server as this will balance the load between servers.

Some encoders include a Backup Server address field – this is ignored on servers with redundancy enabled. Encoders upgraded to V7.1 with this setting configured will switch to using the addresses supplied by the cluster.

Provide the client the external address (either IP Address or DNS name) of ANY of the PCs in the server cluster.

All of the above happens behind the scenes and the end-user should only notice a short period of disruption while the encoders/clients switch over to another PC in the cluster.

Administrators are free to log in to any of the servers, using the standard web interface running on port 9443. Each server retrieves the account information from the same database, providing a consistent set of domains/accounts/alarms regardless of which PC in the cluster they are logged into.

To view the current status of a server cluster click the Advanced server settings link at the bottom of the server’s homepage. Then click the Multi-site resilience option. Clicking on each server in the cluster will allow the user to view which encoders/viewers are connected to each respective server .

In the event of a server failure an administrator attempting to log in to the web interface must manually enter the IP address of another PC in the cluster within their web browser.
​

By default, EdgeVis Server auto load-balances all encoders across all servers within the MSR infrastructure to attempt to share load (and risk). For many scenarios this behaviour is preferred - especially when all servers, their reliability, and their internet connections are roughly equal.

However, there are scenarios where finer grained control of an encoder’s server location is desired:

EdgeVis Server allows the user to create server pools, and then set a preferred server pool on an encoder – either on a server-wide or domain-wide level.

Note: Any encoder that does not have a server preference set will continue to be auto load-balanced. If an encoder is assigned to a server pool, it will fail over between all servers within the pool. Should none of those servers be available it will then attempt to utilise any remaining server in the MSR infrastructure.

There are four steps (on an existing MSR infrastructure) to make the best use of server pools:

As a System Administrator (or a user with Server pool management permission), from the Server Home Page select Advanced Server Settings -> Multi-site resilience -> Manage Server Pools – this will present a list of existing server pools.

On a new server this list will be empty – select Add pool to create a server pool. Enter a name (e.g. ‘Primary-Servers’ or ‘West-Coast’) that will provide users the context of which servers are in the pool and why.

In a primary/backup scenario it is only required to create one server pool, as this is the pool that will be used to hold the primary servers. For other scenarios, create each pool as required for each geographical/organisational structure.

From the Server Home Page select Advanced Server Settings -> Multi-site resilience – this will present a list of servers within the multi-server MSR cluster.

Click on each desired server and select Edit this server’s pool(s). This will present a list of available server pools to select from. In complex scenarios it is possible to select multiple server pools for a server, however it is normally recommended to only add a server to one pool.

Note: This step is optional – if it is not performed the standard behaviour for encoders (unless overridden by Step 4) will be to auto load-balance all encoders across all servers in the MSR cluster.

From the Server Home Page select Advanced Server Settings -> Encoder settings -> Server preference.

This presents two options:

It is possible to lock either of these settings down using the padlock icon – this will block users from performing Step 4.

​

Note: This step is optional – if it is not performed the standard behaviour for encoders (unless overridden by Step 4) is to use the settings (if any) applied in Step 3.

It is possible to set both settings configured in Step 3 on an individual domain basis – this is the primary method of segregating encoders on a geographical/organisational basis. Create Domains to contain encoders that should be grouped into the same server pool.

From the Domain Home Page select Encoder settings -> Server preference. This presents the same options as Step 3. By default, each setting will be configured to Inherit server default but by changing this to Domain specific setting it is possible to provide a domain override to the default behaviour.

For both settings either select the desired pool/setting or allow the domain to use the inherited setting from Step 3.

The server’s web interface will give you feedback on what the current state of the system is. If the system is in anyway degraded, you will see an alert highlighted on the ‘About this server’ page. If you are unable to access the web interface on one of your servers, then try accessing it using the other server(s) in the cluster.

Clicking on the alert link will take you to the ‘Multi-Site Resilience’ page where you will be able to get more information about what the issue is.

On the MSR page you will see an indication of the status of each server in the cluster. The top section is the status of the video server components and the bottom section is the status of the database server components.

Green indicates that the component is online and working.

Grey indicates that the component is online, but in a degraded state. For example, a database will be presented like this when it is syncing data from the cluster after a restart.

Red indicates that the component is offline.

Clicking on a component will give additional information about it. For video components this includes a list of the encoders and viewers attached to it. You can also change the name of the component through this page to make it more relevant to your deployment.
​

There may be situations where having a particular server active in a deployment may cause issues for the deployment as a whole. For example, a particular site has an issue that degrades the bandwidth available to the rest of the system but doesn’t disconnect the server. In this situation any encoders or clients that connect to that site may struggle to stay connected or transmit video.

It is possible to manually ‘fence’ a given site to prevent it from participating in the deployment. This can be done from the MSR page without physical access to the site to be ‘fenced’. This allows temporarily disabling of this site while remedial work takes place.

Both EdgeVis Servers and their database can be independently fenced – to remove a site from the cluster it is recommended that both the server and database be fenced.

Fencing a site

Any servers/databases that have been fenced will now report as FENCED on the main MSR status page.

Unfence a site

After re-joining the cluster, the servers/databases will now report as ONLINE on the main MSR status page.

Both video and database components for a server are marked as offline

This indicates that the other server is completely disconnected from the cluster.

This state is normal during restart of one of the servers and will resolve itself once the server returns.

If the state doesn’t return to normal then check the following:

Once the issue has been resolved, the offline server will reconnect to the cluster and its database will synchronise with the other server. After the synchronisation is complete, the cluster will be in a healthy state again.

Video component is marked as offline

This indicates that while both servers are running, the video components of the cluster have disconnected from each other.

Normally, this should resolve itself after no more than 15 minutes.

If the servers remain in this state, then check the following:

Database is marked as syncing

This occurs when a server is recovering from a failure.

When the server comes back online, it may need to re-synchronise the database with the other server before it is fully healthy and redundant again.

In some cases this may require a full copy of the database. In these cases the server may report this state for some time depending on the speed of the link between the sites.

If this state doesn’t resolve itself, check available bandwidth on the link between the sites as it may not be fast enough to complete the synchronisation.

This mode allows clients listed in the Partially-Compatible Clients section to connect to the server and view streams.

In order for the system to work with these clients, any clients that are behind the same firewall as the server must be able to connect the server’s Remote IP in order to view streams correctly. You can verify this by opening a Web browser on a PC (that would be running a client) entering the address to connect to the server’s configuration interface:

If the page displays correctly then the partially compatible clients will work.

If the test does not work then either the firewall does not support this mode of operation or requires additional configuration (for which you should consult the documentation for your specific make and model of the device).

If this mode is enabled and the firewall is not capable of or configured to allow this type of access, then these clients will still connect to the server and list all video streams - however, when opening a stream, they will not receive any video.

Note: If you have Multi-site Resilience enabled (or are planning on enabling it) these steps need to be repeated on each server in the cluster and when adding new servers.

One of the requirements for MSR is that the server publishes its external address to the encoders and clients that connect to the system. From version 7.2 onwards it is possible to have multiple MSR servers with the same external IP address, allowing multiple servers to co-exist on the same internet connection. This appendix covers how to configure this.

In addition to the standard MSR requirement it is also necessary to have:

By default two servers at the same site would attempt to use the network ports and would not be able offer the same external IP address through the same router/firewall. To allow the servers to be accessible externally via the same IP address the router/firewall must forward a different set of ports for each server.

The standard set of ports for video access is as follows:

It is possible to reconfigure these ports on subsequent servers using the same external address.

For example:

Server 1 (default ports)

TCP - 9300 & 9301

UDP - 2048 & 9300

Server 2 (reconfigured ports)

TCP - 9400 & 9401

UDP - 2148 & 9400

Deploy the cluster as described in the installation steps in the main document. When prompted, enter the same external address for each server behind the same router/firewall.

Each server requires a set of unique ports, i.e. there must be no other EdgeVis Server (or other device) behind the same external address using it. A cluster must have at least one server using the default ports.

NOTE: The server listening on the default port must be active when connecting new devices and new clients.

Configure the router/firewall to ‘port forward’ each set of ports to the corresponding EdgeVis Server’s internal address.

NOTE: Refer to the manufacturer’s router/firewall documentation for further information if necessary.

Troubleshooting:

​