In order for a virtualisation platform to be suitable for running Softdial Contact Center™ the following points must be addressed:
- The hypervisor must support dedicated partitioning of processor, memory and disk resources to the guest machines. In practice this means Intel Nehalem architecture processors (current generation Intel Xeon, core i7 or better) and VMWare ESX/ESXi or Microsoft Hyper-V.
- All Sytel applications are IO intensive for communication and logging. This means that the hypervisor host machine needs fast disk technology. SSD is ideal but 15K SAS drives are an acceptable alternative. It also means that Gigabit networking is mandatory for all installations and 10GB networking for 500 agents or more.
- The network adaptor MUST be configured with a static MAC address otherwise licensing will fail.
Hypervisor Configuration for Softdial Contact Center™
Many companies deploying Softdial Contact Center™ do so on a virtualised architecture. Virtualisation enables multiple VMs to contend access to memory and processor resources in order to maximise use of those resources. However, this contradicts the requirements for real-time media streaming which mandates deterministic access to processing resources. Without this determinism, stream processing and delivery can get interrupted which can cause audio quality issues and occasional mis-signalling of calls.
Below we detail the issues with deploying real-time media streaming and ACD in a virtualisation environment, and define solutions for common hypervisors. RCF2119 conventions apply.
General Hypervisor Configuration Rules
Whatever hypervisor vendor tools that are available for integration and performance of the guest OS must be installed (eg VMWare tools).
See also the VMWare Configuration Checklist below.
High-performance virtual network interfaces capable of supporting QoS are required. A Softdial Telephony Gateway™ VM implemented in a cluster and moving recordings to a SAN will require 140Mbit/s sustained traffic (and peak load around 500Mbit/s). This traffic is mixed QoS with voice taking precedence over network file system.
Resource Allocation
In order to support real-time streaming behaviour, Softdial Telephony Gateway™ servers must have dedicated processor and memory resources allocated to them. If the hypervisor is configured to contend access to resources, this may interrupt stream processing to allocate more resources to the Softdial Telephony Gateway™ VM as required, resulting in packet loss leading to audio breakup and occasional mis-signalling. In addition, switching of hypervisor resources can lead to interruption in stream processing.
To prevent resource allocation issues, a Softdial Telephony Gateway™ VM capable of supporting 200 predictive agents / 600 voice calls / full call recording and G.729 encoding will require configuration as follows:
VMWare ESX/VSphere
- 4 Virtual CPUs
- 7500MHz CPU reservation
- 2.5GB memory reservation
- Processor affinity set to use 4 specific logical CPUs for which no other VM has a specific processor affinity
The VMWare CPU reservation figure is based on worst possible load. This reservation may be reduced if load monitoring indicates that the VM is under-used.
Hyper-V
- 4 virtual CPUs at 100% allocation
- Set processor affinity
- Reserve 2.5GB of memory.
VM Relocation
In a large-scale virtualisation infrastructure with multiple hosts, virtual machines may be relocated across hosts for load-balancing. This relocation requires the temporary suspension of a guest OS while it is relocated onto another host.
Relocating a Softdial Telephony Gateway™ VM between hosts takes a finite amount of time which can result in both voice quality and mis-signalling issues. For this reason hypervisors must be configured to pin the Softdial Telephony Gateway™ VM to specific resources. This is achieved as follows:
VMWare ESX/VSphere
Set processor affinity on VMs that should not move. This will prevent vMotion from relocating the machine. This should be done for the Controller (Softdial CallGem™) VM and must be done for Softdial Telephony Gateway™ VMs
Hyper-V
Failover clustering and live migration must not be used.
Organisation of Pinned Resources
Since Softdial Telephony Gateway™ servers must be pinned to a particular host and Controller Servers should be pinned, pinned resources must be organised in such a way that failure of a host will not result in a permanent failure of a component of Softdial Contact Center™.
This means:
- STG instances must be distributed evenly across physical hosts. For example, if you have 3 Softdial Telephony Gateway™ VMs and 3 hosts, each host must have 1 Softdial Telephony Gateway™ VM pinned to it.
- Where more than one Softdial Telephony Gateway™ VMs is pinned to a host, processor affinity must not conflict.
- Primary and backup Controller instances, if pinned, must be pinned to separate hosts. This applies to any primary and backup pair in a replicated configuration.
VMWare Configuration Checklist
Setting up VMs to run realtime applications. RFC2119 conventions apply
Processor Setup
- The number of CPU cores set should be sufficient for all server activities.
- An application server should configure a minimum of 8 cores (remembering that virtual cores are doubled for Intel processors with hyperthreading)
- CPU reservation must be the maximum for the number of cores. If an 8-core VM is running on a 32-core server advertising 24,000 MHz, the reservation should be set at 6,000 MHz. You MAY set a reservation less than this but may not set the reservation at less than two-thirds of maximum capacity. This ensures that the hypervisor will not pause the VM to reallocate resources.
- Scheduling affinity (commonly known as pinning) must set a contiguous range of processors to use, within the same processor die, unless of course a massive VM spanning multiple processors is used. An 8-core VM on a 32-core machine might use processors 0-7, 8-15, 16-23 or 24-31 for example.
- The person administering the hypervisor must keep a pinning map to ensure that no VM configuration attempts to allocate resources to the same processor.
- The shares value should be set to the maximum. If you are running on a machine that supports hyperthreading you should set the Hyperthreaded core sharing option to Internal
Memory Setup
- You must disable the swap file on the guest OS and make sure that the guest OS is configured with sufficient memory for its workload.
- All guest memory configured must be reserved.
Network Setup
- As a matter of good practice you should set up a separate VM network for servers within SCC, and associate this with either a physical network adapter or a port trunk if you have driver-level support for the multi-port NIC on your host machine.
For details of the licensing options available for virtualised installations, see Licensing Deployment Issues
Virtualised Server Specifications
When referring to cores, this is based on a current generation Xeon or i7 based server clocked at 2.4GHz.
Tenant-per-VM
Having a tenant configured per VM ensures that the processing resources that a tenant uses will not impact on another tenant's use of the system. The benefits of this configuration are offset by the additional cost of multiple OSs.
Tenant-side services to be deployed are:
- Softdial Campaign Manager™
- Softdial Scripter™
- Publisher
- Scheduler
Let us also assume database services are deployed on a VM on the same server. In this case you should follow the advice below:
- Remove the swap file - it is counter productive to have a swap file on a database server.
- Configure the SQL server so that it does not reserve all the available memory. As a guide, on an 8GB server a reservation of 5GB will give enough of a safety margin to allow the rest of the OS to function smoothly.
The resources required to run the tenant services depend on the number of agents.
We suggest:
| No. of Agents | 50 | 100 | 200 |
|---|---|---|---|
| CPU cores | 4 | 6 | 12 |
| Memory | 1GB | 2GB | 4GB |
| Disk | 50GB | 110GB | 200GB |
These figures are a guide based on assumptions we've made about size of customer database.
For running large scale virtualised tenant services, we recommend that the database should be moved to a separate physical server and that the host should be a Dual Quad Core Xeon server running Windows Server 2008 R2 with 16GB. This will enable up 800 agents to be supported for all tenant-side services but OS limits related to network load generated by Softdial Campaign Manager™, Publisher and Softdial Scripter™ dictate that above 250 agents the Softdial Campaign Manager™ and Softdial Scripter™ services should be moved to a separate physical server.
Rules for Hypervisor Setup:
- Configure Static MAC addresses for virtual network adapters.
- Configure 100% reservation of memory and CPU resource.
- Static disk space allocation for all virtual hard disks.
Rules for Guest OS Setup:
- Prior to installing SCC configure the OS as follows:
- Disable the swap file.
- Disable Windows Firewall
- Switch off UAC.
- Enable the application server role (if OS is Server 2008/R2)
- Download and install the FULL version of .NET 4.0. Our installer will detect but if you have only .NET 4 client profile installed this will lead to problems.
- Install Notepad++ so that log file analysis can be done easily when remoting in to the machine.
- Run Windows update and manually install all important and critical updates and service packs. Once you have completed this cycle configure Windows update not to perform any updates.
Hypervisor / SIP Server Compatibility Issues
A third party SIP Server will generally be required with the following configuration scenarios:
- If the user has both internal network SIP phone users and external users (eg homeworkers, hosted or CPE)
- If the user has multiple SIP service providers that have to be registered with to make calls.
- If the user is deploying the Softdial PBX, either in a hosted or CPE environment.
If the user's virtualisation infrastructure is based on Hyper-V, Linux and Hyper-V do not work well together. Users deploying Hyper-V virtualisation and requiring SIP registrar/proxy capabilities will need to deploy a Windows-based SIP server, such as Brekeke™
If the user's virtualisation infrastructure is based on VMWare, deploying Linux-based SIP servers, such as AsteriskNow™, is strongly recommended.
Why Not to Use a Hosted VM for Softdial Telephony Gateway™
The main performance bottleneck we see with running Softdial Telephony Gateway™ on virtual machines is caused by disk access limitations. This means that putting Softdial Telephony Gateway™ on a hosted server delivered by a third party will be problematic unless the hosting provider delivers VM instance storage by one of the following mechanisms:
- a stripe set of 15KRPM SAS drives
- a stripe set of fast current generation SSD drives.
Note that many hosting providers deliver fibre-channel based VM storage. This, or rather the underlying storage architecture on the SAN is unlikely to be fast enough to deliver the IO requirements for hundreds of simultaneous recording file writes.
In order to run Softdial Telephony Gateway™ successfully under load, the virtualisation environment must be able to reserve dedicated processor power to the VM. Hosted VMs do not as a rule do this.
For these reasons we do not advise hosting the Softdial Telephony Gateway™ server on a 3rd party VM server.