Home » Anywhere365
Category Archives: Anywhere365
For the last 6 months I have been delivering a number of global contact centers based on Skype for Business and Anywhere365. The journey has been long and at some points painful, but we have got there and everyone is happy! Along the journey I have learnt a lot, cried a lot and laughed a lot! Now I am going to share with you some information that will help you decide if your Skype for Business deployment can cope with Anywhere365.
Firstly, Anywhere365 is one of only a few native Skype for Business contact centers. Do not get confused between this and contact center vendors who say they integrate with Skype for Business! To help you understand the meaning:
- NATIVE means that the application uses Skype for Business Server functionality and APIs such as UCMA to build the contact center software. This means that it will use SfB Server components and media to deliver the functionality as well as baked in support of the Skype for Business client application.
- INTEGRATED usually means that the contact center has an endpoint(s) registered in Skype for Business that takes the call away from the native components into their own system and deliver this back to the agent maybe to the Skype for Business client by itself, maybe a dedicated agent application, or the Skype client with the aid of a plugin.
There are positives and negatives for each, and this post is not about pitching one over the other.
As a result of being a native contact center, Anywhere365 of course leverages a lot of what Skype for Business has to offer in order to create those special contact center communication scenarios. To begin with Anywhere365 hooks into Skype for Business as a trusted application. As a trusted application, it requires trusted application endpoints. These endpoints act as the route in and out of the Anywhere 365 application and Skype for Business.
Another fundamental aspect is that Anywhere365 does not have its own media engine. In other words, it cannot control media paths, transcoding, media establishment etc. It is oblivious to that. Instead Anywhere365 leverages the conferencing service on the Skype for Business Front End Servers to connect the caller to the agent. This means that Anywhere365 is only part of the SIP signaling loop between caller and agent and as a result, requires less server resources that integrated contact centers.
So How Does a Conversation Work?
When a caller places a call to the contact center, they are calling the LineURI that is assigned to the trusted application endpoint attached to the contact center application. At this point all Skype for Business has done is routed the SIP signaling through to Anywhere365. Anywhere365 will accept the SIP INVITE sent to it and then send a signal back to Skype for Business to create a conference. Once the conference is created, Anywhere365 invites the caller to join that conference. The conference is hosted on the front end pool that the Anywhere365 server is paired to by the trusted application pool.
As soon as the caller is placed into the conference, Anwhere365 will play the welcome and queue messages to the caller within the conference. When you look at the logs, you will see that during these messages temporary SIP URIs join the conference to play the messages or hold music and remove themselves when no longer needed. Once the caller is in a position to be connected to an agent, Anywhere365 hunts an agent based on presence (and other parameters) and once an agent has been found they are invited to join the conference that the caller is in.
As you can see, conference performance is of paramount importance to Anywhere365. A well performing AVMCU will create and join the caller to the conference in about half a second when done programmatically. In layman’s terms this is approx. half an ring. If you call an Anywhere365 contact center you hear multiple rings before hearing the first message, or the call being accepted, then more than likely you have a problem with your conferencing service. Similarly if it takes a noticeably long time for messages to play then this could also be related to conferencing service performance issues.
Luckily there are a few things that you can do to eliminate other possibilities:
- Assign a conferencing policy to every endpoint you create for your contact center. This includes the main, hunting and other system endpoints you may create. Why? Well Anywhere will use a random endpoint to create the conference. If that endpoint does not have a conferencing policy, then the conference creation can fail, then another endpoint is tried etc. etc. resulting in delays.
- The conferencing policy must have the following parameters set; –AllowAnonymousUsersToDialOut $True, –AllowAnonymousParticipantsInMeetings $True, –AllowExternalUsersToRecordMeeting $True (if you want audio recording), –AllowNonEnterpriseVoiceUsersToDialOut $True. If these are not set in the conferencing policy, some random things will happen, usually resulting in performance issues of a varied kind.
- Assign each endpoint an appropriate dial plan and voice policy too if you want dial out to work via the contact center.
Before installing Anywhere365 and trying to use it, you need to spend some time understanding your environment and your current performance across all components and modalities. Luckily Microsoft have invested a lot of time and effort producing performance benchmarking tools such as Key Health Indicators and the Stress Test Tool (often ignored due to complexity and time). These should absolutely be run to establish a baseline of your current performance as underlying issues could be amplified as soon as contact center workloads are placed on the environment.
Next you need to work out whether your conferencing servers (front ends) can handle the expected number of contact center conferences above the normal number currently consumed. This will help you decide whether you need a separate front end pool for your contact centers or not.
Calculating Conference Availability
To calculate this accurately, you need access to the Skype for Business Monitoring and reporting database. Your aim here is to produce a report containing the number of conferences on the pool over the course of a normal working week. For Example,
Once you have the numbers, you can establish an average number of conferences for each pool per day. In my example above the averages look like this:
|Average Per Day||248||773||622|
From the report you also need to establish the average number of conference participants and also the average conference duration in minutes
|Average Number of Participants||5||15||5|
|Average Conference Duration (minutes)||31||35||31|
Now you have this base information, you can now start working out your concurrent conferences per pool. We can do this by first calculating the total average talk time per day, (AVERAGE DURATION x AVERAGE NUMBER OF CONFERENCES ON POOL) e.g. For UK that’s 35 x 773 = 27055 minutes.
Next, the working day for the UK is 8 hours, so we can deduct from the reporting that these conferences took part over the course of an 8 hour window. So we need to find out the concurrency. For that we need to find out how many conferences are taking place every second. This can be achieved by (AVERAGE TOTAL TALK TIME PER DAY / WORKING DAY) / 60 In the UK’s example that’s (27055/8) / 60 = 57 concurrent conferences.
Once this has been calculated we are closer to finding out the available capacity. The theoretical maximum number of conferences on a front end pool is calculated by the total number of conferencing users in a pool at any one time. The theoretical limit is 3,996 users in a 12-node pool, or 333 per front end server. But as not everyone has 12-node front end servers and conference consumption is hugely different between companies, the calculation between conferencing users and total number of conferences per pool can vary massively. For instance if your average participation in conferences is 3 users, then you have 111 conferences available per FE. However, if your average participation is 12, then you only have 27 conferences available per server!
In the UK example here, I have an average participation of 15 users per conference. The UK pool is a 3-node FE Pool. So using the logic applied we can calculate that each Front End has a capacity of 22 x 15 person conferences per server, resulting in a pool maximum of 66 x 15 person conferences
Now how are these conferences divided? We can calculate we have 57 simultaneous conferences going on, so (SIMULTANEOUS CONFERENCES / NUMBER OF SERVERS IN POOL) e.g. 57 / 3 = 19 conferences per server
To find the total number of conferencing users per pool you need to multiply the average number of participants by the number of simultaneous conferences taking place (AVERAGE NUMBER OF PARTICIPANTS x NUMBER OF SIMULTANEOUS CONFERENCES) e.g. 15 x 57 = 855 in the UK example
Next, you need to find calculate the distribution of these conference users over the number of servers in the FE pool (NUMBER OF CONFERENCING USERS / NUMBER OF FE SERVERS IN POOL) e.g. 855 / 3 = 285 in the UK example based on a 3-node FE pool. So the UK has 48 conferencing users available per server or 144 available conferencing users in the pool.
So the baseline result for Anywhere 365 conference availability for the UK is as follows:
- 144 x conferencing users available in the pool
- 6 x 15 person conferences available in the pool
Calculate the Predicted Anywhere365 Conference Consumption
In this example we are limited by the number of available conferences being 6. However, it is likely that contact center calls are only going to have 3 participants at any one time (caller, agent and system endpoint). So if we multiply the 6 available conferences by 15 (number of users) then we can see that we could have a further 90 x 1 person conferences instead of 6 x 15 person conferences. Divide 90 by 3 and we can subdivide those 6 x 15 person conferences into 30 x 3 person Anywhere365 available conferences.
So now we have a potential upper limit of 30 Anywhere365 conferences available for the UK pool. But we aren’t finished yet!
From the contact center design, you need to know the expected average call duration of inbound and outbound calls combined, for example 4 minutes. On top of that you need to calculate the average hold time and the time required to play messages to the user, because the conference begins on acceptance of the call, not when the agent begins to talk! So the calculation is (AVERAGE TALK TIME+AVERAGE WAIT TIME+AVERAGE SYSTEM TIME) e.g. for the UK pool we estimate in this example 4 minutes of actual talk time, 3 minutes of wait time and 2 minutes of messages (welcome, IVR, busy etc.) i.e. 4+3+2 = 9 minutes
We also need to know how many calls are expected per day across all contact centers deployed on the same Anywhere365 pool. In this example we will assume that there will be 300 calls per day across all UCCs.
So now we can do the same calculations to predict the Anywhere365 conference consumption.
1) Calculate average total conference duration over the course of the day = 300 calls per day * 9 minute duration = 2700 minutes total conference time
2) Calculate the total number of simultaneous Anywhere365 conferences at any time = (2700 / 8)/60 = 6 conferences per second
3) Calculate the total number of conferencing users per simultaneous conference = 6*3 = 18 (subtract 18 from 144 = 126 conferencing users left in the pool)
4) Subtract the number of expected conferences from the available Anywhere365 conferences = 30 – 6 = 24 remaining Anywhere365 conferences in the pool
From this we can summarise that the existing front end pool in the UK can support the expected conference workload without adding capacity or dedicating a conferencing pool to just Anywhere365 workloads. Keep the metric under the potential limit and you should be within optimum performance ranges.
I hope this helps you calculate your capacity when deploying Anywhere365.
Mark is an Independent Microsoft Teams Consultant with over 15 years experience in Microsoft Technology. Mark is the founder of Commsverse, a dedicated Microsoft Teams conference and former MVP. You can follow him on twitter @UnifiedVale
Over the last few months I have been delivering a project that uses Skype for Business and Anywhere 365 to deliver some complex routing scenarios for a customer. As this is my first real heavy weight deployment with Anywhere365 I wanted to share some battle notes.
I will state at this moment that I enjoy working with the product, you can do some really cool and cutting edge stuff with it. The notes are the bumps I found along the way and something you need to consider as you design and deploy. I do not mean for this post to be negative, but rather constructive.
The system requires Trusted Application Endpoints to route calls in and out of a UCC. Depending on your requirements this can range from 4 to 10’s.
- Each UCC requires at least 4 Trusted Application Endpoints (TAE) to start. 1 x Main Endpoint and 3 x Hunting Endpoints. The Main TAE is the one that you will be assigning a Line URI to and will be your main inbound and outbound endpoint from which you make and receive calls into the contact center. The hunting endpoints are used to hunt available agents and invite them into the call.
- All endpoints require a conferencing policy. This is because all calls are conferences, therefore the endpoints need permission to create a conference on the conferencing pool.
- If you want to also have an outbound contact center, you need an additional endpoint that is used for routing. This routing endpoint is also required for inbound agent interception (more on this later)
- Also if you want outbound calling, every endpoint requires a dial plan and voice policy. Calls will be made using these policiesand not the policies assigned to the agent’s Skype object. The endpoint is selected at random. The Caller ID is that of the main endpoint to the contact center or the Agent’s Skype number depending on the setting chosen. This means that hunting endpoints do not require LineURIs, but require EV to be enabled and dial plans, voice policies to be granted.
- Have multiple inbound lines? A modality endpoint is required for each one. The modality endpoint is like the main endpoint but instead can be used to either enter the same flow as the main endpoint, or be used for side insteps. A side instep is where you call a particular number to bypass the default call flow e.g. field service technician calls a service desk number to get through to a dedicated queue rather than having to listen to a bunch of IVR options.
- Want webchat? That’s another 4 x Chat Endpoints. 1 for the main chat endpoint, 3 for chat hunting endpoints. 3 chat hunts allow for up to 3 chat conversations per agent, if you dont add chat hunting endpoints, you can only have 1 chat per agent.
- If you use the WSP naming convention for these endpoints e.g.ucc_hunting00X@domain.com and you get to 100+ then use firstname.lastname@example.org keeping the double zero. This is because their failover script looks for 00 in the sip address to decide whether or not to hide them from the address list. However, if you use your own naming convention, this is also fine, just make WSP aware so they can alter their scripts accordingly.
- More of a Skype nuance this one, if you create multiple endpoints at once (and you will), be sure to check that you haven’t accidentally named two or more endpoints with the same SIP address. Due to the replication delay in Skype if you hit execute duplicates will be entered into the configuration and the unique sip address validation check is “skipped”. This means pain afterwards trying to figure out the duplicates and then deleting and re-creating them.
In order to use the Agent Extension Window, the Inflight Snapper or the Wallboard, you need to connect to a web service. The web service is hosted on all Anywhere365 servers, but is valid on the active server only. By default all scripts and deployment documentation provided by WSP do not clearly show what is required for HTTPS, only HTTP. If you have external agents then you’ll want this web service to be accessible over the internet. Therefore, HTTP is a bad idea, and HTTPS should be used. My recommendation is to use HTTPS both inside and outside of your organisation. Avoid using direct web connection to the Anywhere servers and use a reverse proxy instead, for both internal and external client ingress. This allows you greater flexibility for DR and does not require a URL update to the client software.
Use Split-DNS, or pin point to make your external FQDN of the web service accessible internally e.g. uccweb1.domain.com.
You can use a wildcard certificate, the application is IIS based so it doesn’t care so much for specific SAN entries. However, dedicated web SSL is always recommended. If using HTTPS the web.config file for each Anywhere365 web service will need to be updated to include a HTTPS binding, only HTTP is included by default. If you don’t do this, you will get a 500 Internal Server Error!
If you expose your web service over the internet to accommodate external agents, then there are additional IIS configurations you will want to consider to secure the Anywhere 365 web services. My recommendation is to configure two web sites in IIS. One for Internal and One for External. The internal site would require no authentication or NTLM, while the external website can be configured for Basic Auth to prompt the external agent to authenticate with the web site before being allowed access to the data via the Anywhere365 client side apps. Much like Skype’s web setup, we are taking additional steps to ensure secure access when external.
The interceptor is used to “intercept” incoming and outgoing calls of an agent. If I was the caller and I called the Agent’s Skype Line URI or a federated call over Skype, the conversation I had with that agent would not be recorded in the contact center. Therefore, I have circumnavigated process and jumped the queue. With the interceptor, Anywhere can prevent this from happening, but looking at each invite to an agent and if it contains audio in the SDP body, the invite will be intercepted and re routed to the contact center in which the agent is added.
There is a setting to ignore any incoming audio invites coming from internal domains, or perhaps you want to enable this for internal callers too.
The interceptor is needed for outbound calls too. If outbound calls require to be recorded (conversation and/or CDR) by the contact center then they need to be intercepted. The interception will ensure that the agent is placed into a conference first before placing the outbound call to the callee. The default experience to the callee is that of a normal call, but to the agent it looks like a conference, although, the agent will hear the dial tone played.
A side effect of having interception enabled means that you cannot use reason codes without the agent being a member of another UCC that is not enabled for interception (subscriber UCC). There is a script available for WSP to keep these two UCCs agent lists in sync so at least the admin effort is reduced for MACDs. I fed this back to WSP and they plan to address this in a future release, so hopefully by the time you are deploying, this is no longer an issue.
The interceptor is an MSPL script on each front end server. It has its own service that must be started. Sometimes it will fail to start if RTCSRV takes too long to start up, so make sure it is started or set to delayed start. As a result, if an agent requires interception, then they cannot be registered to an SBA. They must be registered to the FE pool that has the interceptor installed. Otherwise, the invite never makes it to the FE and the interceptor will not know about the conversation
Anywhere gets its configuration from SharePoint lists. Every 10 seconds the configuration is refreshed and cached on the Anywhere server. In delta syncs if you have edited an agent’s SIP address in the agent list, i.e. marital name change for instance, this is not picked up by the delta sync. It seems this is by design and the sync looks for line IDs greater than or missing in that of the last sync to bring down changes, this is why adds / removes work, but edits do not. This appears to only affect the agent list only. So therefore, you have two options. Option 1 – add the agents as a new agent and replace their skills assigned to their old SIP address, then remove their old identity. Or option 2 – restart the UCC with a clean cache to force a full sync. (There is a fix coming).
First Call Blues
After you have restarted or started the UCC for the first time, the first call can take a while to initiate. This is especially noticeable when forwarding between UCCs. Subsequent calls work fine. This is because there is a TLS handshake that needs to be established between the trusted application and the front end pool that it is registered to. This happens on first call, and subsequent calls use the same authentication token from the original handshake.
HA is Server HA, not Voice HA
When discussing High Availability it is important to remember that Anywhere does not perform HA in the same way as Enterprise Voice HA. If an Anywhere server goes down all in progress calls are terminated. No new calls can be received either until such time as the standby Anywhere server has started the UCC service and started the contact centers. This means that HA in Anywhere world is akin to HA in VMware when a host is lost for instance, and all virtual servers are started from a cold start on another host.
Active / Active Servers
It is possible to have two Anywhere servers running in a trusted application pool actively running uccs. However, they must not be running the same ucc application at the same time. If this happens, you will experience weird behaviours like agents not being placed into the correct call, callers hearing hold music as well as the agent speak, caller ID not working correctly and a whole raft of other weirdness.
The Anywhere application is a techie product. What I mean by that is it requires some significant hands on to setup all the advanced features like, automatic HA, DR, Extension Windows etc. When you deploy it, it’s not like a traditional MSI, input a few parameters and away you go, the whole product is deployed and now configurable in an Admin Control Panel, or PS Module. It requires config file manipulations and the deployment of supporting scripts as scheduled tasks to glue some of it together to make it work. Once it’s in it works and works reliably, my intention here is not to tar the product but to ensure it is understood. I do feel that the install process could be worked on to deploy all features and there is definite scope to use SharePoint lists to configure some of the server back end configuration elements in a pure System Admin Portal. This would make the product seem less custom developed app and more enterprise product.
DR is a Dream
I have to say, their DR process and script is probably the best I have seen. In order to get DR to work there are several scheduled scripts that run daily to ensure DR pain is minimal. Each day the config and applications are backed up to the DR standby server. Applications are synched with the DR pool so they exist in both production and DR pools. In a DR event, all that is then required is to execute the failover script to move the trusted application endpoints from the failed pool to the DR pool, update the CDR DB connection string and start the UCC Service on the DR pool. This means that you can perform a DR of Anywhere in less than 30 mins of initial failure.
CDR Database Resiliency and HA
Anywhere relies on the CDR database for historic records and last agent routing. It is recommended therefore to ensure that the CDR database is as close to the Anywhere servers as possible. Anywhere doesn’t have any baked in control or management of SQL like Skype has and therefore supports any native HA and resilient SQL strategy. To Anywhere it’s just an app database that they need to read and write CDR data to, it contains no configuration. So AlwaysOn, Clustering and Mirroring technology can be used for HA.
In a DR situation though, you’ll want to ensure that the CDR database is also replicated to the DR datacenter. My guidance is to follow the supported cross datacenter topologies for the SQL HA technology you are deploying. For Mirroring, it is best to mirror to a local peer with a local witness and configure log shipping to the DR database. In the same way as you would with pChat in Skype. The failover script allows you to update the CDR DB connection string so that Anywhere will be looking in the right place for data.
Wallboard and Snapper Improvements
The client side applications seem to be treated like independent applications rather than extensions of the contact center. Therefore, require localised configuration files to configure for each contact center. The snapper isn’t so bad, but could benefit from an autodiscovery service to auto configure the snapper to look at the agent’s contact center membership. Initial deployment is fine, but MACDs would require snapper reconfiguration by the agent. This is GUI based but nevertheless a step that could be improved.
The Wallboard however relies on the config file for SLA calculations. If a manager wants to change the default SLA calculations this is done in the wallboard config.xml file. It would seemingly be feasible to include these SLA type calculations in SharePoint and push these out to the wallboard
These two small improvements would make the client side applications feel more integrated and easier to manage.
Mark is an Independent Microsoft Teams Consultant with over 15 years experience in Microsoft Technology. Mark is the founder of Commsverse, a dedicated Microsoft Teams conference and former MVP. You can follow him on twitter @UnifiedVale
Today while creating some test and training plans for a Anywhere365 deployment I am in the middle of, I came across an issue when an agent is signed into a Yealink T48G phone they are not available to be hunted in the Anywhere365 contact center. I must state that I suspect this problem to be present on any Skype for Business certified phone (non-LPE), but the Yealink is the only one I have to test with.
If the agent signed into their desktop client on their laptop, the phone would ring when they where hunted by the contact center. But if they weren’t, nothing.
In part 2, I described the end user / secretary experience. In this part I wanted to go through more of an advanced scenario. One of the problems with Lync delegate ringing is that when a call comes into the boss, all their delegates will simultaneously ring. For some customers and secretaries, this is a problem because bosses may have a direct secretary that should answer their calls, but equally the secretary has a hierarchy of other bosses secretaries who can answer the call in their absence. For instance, boss 1 mandates that secretary 1 should answer their call as a top priority. Other bosses secretaries should not be disturbed at this point. If secretary 1 does not answer the call due to either being away, or busy on another call, then boss 2’s secretary should answer the call and so on. How do we create this scenario?
(Easter Egg at the end of this article)
In part 1 of this mini series focussed around boss / secretary scenarios we looked at how to create a simple call flow that would enable secretaries to answer calls originally destined to the boss without the boss every knowing that they had a call. We took this further and created exceptions to allow some people to call the boss directly, but prevented everyone else. In this part, we are going to look at the experience from the secretary’s perspective.
So when the secretary’s client rings, the toast message looks like any other Skype for Business toast. Except that we don’t immediately see the caller ID as we do in a normal toast popup.
If you are running Lync Server 2013 or later, then there is a boss / admin role built-in to the core that allows you to configure delegates for a boss or manager. However, in order to do this you need SEFAUtil, but more recently in Skype for Business these commands have been introduced into native Powershell. The standard offer from the system for boss / secretary role does not always suit the scenario you are trying to configure for. By default, when enabled the boss’s endpoint will still ring at the same time as the delegates. However, the delegates can pick up calls for the boss using their endpoint (client or desk phone). The delegates can also place a call on hold and the boss can pick up the call on their endpoint. Another feature is that the deleagte can place calls on behalf of the boss.
Now for most situations this functionality is sufficient, but what happens if the boss does not want their endpoint to ring? What about if there is a dedicated secretary for a boss, but if they are not available a secretary of another boss should get the call? These scenarios become more challenging to configure that increase system admin effort, and also the end user experience is less than desirable.