Skype for Business – Central Management Backup Module Failed

I recently deployed Skype for Business 2015 standard edition pool pairing.  The CMS was originally on a Lync 2010 Standard edition server and was moved over to a Skype4B Pool1 when Lync was decommissioned.

Skype4B Pool2 (FE2) was logging the following event IDs every few minutes:

Event ID 4082, LS Backup Service
Skype for Business Server 2015, Backup Service central management backup module failed to complete import operation.

Configurations: 
Backup Module Identity:CentralMgmt.CMSMaster
Working Directory path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\Temp
Local File Store Unc path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore
Remote File Store Unc path:\\FRONTEND.domain\SfBFileShare$\2-BackupService-3\BackupStore

 Additional Message: 
 Exception: Microsoft.Rtc.BackupService.ImportOperationException: Import operation (from zip archive \\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\CentralMgmt\CMSMaster\Data\Backup.zip) is failed due to: Failed to execute stored procedure XdsUpdateReplicaData. Native Error: 2000, Exception: Database layer is not in acquired state. Retriable: False. Cookie: <repl:Status xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008"><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="2cb13a5a-81ad-47f0-9162-5de8af04d85b" FullVersion="1" PartialVersion="248" /><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="b6a684fe-9989-4659-bffa-9f15eb07a7df" FullVersion="1" PartialVersion="6" /></repl:Status>. ---> Microsoft.Rtc.Common.Data.SqlConnectionException: Database layer is not in acquired state
   --- End of inner exception stack trace ---
   at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.UpdateBackupClusterInSingleTransaction(Unzipper unzipper, String newCookie)
   at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.ApplyChanges(Unzipper unzipper, String& newCookie, Boolean& forceSetErrorState)
   at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.ApplyChanges(Boolean& forceSetErrorState)
   at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute()
   at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state)

Cause: Either network or permission issues. Please look through the exception details for more information.
Resolution:
Resolution

Event 4097, LS Backup Service

Skype for Business Server 2015, Backup Service central management backup module failed to register with back-end database. The module will continuously attempt to reconnect to the back-end. While this condition persists, no backup will be done.

The Connection string: Data Source=(local)\rtc;Initial Catalog=xds;Integrated Security=True;Enlist=False;Pooling=True;Max Pool Size=3;Connect Timeout=60;Application Name=Microsoft.Rtc.Management.Core
Exception: [2000] Cannot open database "xds" requested by the login. The login failed.
Login failed for user 'NT AUTHORITY\NETWORK SERVICE'.
Cause: Possible issues with central management backup module back-end database.
Resolution:
Ensure central management backup module back-end is functioning correctly.

I found the following post TechNet but Windows Firewall was disabled for Domain Networks on the customers environment.

I downloaded SQL 2014 Management Studio (MgmtStudio 64BIT\SQLManagementStudio_x64_ENU.exe) from Microsoft Download.  I then connected to the RTC (FRONTEND\RTC) Instance on both Front End servers and opened the security on the XDS (Central Management Store) database:

Front End 2:

Front End 1: 

It was immediately apparent that FE2 had different security to FE1.  To resolve on FE2:

• Added “FRONTEND\RTC Local Config Replicator” as a SQL Login under Security
• Granted the following groups the correct role on the XDS Database:
  • FRONTEND\RTC Local Administrators – Publisher Role
  • FRONTEND\RTC Local Config Replicator – Replicator Role
  • FRONTEND\RTC Local Read-only Administrators – Consumer Role

For good measure I restarted the Skype4B services on FE2 (Stop-CsWindowsService / Start-CsWindowsService)

The event logs are now clean on FE2.

Chris Hayward

Technical Architect at Symity