I recently deployed Skype for Business 2015 standard edition pool pairing. The CMS was originally on a Lync 2010 Standard edition server and was moved over to a Skype4B Pool1 when Lync was decommissioned.
Skype4B Pool2 (FE2) was logging the following event IDs every few minutes:
Event ID 4082, LS Backup Service Skype for Business Server 2015, Backup Service central management backup module failed to complete import operation. Configurations: Backup Module Identity:CentralMgmt.CMSMaster Working Directory path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\Temp Local File Store Unc path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore Remote File Store Unc path:\\FRONTEND.domain\SfBFileShare$\2-BackupService-3\BackupStore Additional Message: Exception: Microsoft.Rtc.BackupService.ImportOperationException: Import operation (from zip archive \\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\CentralMgmt\CMSMaster\Data\Backup.zip) is failed due to: Failed to execute stored procedure XdsUpdateReplicaData. Native Error: 2000, Exception: Database layer is not in acquired state. Retriable: False. Cookie: <repl:Status xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008"><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="2cb13a5a-81ad-47f0-9162-5de8af04d85b" FullVersion="1" PartialVersion="248" /><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="b6a684fe-9989-4659-bffa-9f15eb07a7df" FullVersion="1" PartialVersion="6" /></repl:Status>. ---> Microsoft.Rtc.Common.Data.SqlConnectionException: Database layer is not in acquired state --- End of inner exception stack trace --- at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.UpdateBackupClusterInSingleTransaction(Unzipper unzipper, String newCookie) at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.ApplyChanges(Unzipper unzipper, String& newCookie, Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.ApplyChanges(Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute() at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state) Cause: Either network or permission issues. Please look through the exception details for more information. Resolution: Resolution
Event 4097, LS Backup Service Skype for Business Server 2015, Backup Service central management backup module failed to register with back-end database. The module will continuously attempt to reconnect to the back-end. While this condition persists, no backup will be done. The Connection string: Data Source=(local)\rtc;Initial Catalog=xds;Integrated Security=True;Enlist=False;Pooling=True;Max Pool Size=3;Connect Timeout=60;Application Name=Microsoft.Rtc.Management.Core Exception:  Cannot open database "xds" requested by the login. The login failed. Login failed for user 'NT AUTHORITY\NETWORK SERVICE'. Cause: Possible issues with central management backup module back-end database. Resolution: Ensure central management backup module back-end is functioning correctly.
I found the following post TechNet but Windows Firewall was disabled for Domain Networks on the customers environment.
I downloaded SQL 2014 Management Studio (MgmtStudio 64BIT\SQLManagementStudio_x64_ENU.exe) from Microsoft Download. I then connected to the RTC (FRONTEND\RTC) Instance on both Front End servers and opened the security on the XDS (Central Management Store) database:
It was immediately apparent that FE2 had different security to FE1. To resolve on FE2:
- Added “FRONTEND\RTC Local Config Replicator” as a SQL Login under Security
- Granted the following groups the correct role on the XDS Database:
- FRONTEND\RTC Local Administrators – Publisher Role
- FRONTEND\RTC Local Config Replicator – Replicator Role
- FRONTEND\RTC Local Read-only Administrators – Consumer Role
For good measure I restarted the Skype4B services on FE2 (Stop-CsWindowsService / Start-CsWindowsService)
The event logs are now clean on FE2.
Technical Architect at Symity