Skype for Business – Failed to execute a stored procedure on the back-end

I recently ran an “In-Place” upgrade from Lync 2013 to Skype for Business 2015 on a 3 node Enterprise Front End pool. After the upgrade all 3 Front Ends were logging the following error every 10-15 minutes. Log Name: Lync Server Source: LS Data Collection Date: 20/11/2015 16:40:38 Event ID: 56407 Task Category: (2271) Level: Error Keywords: Classic User: N/A Computer: LYNCFE00V.domain.local Description: Failed to execute a stored procedure on the back-end. Component: QoE Adaptor Stored Procedure: QoeInsertSessionReport2 Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s). at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction) ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d Error Number:500,State:1,Class:16 Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error. Resolution: Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="LS Data Collection" /> <EventID Qualifiers="51423">56407</EventID> <Level>2</Level> <Task>2271</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-11-20T16:40:38.000000000Z" /> <EventRecordID>49880</EventRecordID> <Channel>Lync Server</Channel> <Computer>LYNCFE00V.domain.local</Computer> <Security /> </System> <EventData> <Data>QoE Adaptor</Data> <Data>QoeInsertSessionReport2</Data> <Data>System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s). at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp; task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp; task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction) ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d Error Number:500,State:1,Class:16</Data> </EventData> </Event> I gathered something was wrong with the QOEMetrics database due to “Component QOE Adaptor”. I ran the following PowerShell command to test the databases on my SQL Backend: ...

20 November 2015 · 3 min · Chris Hayward

Skype for Business – Central Management Backup Module Failed

I recently deployed Skype for Business 2015 standard edition pool pairing. The CMS was originally on a Lync 2010 Standard edition server and was moved over to a Skype4B Pool1 when Lync was decommissioned. Skype4B Pool2 (FE2) was logging the following event IDs every few minutes: Error1 Event ID 4082, LS Backup Service Skype for Business Server 2015, Backup Service central management backup module failed to complete import operation. Configurations: Backup Module Identity:CentralMgmt.CMSMaster Working Directory path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\Temp Local File Store Unc path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore Remote File Store Unc path:\\FRONTEND.domain\SfBFileShare$\2-BackupService-3\BackupStore Additional Message: Exception: Microsoft.Rtc.BackupService.ImportOperationException: Import operation (from zip archive \\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\CentralMgmt\CMSMaster\Data\Backup.zip) is failed due to: Failed to execute stored procedure XdsUpdateReplicaData. Native Error: 2000, Exception: Database layer is not in acquired state. Retriable: False. Cookie: <repl:Status xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008"><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="2cb13a5a-81ad-47f0-9162-5de8af04d85b" FullVersion="1" PartialVersion="248" /><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="b6a684fe-9989-4659-bffa-9f15eb07a7df" FullVersion="1" PartialVersion="6" /></repl:Status>. ---> Microsoft.Rtc.Common.Data.SqlConnectionException: Database layer is not in acquired state --- End of inner exception stack trace --- at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.UpdateBackupClusterInSingleTransaction(Unzipper unzipper, String newCookie) at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.ApplyChanges(Unzipper unzipper, String& newCookie, Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.ApplyChanges(Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute() at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state) Cause: Either network or permission issues. Please look through the exception details for more information. Resolution: Resolution Error2 ...

18 September 2015 · 2 min · Chris Hayward

Skype for Business (Lync) – Easiest way to add trusted root certificate authorities to Edge

Ok here is a little tip when trying to add trusted root certificate authorities to Skype for Business 2015 and Lync 2010/2013 Edge servers. Microsoft have a list of Unified Communications Certificate Providers detailed here: https://support.microsoft.com/en-us/kb/929395 Windows now automatically downloads Trusted Root certificates when you access a web page on HTTPS. We can exploit this to get all Trusted Root certificates on our Edge servers which is required for Federation to work correctly. An example of this is where you only get one way IM or a federated partner may see your presence as unknown. ...

17 September 2015 · 2 min · Chris Hayward

Lync – Move On Prem users to Lync Online from CSV

Microsoft detail how to move users from Lync On-Prem to Lync Online and how to bulk move users in an OU or with a particular AD attribute: https://technet.microsoft.com/en-us/library/jj204969%28v=ocs.15%29.aspx?f=255&MSPPError=-2147217396. If you want to bulk move users to Lync Online from a CSV you can use the PowerShell below: $creds=Get-Credential Import-Csv movelyncusers.csv -Header Identity | Foreach-Object {Move-CsUser -Identity $_.identity -Target sipfed.online.lync.com -Credential $creds -HostedMigrationOverrideUrl https://admin1e.online.lync.com/HostedMigration/hostedmigrationService.svc} movelyncusers.csv should contain a list of the UPNs of the users you wish to move. No headings are required.

17 April 2015 · 1 min · Chris Hayward

Lync – Increase Internal Certificate Validity Period

When you deploy Lync and assign internal certificates to your Lync servers (E.g. Front End, Internal Edge, SBA and Gateways) by default you will only have a 2 year certificate from the “Web Server” template. I like to create a new Certificate Template for Lync with a 5 year lifespan as it can be quite annoying having to renew certificates, especially on the Edge servers and Gateways where you have to export the cert request. I have tried creating a 10 year Certificate Template but the Lync Deployment Wizard complains that the cert is valid for over 5 years. It’s also a good idea to set your Certificate Authority validity period to a long period, I usually opt for 20/25 years. This article will cover how to create a 5 year Certificate Template for Lync. By assigning Certificates to your Lync servers with a longer validity period you won’t have to renew them as often. ...

24 February 2014 · 2 min · Chris Hayward