Skype for Business – Failed to execute a stored procedure on the back-end

I recently ran an “In-Place” upgrade from Lync 2013 to Skype for Business 2015 on a 3 node Enterprise Front End pool.  After the upgrade all 3 Front Ends were logging the following error every 10-15 minutes.

Log Name:      Lync Server
Source:        LS Data Collection
Date:          20/11/2015 16:40:38
Event ID:      56407
Task Category: (2271)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      LYNCFE00V.domain.local
Description:
Failed to execute a stored procedure on the back-end.

Component: QoE Adaptor 
Stored Procedure: QoeInsertSessionReport2 
Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s).
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d
Error Number:500,State:1,Class:16
Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error.
Resolution:
Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Data Collection" />
    <EventID Qualifiers="51423">56407</EventID>
    <Level>2</Level>
    <Task>2271</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-11-20T16:40:38.000000000Z" />
    <EventRecordID>49880</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>LYNCFE00V.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>QoE Adaptor</Data>
    <Data>QoeInsertSessionReport2</Data>
    <Data>System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s).
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp; task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp; task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d
Error Number:500,State:1,Class:16</Data>
  </EventData>
</Event>

I gathered something was wrong with the QOEMetrics database due to “Component QOE Adaptor”.  I ran the following PowerShell command to test the databases on my SQL Backend:

Test-CsDatabase -ConfiguredDatabases –SqlServerFqdn SQL03.domain.local | Out-GridView

(Where -SqlServerFqdn matches your Backend SQL server.)

You can see immediately that the QOEMetrics InstalledVersion is a point release lower than the ExpectedVersion.  To resolve this I ran the following PowerShell command:

Install-CsDatabase -DatabaseType Monitoring -SqlServerFqdn SQL03.domain.local -SqlInstanceName LyncBackend

(Where -SqlServerFqdn & -SqlInstanceName matches your Backend SQL server.)  Note, I only specified Monitoring for -DatabaseType.

I then ran Test-CsDatabase again:

Test-CsDatabase -ConfiguredDatabases –SqlServerFqdn SQL03.domain.local | Out-GridView

You can see that the ExpectedVersion and InstalledVersion now match.  This resolved this problem and the Event ID is no longer generated on the Front End Servers.

Chris Hayward

Technical Architect at Symity