Skype for Business – Failed to execute a stored procedure on the back-end

I recently ran an “In-Place” upgrade from Lync 2013 to Skype for Business 2015 on a 3 node Enterprise Front End pool.  After the upgrade all 3 Front Ends were logging the following error every 10-15 minutes.

Log Name:      Lync Server
Source:        LS Data Collection
Date:          20/11/2015 16:40:38
Event ID:      56407
Task Category: (2271)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      LYNCFE00V.domain.local
Description:
Failed to execute a stored procedure on the back-end.

Component: QoE Adaptor 
Stored Procedure: QoeInsertSessionReport2 
Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s).
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d
Error Number:500,State:1,Class:16
Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error.
Resolution:
Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Data Collection" />
    <EventID Qualifiers="51423">56407</EventID>
    <Level>2</Level>
    <Task>2271</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-11-20T16:40:38.000000000Z" />
    <EventRecordID>49880</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>LYNCFE00V.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>QoE Adaptor</Data>
    <Data>QoeInsertSessionReport2</Data>
    <Data>System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s).
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp; task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp; task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d
Error Number:500,State:1,Class:16</Data>
  </EventData>
</Event>

I gathered something was wrong with the QOEMetrics database due to “Component QOE Adaptor”.  I ran the following PowerShell command to test the databases on my SQL Backend:

Test-CsDatabase -ConfiguredDatabases –SqlServerFqdn SQL03.domain.local | Out-GridView

(Where -SqlServerFqdn matches your Backend SQL server.)

You can see immediately that the QOEMetrics InstalledVersion is a point release lower than the ExpectedVersion.  To resolve this I ran the following PowerShell command:

Install-CsDatabase -DatabaseType Monitoring -SqlServerFqdn SQL03.domain.local -SqlInstanceName LyncBackend

(Where -SqlServerFqdn & -SqlInstanceName matches your Backend SQL server.)  Note, I only specified Monitoring for -DatabaseType.

I then ran Test-CsDatabase again:

Test-CsDatabase -ConfiguredDatabases –SqlServerFqdn SQL03.domain.local | Out-GridView

You can see that the ExpectedVersion and InstalledVersion now match.  This resolved this problem and the Event ID is no longer generated on the Front End Servers.