I recently ran an “In-Place” upgrade from Lync 2013 to Skype for Business 2015 on a 3 node Enterprise Front End pool. After the upgrade all 3 Front Ends were logging the following error every 10-15 minutes. Log Name: Lync Server Source: LS Data Collection Date: 20/11/2015 16:40:38 Event ID: 56407 Task Category: (2271) Level: Error Keywords: Classic User: N/A Computer: LYNCFE00V.domain.local Description: Failed to execute a stored procedure on the back-end. Component: QoE Adaptor Stored Procedure: QoeInsertSessionReport2 Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s). at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction) ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d Error Number:500,State:1,Class:16 Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error. Resolution: Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="LS Data Collection" /> <EventID Qualifiers="51423">56407</EventID> <Level>2</Level> <Task>2271</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-11-20T16:40:38.000000000Z" /> <EventRecordID>49880</EventRecordID> <Channel>Lync Server</Channel> <Computer>LYNCFE00V.domain.local</Computer> <Security /> </System> <EventData> <Data>QoE Adaptor</Data> <Data>QoeInsertSessionReport2</Data> <Data>System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s). at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp; task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp; task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction) ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d Error Number:500,State:1,Class:16</Data> </EventData> </Event> I gathered something was wrong with the QOEMetrics database due to “Component QOE Adaptor”. I ran the following PowerShell command to test the databases on my SQL Backend: ...

Here is an example Hyper-V replica email notification script. Scenario is Hyper-V cluster replicating business critical VMs to a single Hyper-V host. The script can be ran as a scheduled task on the DR Hyper-V host to send email notifications. The account running the script must have Hyper-V permissions. Script: Hyper-V 2012 Replica Email Script # Created by Chris Hayward # Purpose - To send a daily email displaying Hyper-V Replica status # Version 1.0 # Get date in UK format day/month/year $date = Get-Date -Format dd/MM/yy # Variables $MailFrom = 'hyperv_replica@yourdomain.com' $MailTo = 'you@yourdomain.com' $MailSubject = "Hyper-V Replica Report $date" $MailServer = 'smtp.yourdomain.coml' # Get replication status in HTML format $status = Get-VMReplication | Select-Object Name, State, Health, Mode, FrequencySec, PrimaryServer, ReplicaServer, ReplicaPort | ConvertTo-Html # Send email message Send-MailMessage -From $MailFrom -To $MailTo -Subject $MailSubject -BodyAsHtml -Body "$status" -SmtpServer $MailServer Example output: ...

I recently deployed Skype for Business 2015 standard edition pool pairing. The CMS was originally on a Lync 2010 Standard edition server and was moved over to a Skype4B Pool1 when Lync was decommissioned. Skype4B Pool2 (FE2) was logging the following event IDs every few minutes: Error1 Event ID 4082, LS Backup Service Skype for Business Server 2015, Backup Service central management backup module failed to complete import operation. Configurations: Backup Module Identity:CentralMgmt.CMSMaster Working Directory path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\Temp Local File Store Unc path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore Remote File Store Unc path:\\FRONTEND.domain\SfBFileShare$\2-BackupService-3\BackupStore Additional Message: Exception: Microsoft.Rtc.BackupService.ImportOperationException: Import operation (from zip archive \\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\CentralMgmt\CMSMaster\Data\Backup.zip) is failed due to: Failed to execute stored procedure XdsUpdateReplicaData. Native Error: 2000, Exception: Database layer is not in acquired state. Retriable: False. Cookie: <repl:Status xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008"><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="2cb13a5a-81ad-47f0-9162-5de8af04d85b" FullVersion="1" PartialVersion="248" /><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="b6a684fe-9989-4659-bffa-9f15eb07a7df" FullVersion="1" PartialVersion="6" /></repl:Status>. ---> Microsoft.Rtc.Common.Data.SqlConnectionException: Database layer is not in acquired state --- End of inner exception stack trace --- at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.UpdateBackupClusterInSingleTransaction(Unzipper unzipper, String newCookie) at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.ApplyChanges(Unzipper unzipper, String& newCookie, Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.ApplyChanges(Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute() at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state) Cause: Either network or permission issues. Please look through the exception details for more information. Resolution: Resolution Error2 ...

Ok here is a little tip when trying to add trusted root certificate authorities to Skype for Business 2015 and Lync 2010/2013 Edge servers. Microsoft have a list of Unified Communications Certificate Providers detailed here: https://support.microsoft.com/en-us/kb/929395 Windows now automatically downloads Trusted Root certificates when you access a web page on HTTPS. We can exploit this to get all Trusted Root certificates on our Edge servers which is required for Federation to work correctly. An example of this is where you only get one way IM or a federated partner may see your presence as unknown. ...

I recently came across an error where the Cluster Service stopped on a Windows Server 2012 R2 failover cluster and would not start. Event Logs generated were: Event ID: 7024 The Cluster Service service terminated with the following service-specific error: Keyset does not exist Event ID: 7031 The Cluster Service service terminated unexpectedly. It has done this 282 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service. ...

I have recently encountered a strange issue where all 3 Front End servers in the same pool ran out of disk space at the same time. When investigating both sqlservr.exe processes (RTCLOCAL and LYNCLOCAL) were consuming the whole page file (or all of the available space on C: as the pagefile was system managed). In Resource Monitor I could see the commit for both sqlservr.exe processes was extremely high. Apologies for the poor images, I was using iMessage to send the images to a colleague and didn’t screenshot: ...

I recently came across an issue where the SCCM 2012 R2 Client would not install on one machine (OS Server 2012 R2). The machine had previously had a working SCCM client that was updated to CU4. The Configuration Manager icon in Control Panel was still visible, but when opening it all of the values were blank. I was also unable to uninstall the client using ccmsetup.exe /uninstall. Forcing an install from the SCCM Console and selecting uninstall previous SCCM clients also did not work. ...

Microsoft detail how to move users from Lync On-Prem to Lync Online and how to bulk move users in an OU or with a particular AD attribute: https://technet.microsoft.com/en-us/library/jj204969%28v=ocs.15%29.aspx?f=255&MSPPError=-2147217396. If you want to bulk move users to Lync Online from a CSV you can use the PowerShell below: $creds=Get-Credential Import-Csv movelyncusers.csv -Header Identity | Foreach-Object {Move-CsUser -Identity $_.identity -Target sipfed.online.lync.com -Credential $creds -HostedMigrationOverrideUrl https://admin1e.online.lync.com/HostedMigration/hostedmigrationService.svc} movelyncusers.csv should contain a list of the UPNs of the users you wish to move. No headings are required.

A customer was having problems with their Lync 2010 deployment federating with Office 365. When searching for a Lync Online user they would see “Presence Unknown” and be unable to send Instant Messages. They could Federate with other On-Prem Lync organisations OK. They had LyncOnline configured as a Hosted Provider and Partner Domain Discovery Enabled (Dynamic Federation). Their _sipfederationtls._tcp DNS record was configured correctly as was the O365 customer. Using OCSLogger and Snooper on the Edge server I was able to spot the problem. ...

I recently ran into a problem with Checkpoint Firewalls on a Lync 2010 deployment. The customer already had a single 2010 Front End and wished to enable external sign in over the internet as it did not work over their VPN solution (Direct Access). I installed the Edge server and Reverse Proxy but then encountered a strange issue where I was unable to sign in over the Edge server. Some of the troubleshooting steps I took (Thinking it was my fault): ...