Lync / Skype4B

When enabling users for Skype for Business you may encounter the following error: “Active Directory operation failed on “Domain Controller”. You cannot retry this operation: “Insufficient access rights to perform the operation” I first blogged about this in 2011 for Microsoft Lync 2010 when moving users from an OCS 2007 R2 to Lync 2010 pool: /2011/03/28/lync-2010-move-user-1-errors-failed-while-updating-destination-pool/ This problem can occur when: Enabling a user Moving a user between pools Changing any of the users Skype for Business properties, including policies You verify that your administrative account is a member of CsAdministrator and you may also be a member of RTCUniversalServerAdmins and Domain Admins. ...

I recently came across an issue where the SCOM (System Center Operations Manager) Agent was installed on all servers in a customers Skype for Business (SfB) Edge pool but they were not appearing in the SfB Management Pack (Skype for Business 2015 Server Health). All Edge servers were showing as “Agent Managed” in SCOM and appeared in “Discovered Inventory” as Critical. Using Health Explorer on each Edge server showed the following: ...

A few years ago I created a Lync 2013 backup script that would backup data from all servers in the Topology. Lync 2013 Automated Backup Script This script also works with Skype for Business 2015 but it specifically backed up Response Group and User data by identifying the pool version. The script below has been updated and tested on SfB 2015. <# Author: Chris Hayward, chrishayward.co.uk Purpose: Centralised script to backup Lync 2013 and SfB 2015 Version: 1.2 Changes: DATE - Change 06/11/2013 - Release 1.1 - Fixed code that deletes old backup directories (CH) 27/11/2015 - 1.2 Updated for Skype for Business (version -gt 7 rather than -eq 6) #> Import-Module Lync # Script Config $BackupPath = "D:\_Backup\" # Backup Path $BackupRetention = -14 # Number of days to keep old backups (Requires a "-", e.g. two weeks would be -14, 5 days would be -5) # *** Start of script *** # Delete old backup directories Get-ChildItem $BackupPath |? {$_.psiscontainer -and $_.lastwritetime -le (get-date).adddays($BackupRetention)} |% {remove-item $_.FullName -recurse -force} # Get date in day-month-year format and set backup path $DateStamp = Get-Date -Format dd-MM-yy $BackupPath = $BackupPath + "LyncBackup_" + $DateStamp +"\" New-Item $BackupPath -type directory -force # Backup the Lync Central Management Store $CMSBackupFile = $BackupPath + "Lync-CMS-Config.zip" Export-CsConfiguration -FileName $CMSBackupFile # Backup Location Information Service data $LISBackupFile = $BackupPath + "Lync-LIS-Config.zip" Export-CsLisConfiguration -FileName $LISBackupFile # Backup User Data (Lync 2013 & SfB 2015 servers only) $UserServer = Get-CsService -UserServer | Where-Object {$_.version -ge 6} ForEach ($Service in $UserServer) { $UserDataBackupFile = $BackupPath + "Lync-User-Config_" + $($Service.PoolFqdn) + ".zip" Export-CsUserData -PoolFQDN $($Service.PoolFqdn) -FileName $UserDataBackupFile } # Backup Response Group Configuration (Lync 2013 & SfB 2015 servers only) $ApplicationServer = Get-CsService -ApplicationServer | Where-Object {$_.version -ge 6} ForEach ($Service in $ApplicationServer) { $RGSBackupFile = $BackupPath + "Lync-RGS-Config_" + $($Service.PoolFqdn) + ".zip" Export-CsRgsConfiguration -Source ApplicationServer:$($Service.PoolFqdn) -FileName $RGSBackupFile } # Backup Voice Configuration $BackupPath = $BackupPath + "Lync-Voice-Config\" New-Item $BackupPath -type directory -force Get-CsDialPlan | Export-Clixml -path $BackupPath\DialPlan.xml Get-CsVoicePolicy | Export-Clixml -path $BackupPath\VoicePolicy.xml Get-CsVoiceRoute | Export-Clixml -path $BackupPath\VoiceRoute.xml Get-CsPstnUsage | Export-Clixml -path $BackupPath\PSTNUsage.xml Get-CsVoiceConfiguration | Export-Clixml -path $BackupPath\VoiceConfiguration.xml Get-CsTrunkConfiguration | Export-Clixml -path $BackupPath\TrunkConfiguration.xml To run the script as a scheduled task: ...

I recently ran an “In-Place” upgrade from Lync 2013 to Skype for Business 2015 on a 3 node Enterprise Front End pool. After the upgrade all 3 Front Ends were logging the following error every 10-15 minutes. Log Name: Lync Server Source: LS Data Collection Date: 20/11/2015 16:40:38 Event ID: 56407 Task Category: (2271) Level: Error Keywords: Classic User: N/A Computer: LYNCFE00V.domain.local Description: Failed to execute a stored procedure on the back-end. Component: QoE Adaptor Stored Procedure: QoeInsertSessionReport2 Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s). at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction) ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d Error Number:500,State:1,Class:16 Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error. Resolution: Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="LS Data Collection" /> <EventID Qualifiers="51423">56407</EventID> <Level>2</Level> <Task>2271</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-11-20T16:40:38.000000000Z" /> <EventRecordID>49880</EventRecordID> <Channel>Lync Server</Channel> <Computer>LYNCFE00V.domain.local</Computer> <Security /> </System> <EventData> <Data>QoE Adaptor</Data> <Data>QoeInsertSessionReport2</Data> <Data>System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s). at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp; task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp; task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction) ClientConnectionId:35301abd-a45b-4593-a2e4-5faf4257097d Error Number:500,State:1,Class:16</Data> </EventData> </Event> I gathered something was wrong with the QOEMetrics database due to “Component QOE Adaptor”. I ran the following PowerShell command to test the databases on my SQL Backend: ...

I recently deployed Skype for Business 2015 standard edition pool pairing. The CMS was originally on a Lync 2010 Standard edition server and was moved over to a Skype4B Pool1 when Lync was decommissioned. Skype4B Pool2 (FE2) was logging the following event IDs every few minutes: Error1 Event ID 4082, LS Backup Service Skype for Business Server 2015, Backup Service central management backup module failed to complete import operation. Configurations: Backup Module Identity:CentralMgmt.CMSMaster Working Directory path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\Temp Local File Store Unc path:\\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore Remote File Store Unc path:\\FRONTEND.domain\SfBFileShare$\2-BackupService-3\BackupStore Additional Message: Exception: Microsoft.Rtc.BackupService.ImportOperationException: Import operation (from zip archive \\FRONTEND.domain\SfBFileShare$\3-BackupService-2\BackupStore\CentralMgmt\CMSMaster\Data\Backup.zip) is failed due to: Failed to execute stored procedure XdsUpdateReplicaData. Native Error: 2000, Exception: Database layer is not in acquired state. Retriable: False. Cookie: <repl:Status xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008"><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="2cb13a5a-81ad-47f0-9162-5de8af04d85b" FullVersion="1" PartialVersion="248" /><repl:Batch xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" Id="b6a684fe-9989-4659-bffa-9f15eb07a7df" FullVersion="1" PartialVersion="6" /></repl:Status>. ---> Microsoft.Rtc.Common.Data.SqlConnectionException: Database layer is not in acquired state --- End of inner exception stack trace --- at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.UpdateBackupClusterInSingleTransaction(Unzipper unzipper, String newCookie) at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.ApplyChanges(Unzipper unzipper, String& newCookie, Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.ApplyChanges(Boolean& forceSetErrorState) at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute() at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state) Cause: Either network or permission issues. Please look through the exception details for more information. Resolution: Resolution Error2 ...

Ok here is a little tip when trying to add trusted root certificate authorities to Skype for Business 2015 and Lync 2010/2013 Edge servers. Microsoft have a list of Unified Communications Certificate Providers detailed here: https://support.microsoft.com/en-us/kb/929395 Windows now automatically downloads Trusted Root certificates when you access a web page on HTTPS. We can exploit this to get all Trusted Root certificates on our Edge servers which is required for Federation to work correctly. An example of this is where you only get one way IM or a federated partner may see your presence as unknown. ...

I have recently encountered a strange issue where all 3 Front End servers in the same pool ran out of disk space at the same time. When investigating both sqlservr.exe processes (RTCLOCAL and LYNCLOCAL) were consuming the whole page file (or all of the available space on C: as the pagefile was system managed). In Resource Monitor I could see the commit for both sqlservr.exe processes was extremely high. Apologies for the poor images, I was using iMessage to send the images to a colleague and didn’t screenshot: ...

Microsoft detail how to move users from Lync On-Prem to Lync Online and how to bulk move users in an OU or with a particular AD attribute: https://technet.microsoft.com/en-us/library/jj204969%28v=ocs.15%29.aspx?f=255&MSPPError=-2147217396. If you want to bulk move users to Lync Online from a CSV you can use the PowerShell below: $creds=Get-Credential Import-Csv movelyncusers.csv -Header Identity | Foreach-Object {Move-CsUser -Identity $_.identity -Target sipfed.online.lync.com -Credential $creds -HostedMigrationOverrideUrl https://admin1e.online.lync.com/HostedMigration/hostedmigrationService.svc} movelyncusers.csv should contain a list of the UPNs of the users you wish to move. No headings are required.

A customer was having problems with their Lync 2010 deployment federating with Office 365. When searching for a Lync Online user they would see “Presence Unknown” and be unable to send Instant Messages. They could Federate with other On-Prem Lync organisations OK. They had LyncOnline configured as a Hosted Provider and Partner Domain Discovery Enabled (Dynamic Federation). Their _sipfederationtls._tcp DNS record was configured correctly as was the O365 customer. Using OCSLogger and Snooper on the Edge server I was able to spot the problem. ...

I recently ran into a problem with Checkpoint Firewalls on a Lync 2010 deployment. The customer already had a single 2010 Front End and wished to enable external sign in over the internet as it did not work over their VPN solution (Direct Access). I installed the Edge server and Reverse Proxy but then encountered a strange issue where I was unable to sign in over the Edge server. Some of the troubleshooting steps I took (Thinking it was my fault): ...