Lync 2013 – Standard Edition Pool Pair Failover Script

Lync 2013 introduced a new feature called Pool Pairing. Given the scenario below with 2 sites using Lync Standard Edition with paired pools, in the event that Front End 01 was to fail, or Site 1 failed completely all users will automatically register to Front End 02. However they will receive an error in their Lync client “Limited Functionality due to Outage”, lose their buddy lists, presence information, conferencing features and any Response Groups that were homed on FE01. They will still be able to make inbound and outbound PSTN calls (assuming you have resilient Voice Routing) and they will still be able to search contacts to make P2P audio and video calls. To restore full functionality you need to manually failover the pool following the procedure here: http://technet.microsoft.com/en-us/library/jj204678.aspx

Remembering or finding the various commands to run in a disaster scenario wastes valuable restore time so I have created a script to do all the hard work for you. The script can also fail back a pool once it is back in service. This script relies on my Lync Backup script to restore Response Groups here (I save these on the standby server if Active/Passive): http://weakestlync.com/2013/07/07/lync-2013-automated-backup-script/

Simple Lync Standard Edition Pool Pair topology:

Limited functionality error in Lync 2013 client:

The script is menu driven, the splash screen will show the script configuration, it is important to review all of this information before running:

From the main menu you have various options.

Options:

• a – failover pool01 to pool02
• b – failover pool01’s response groups to pool02
• c – fail back pool01’s users to pool01
• d – fail back pool01’s response groups to pool01
• i – view manual DNS changes required during failover
• j – shows pool state – Use to check if a pool is currently in a failover state
• k – shows CMS status – Check replication is working and which server is master
• l – view failed over response group status for both pools – Use to check if any Response Groups have been failed over
• m – view backup service status for both pools – Use to check that the backup service is working OK
• z – enable advanced mode (show failover options for both pools and CMS) – Use if you want to failover Pool02 to Pool01 or manually move CMS

Option z:

Here is the script, you need to modify the Script Configuration at the top to reflect your environment.:

Use this script at your own risk. I have not programmed Lync Enterprise Edition but it could easily be modified to failover an Enterprise Pool.

# Author: Created by Weakest Lync, www.WeakestLync.com
# Purpose: Script to failover Lync 2013 Standard Edition Pool Pair and Response Groups in a DR scenario
# Version: 1.0
# Changes: DATE - Change
# 22/01/2014 - Release 1.0

# Script Configuration
$CustomerName = "Weakest Lync" # Customer Name
$PrimaryFE = "frontend01.WeakestLync.com" # Primary Lync Front End Pool
$PrimaryEd = "edge01.WeakestLync.com" # Primary Lync Edge Pool
$SecondaryFE = "frontend02.WeakestLync.com" # Secondary Lync Front End Pool
$SecondaryEd = "edge02.WeakestLync.com" # Secondary Lync Edge Pool
$BackupPath = "C:LyncBackupsBackups" # Backup Path - Uses Lync Automated Backup script to restore Response Groups: http://weakestlync.com/2013/07/07/lync-2013-automated-backup-script/
$ActivePassive = "Yes" # yes/no. Is the Secondary Pool used for failover only? This will hide menu options to failover the Secondary Pool.
$RGSRestorePath = "c:Temp" # Temp folder

Function WindowColour {
$a = (Get-Host).UI.RawUI
$a.BackgroundColor = "DarkBlue"
}

Function FailoverDR ($Pool, $BackupPool, $Edge){
cls
MainHeader
# Prompt user to confirm
Write-Host "You are about to initiate a pool failover from " -NoNewline
Write-Host $Pool -ForegroundColor Green -NoNewline
Write-Host " to " -NoNewline
Write-Host $BackupPool -ForegroundColor Green
Write-Host ""
$confirm = Read-Host "Please type YES to confirm"
If ($confirm -ne "YES"){
MainMenu
}

# Check that CMS is on Secondary server
$LyncCMS = Get-CsManagementStoreReplicationStatus -CentralManagementStoreStatus
If ($LyncCMS.ActiveMasterFqdn -ne $BackupPool){
Write-Warning "CMS is not on Secondary server, running CMS failover command"
Invoke-CsManagementServerFailover -BackupSqlServerFqdn $BackupPool -BackupSqlInstanceName RTC -Force
}

# Change Primary Edge next hop to Secondary Front End
Set-CsEdgeServer -Identity $Edge -Registrar Registrar:$BackupPool

# Failover pool
Invoke-CsPoolFailOver -PoolFqdn $Pool -DisasterMode -Confirm:$false
ReturnToMenu
}

Function FailbackDR ($Pool, $BackupPool, $Edge){
cls
MainHeader
# Prompt user to confirm
Write-Host "You are about to initiate a pool failback from " -NoNewline
Write-Host $BackupPool -ForegroundColor Green -NoNewline
Write-Host " to " -NoNewline
Write-Host $Pool -ForegroundColor Green
Write-Host ""
$confirm = Read-Host "Please type YES to confirm"
If ($confirm -ne "YES"){
MainMenu
}

# Start CMS Replication
Invoke-CsManagementStoreReplication -ReplicaFqdn $Pool

# Change Primary Edge next hop to Primary Front End
Set-CsEdgeServer -Identity $Edge -Registrar Registrar:$Pool

# Failback pool
Invoke-CsPoolFailBack –PoolFqdn $Pool -Confirm:$false
ReturnToMenu
}

Function FailoverRGS ($Pool, $BackupPool){
cls
MainHeader
# Prompt user to confirm
Write-Host "You are about to initiate a Response Group failover from " -NoNewline
Write-Host $Pool -ForegroundColor Green -NoNewline
Write-Host " to " -NoNewline
Write-Host $BackupPool -ForegroundColor Green
Write-Host ""
$confirm = Read-Host "Please type YES to confirm"
If ($confirm -ne "YES"){
MainMenu
}

# Get the latest RGS Backup file
$LatestBackup = Get-ChildItem -Directory $BackupPath | Sort CreationTime -Descending | select -First 1
$RGSBackupPath = $LatestBackup.FullName + "Lync-RGS-Config_$Pool.zip"

# Test RGS Backup file exists
If(!(Test-Path $RGSBackupPath)){
Write-Warning "No RGS Backup File"
Write-Warning "Ensure that an RGS backup is located in the following location $RGSBackupPath"
ReturnToMenu
}

# Import Response Group config to secondary pool
Import-CsRgsConfiguration -Destination "service:ApplicationServer:$BackupPool" -FileName $RGSBackupPath -ReplaceExistingRgsSettings -Force
ReturnToMenu
}

Function FailbackRGS ($Pool, $BackupPool){
cls
MainHeader
# Prompt user to confirm
Write-Host "You are about to initiate a Response Group failback from " -NoNewline
Write-Host $BackupPool -ForegroundColor Green -NoNewline
Write-Host " to " -NoNewline
Write-Host $Pool -ForegroundColor Green
Write-Host ""
$confirm = Read-Host "Please type YES to confirm"
If ($confirm -ne "YES"){
MainMenu
}

# Check temp path exists
If(!(Test-Path $RGSRestorePath)){
New-Item -ItemType directory -Path $RGSRestorePath
}

# Temp export file name
$RGSRestoreFile = $RGSRestorePath + "RGSExportPrimaryPoolRestore.zip"

# If we already have a temp export file, delete it.
If(Test-Path $RGSRestoreFile){
Remove-Item $RGSRestoreFile
}

# Export Response Groups from Secondary pool that are owned by the Primary, then remove them.
Export-CsRgsConfiguration –Source "service:ApplicationServer:$BackupPool" –Owner "service:ApplicationServer:$Pool" –Filename $RGSRestoreFile -RemoveExportedConfiguration -Force

# Import Response Groups owned by the Primary pool back in to the Primary.
Import-CsRgsConfiguration –Destination "service:ApplicationServer:$Pool" –OverwriteOwner –Filename $RGSRestoreFile -Force
ReturnToMenu
}

Function MoveCMS($BackupPool){
cls
MainHeader
# Prompt user to confirm
Write-Host "You are about to move the Active CMS to " -NoNewline
Write-Host $BackupPool -ForegroundColor Green
Write-Host ""
$confirm = Read-Host "Please type YES to confirm"
If ($confirm -ne "YES"){
MainMenu
}
$LyncCMS = Get-CsManagementStoreReplicationStatus -CentralManagementStoreStatus
If ($LyncCMS.ActiveMasterFqdn -ne $BackupPool){
Write-Warning "Server is not CMS Active Master, running CMS failover command"
Invoke-CsManagementServerFailover -BackupSqlServerFqdn $BackupPool -BackupSqlInstanceName RTC -Force
Write-Host "CMS Active Master moved to $BackupPool"
}
Else{
Write-Host "CMS Active Master is already on $BackupPool"
}
ReturnToMenu
}

Function ManualChanges {
cls
MainHeader
Write-Host "DNS Changes during Failover:"
Write-Host "* Point lyncdiscover.sipdomain DNS record to Secondary Pool Web Services"
Write-Host "* Point dialin.sipdomain DNS record to point to Secondary Pool Web Services"
Write-Host "* Point meet.sipdomain DNS record to point to Secondary Pool Web Services"
Write-Host ""
Write-Host "DNS Changes during Failback:"
Write-Host "* Point lyncdiscover.sipdomain DNS record to Primary Pool Web Services"
Write-Host "* Point dialin.sipdomain DNS record to point to Primary Pool Web Services"
Write-Host "* Point meet.sipdomain DNS record to point to Primary Pool Web Services"
Write-Host ""
ReturnToMenu
}

Function FailoverStatus{
cls
MainHeader
Get-CsRegistrarConfiguration | Select-Object Identity, PoolState | Format-Table
ReturnToMenu
}

Function CMSStatus{
cls
MainHeader
Get-CsManagementStoreReplicationStatus
Read-Host "Press any key to view more details"
Get-CsManagementStoreReplicationStatus -CentralManagementStoreStatus
ReturnToMenu
}

Function RGSStatus{
cls
MainHeader
Read-Host "Press any key to view RGS Workflows in $SecondaryFE owned by $PrimaryFE"
Get-CsRgsWorkflow -Identity "service:ApplicationServer:$SecondaryFE" -Owner "service:ApplicationServer:$PrimaryFE" | Select-Object OwnerPool, Name, LineUri | Format-Table
Read-Host "Press any key to view RGS Workflows in $PrimaryFE owned by $SecondaryFE"
Get-CsRgsWorkflow -Identity "service:ApplicationServer:$PrimaryFE" -Owner "service:ApplicationServer:$SecondaryFE" | Select-Object OwnerPool, Name, LineUri | Format-Table
ReturnToMenu
}

Function BackupServiceStatus{
cls
MainHeader
Read-Host "Press any key to view the Backup Service status for $PrimaryFE"
Get-CsBackupServiceStatus -PoolFqdn $PrimaryFE | Format-List
Read-Host "Press any key to view the Backup Service status for $SecondaryFE"
Get-CsBackupServiceStatus -PoolFqdn $SecondaryFE | Format-List
ReturnToMenu
}

Function AdvancedMode{
# Forces script to show failover options for both pools
$ActivePassive = "no"
# Show advanced options on the menu
$AdvancedMode = "Yes"
MainMenu
}

Function ReturnToMenu{
Read-Host "Press any key to return to main menu"
MainMenu
}

Function MainHeader{
cls
Write-Host " *** Lync 2013 Standard Edition Failover Script ***" -Foregroundcolor Magenta
Write-Host ""
}

Function JustExit{
cls
exit
}

Function MainMenu ($blank){
Get-Variable true | Out-Default; Clear-Host; # Required due to PowerShell cls bug not clearing display
MainHeader
# Menu
Write-Host "Primary Pool Disaster Recovery: " -NoNewline -ForegroundColor Yello
Write-Host $PrimaryFE -ForegroundColor Green
Write-Host "a. Disaster - Failover Pool + CMS"
Write-Host "b. Disaster - Failover Response Groups"
Write-Host "c. Recovery - Failback Pool"
Write-Host "d. Recovery - Failback Response Groups"
If ($ActivePassive -eq "no"){
Write-Host "Secondary Pool Disaster Recovery: " -NoNewline -ForegroundColor Yellow
Write-Host $SecondaryFE -ForegroundColor Green
Write-Host "e. Disaster - Failover Pool + CMS"
Write-Host "f. Disaster - Failover Response Groups"
Write-Host "g. Recovery - Failback Pool"
Write-Host "h. Recovery - Failback Response Groups"
}
Write-Host "Administration: "-ForegroundColor Yellow
Write-Host "i. View manual changes required during failover/failback"
Write-Host "j. View Registrar failover status"
Write-Host "k. View Central Management Store status"
Write-Host "l. View failed over Response Group status"
Write-Host "m. View Backup Service status"
If ($AdvancedMode -eq "Yes"){
Write-Host "CMS Management: "-ForegroundColor Yellow
Write-Host "n. Move CMS to Primary Pool"
Write-Host "o. Move CMS to Secondary Pool"
}
else{
Write-Host "z. Advanced Mode"
}
Write-Host "q. Exit Script (q, exit, quit)"
Write-Host ""
If ($blank -eq "1"){
Write-Warning "Invalid selection"
}
$a = Read-Host "Select an option (e.g. q)"

switch ($a){
a {FailoverDR -Pool $PrimaryFE -BackupPool $SecondaryFE -Edge $PrimaryEd}
b {FailoverRGS -Pool $PrimaryFE -BackupPool $SecondaryFE}
c {FailbackDR -Pool $PrimaryFE -BackupPool $SecondaryFE -Edge $PrimaryEd}
d {FailbackRGS -Pool $PrimaryFE -BackupPool $SecondaryFE}
e {FailoverDR -Pool $SecondaryFE -BackupPool $PrimaryFE -Edge $SecondaryEd}
f {FailoverRGS -Pool $SecondaryFE -BackupPool $PrimaryFE}
g {FailbackDR -Pool $SecondaryFE -BackupPool $PrimaryFE -Edge $SecondaryEd}
h {FailbackRGS -Pool $SecondaryFE -BackupPool $PrimaryFE}
i {ManualChanges}
J {FailoverStatus}
k {CMSStatus}
l {RGSStatus}
m {BackupServiceStatus}
n {MoveCMS -BackupPool $PrimaryFE}
o {MoveCMS -BackupPool $SecondaryFE}
q {JustExit}
z {AdvancedMode}
exit {JustExit}
quit {JustExit}
default {MainMenu -blank "1"}
}
}

Function SplashScreen{
MainHeader
Write-Host "* This script can be used to failover a Lync Standard Edition Pool Pair."
Write-Host "* Make sure you are running this script as Administrator (Lync Management Shell > Run As Administrator)."
Write-Host "* Your account must be a member of the RTCUniversalServerAdmins and CsAdministrator groups."
Write-Host ""
Write-Host "* Customer Platform is " -NoNewline
Write-Host $CustomerName -ForegroundColor Green
Write-Host "* Primary Front End is " -NoNewline
Write-Host $PrimaryFE -ForegroundColor Green
Write-Host "* Primary Edge is " -NoNewline
Write-Host $PrimaryEd -ForegroundColor Green
Write-Host "* Secondary Front End is " -NoNewline
Write-Host $SecondaryFE -ForegroundColor Green
Write-Host "* Secondary Edge is " -NoNewLine
Write-Host $SecondaryEd -ForegroundColor Green
Write-Host "* Backup path (For RGS failover) is " -NoNewLine
Write-Host $BackupPath -ForegroundColor Green
Write-Host "* Is Pool Pair Active/Passive? (Secondary Pool is standby only) " -NoNewLine
Write-Host $ActivePassive -ForegroundColor Green
Write-Host ""
Write-Host "Please review the details above, amend the script config if required."
Write-Host ""
Read-Host "Press any key to load menu"
MainMenu
}
WindowColour
SplashScreen

Chris Hayward

Technical Architect at Symity