Lync 2013 – TLS Negotiation took too long to complete

I recently came across an issue with Lync 2013 where SCOM (System Center Operations Manager) was periodically reporting errors with the Conferencing Attendant and IM MCU.  The SCOM errors were as follows:

Alert: [LYNC] Conferencing Attendant failed to join the conference.
Source: LS Conferencing Auto Attendant Component [F05.DOMAIN.LOCAL]
Path: FE05.DOMAIN.LOCAL
Last modified by: System
Last modified time: 4/11/2016 11:01:50 AM Alert description: Conferencing Attendant failed to join the conference.

Microsoft.Rtc.Signaling.ConnectionFailureException:Operation failed because the network connection was not available. ---> Microsoft.Rtc.Internal.Sip.TLSException: TLS Negotiation took too long to complete

Alert: [LYNC] Lync Server IM MCU failed to connect to the SIP Front End.
Source: LS Instant Message Conferencing Component [FE03.DOMAIN.LOCAL]
Path: FE03.DOMAIN.LOCAL
Last modified by: System
Last modified time: 4/11/2016 11:35:24 AM Alert description: Lync Server IM MCU failed to connect to the SIP Front End.

Local Machine name: FE03.DOMAIN.LOCAL FE hostname: FE06.DOMAIN.LOCAL IP address: 10.10.10.13:5061 Message: TLS Negotiation took too long to complete

Strangely, no users had reported issues and initial thoughts were performance problems with the Lync Front End servers.  This is quite a large Pool with 7 Front Ends and each Front End would periodically generate the above errors in SCOM.  All servers were running the latest CU and all certificates were valid for 2 years.

These errors could also be seen in the Lync event logs on each Front End:

Log Name:      Lync Server
Source:        LS Conferencing Announcement Service
Date:          2/10/2016 1:05:07 PM
Event ID:      33129
Task Category: (1301)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE04.DOMAIN.LOCAL
Description:
Conferencing Announcement Service could not negotiate TLS and failed to establish a SIP connection.

Reason=Other (1460). Front End=FEPOOL01.DOMAIN.LOCAL.
Cause: This issue may occur due to TLS certificates on the Microsoft Lync Server 2013 Front End and the Conferencing Announcement Service being expired, or not in sync.
Resolution:
Check and resolve certificate issues, mismatches between the Microsoft Lync Server 2013 Front End and Conferencing Announcement Service to restore connectivity.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Conferencing Announcement Service" />
    <EventID Qualifiers="50453">33129</EventID>
    <Level>2</Level>
    <Task>1301</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-02-10T13:05:07.000000000Z" />
    <EventRecordID>129050</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE04.DOMAIN.LOCAL</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Other (1460)</Data>
    <Data>FEPOOL01.DOMAIN.LOCAL</Data>
  </EventData>
</Event>

Log Name:      Lync Server
Source:        LS IM MCU
Date:          2/9/2016 6:02:34 PM
Event ID:      33019
Task Category: (1019)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE06.DOMAIN.LOCAL
Description:
Lync Server IM MCU failed to connect to the SIP Front End.

Local Machine name: FE06.DOMAIN.LOCAL FE hostname: FE05.DOMAIN.LOCAL IP address: 10.10.10.12:5061 Message: Peer disconnected while outbound TLS negotiation was in progress
Stack: 

Cause: Lync Server IM MCU cannot communicate with the Microsoft Lync Server 2013 Front End Service over SIP due to network connectivity issues or unavailability of the Microsoft Lync Server 2013 Front End Service. This condition prevents setting up new conferencing sessions, forwarding Instant Messages and is disruptive and potentially fatal for existing conferencing sessions.
Resolution:
Please ensure network connectivity and availability of the Microsoft Lync Server 2013 Front End Service for the IM Conferencing Service to be able to function correctly.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS IM MCU" />
    <EventID Qualifiers="50171">33019</EventID>
    <Level>2</Level>
    <Task>1019</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-02-09T18:02:34.000000000Z" />
    <EventRecordID>77079</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE06.DOMAIN.LOCAL</Computer>
    <Security />
  </System>
  <EventData>
    <Data>FE06.DOMAIN.LOCAL</Data>
    <Data>FE05.DOMAIN.LOCAL</Data>
    <Data>10.10.10.12:5061</Data>
    <Data>Peer disconnected while outbound TLS negotiation was in progress</Data>
    <Data>
    </Data>
  </EventData>
</Event>

It transpired that the Lync Front End servers could not access the CRL (Certificate Revocation List) due to Firewalls between the Lync servers and the customers Certificate Authority.  You can view the CRL distribution points via the certificate Details tab, example below (Note the CRL may also be published via LDAP internally):

You can check Lync servers have successfully accessed the CRL by running: certutil -urlcache CRL.  If the CRL has been accessed you will see it in the list returned.

You can also verify if the Lync servers can access the CRL by doing the following:

1. Export Front End certificate from MMC to C:\temp\LyncFE.cer (No need to export Private Key)
2. Run from elevated command prompt: certutil.exe -v -verify -URLfetch c:\temp\LyncFE.cer > C:\temp\certutil.txt
3. Open certutil.txt from C:\temp

If the CRL can’t be accessed you will see the following error towards the bottom of the file: ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)

If the CRL can be accessed you will see the following towards the bottom of the file: Leaf certificate revocation check passed

Once the Firewall ports were opened between Lync and the CA server hosting the CRL the TLS Timeout Errors in the Lync Event logs and SCOM stopped.  It is likely that if you see this issue in Skype for Business 2015, it’s also caused by the CRL being inaccessible.

Chris Hayward

Technical Architect at Symity