VPN Error 809 for L2TP/IPSec on Windows behind NAT

• Windows Server
  •  
  •  
  •  
  •  
  •  
  •  
  •  
• Active Directory
  •  
  •  
• Windows Clients
  •  
  •  
  •  
  •  
  •  
  •  
  •  
• Virtualization
  •  
  •  
  •  
• PowerShell
• Exchange
• Cloud
  •  
  •  
  •  
• Linux
  •  
  •  
  •  

  Windows OS Hub / Windows 10 / Configuring L2TP/IPSec VPN Connection Behind a NAT, VPN Error Code 809

September 18, 2020 Windows 10Windows Server 2016

Configuring L2TP/IPSec VPN Connection Behind a NAT, VPN Error Code 809

Due to disabling PPTP VPN support in iOS, one of my clients decided to reconfigure the VPN server running Windows Server 2012 R2 from PPTP to L2TP/IPSec. Internal VPN clients from inside LAN connect to the VPN server without any problems, however external Windows clients get the error 809 when trying to establish the connection with the L2TP VPN server:

Can’t connect to L2TP-IPsec-VPN-Server.hostname

The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g. firewalls, NAT, routers, etc) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem.

In other Windows versions, the connection errors 800, 794 or 809 may indicate the same problem.

It is worth to note that the VPN server is behind a NAT, and the router is configured to forward L2TP ports:

• UDP 1701 — Layer 2 Forwarding Protocol (L2F) & Layer 2 Tunneling Protocol (L2TP)
• UDP 500
• UDP 4500 NAT-T – IPSec Network Address Translator Traversal
• Protocol 50 ESP

These ports are also open in the Windows Firewall rules for VPN connection. Those, the classic configuration is used. The built-in Windows VPN client is used for connection.

If you connect to the same VPN server via PPTP, the connection is successfully established.

VPN Error 809 for L2TP/IPSec on Windows behind NAT

As it turned out, the problem is already known and described in the article https://support.microsoft.com/en-us/kb/926179. The Windows built-in VPN client doesn’t support by default L2TP/IPsec connections through NAT. This is because IPsec uses ESP (Encapsulating Security Payload) to encrypt packets, and ESP doesn’t support PAT (Port Address Translation). If you want to use IPSec for communication, Microsoft recommends using public IP addresses on the VPN server.

But there is also a workaround. You can fix this drawback by enabling support for the NAT-T protocol, which allows you to encapsulate ESP 50 packets in UDP packets on port 4500. NAT-T is enabled by default in almost all operating systems (iOS, Android, Linux) except Windows.

If the L2TP/IPsec VPN server is behind a NAT device, in order to connect external clients through NAT correctly, you have to make some changes to the registry both on the server and client side to allow UDP packet encapsulation for L2TP and NAT-T support in IPsec.

• Open the Registry Editor (regedit.exe) and go to the following registry key:
  • Windows 10/8.1/Vista  and Windows Server 2016/2012R2/2008R2 — HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesPolicyAgent
  • Windows XP/Windows Server 2003 — HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesIPSec
• Create a DWORD parameter with the name AssumeUDPEncapsulationContextOnSendRule and the value 2;Note. Possible AssumeUDPEncapsulationContextOnSendRule values are:
  • 0 – (a default value) suggests that the server is connected to the Internet without NAT;
  • 1 – the VPN server is behind a NAT device ;
  • 2 — both VPN server and client are behind a NAT.
• Just restart your computer and make sure that the VPN tunnel is established successfully

[alert] If both Windows VPN server and client are behind NAT, you need to change this setting on both devices.

Also, you can use a PowerShell cmdlet to make changes to the registry:

Set-ItemProperty -Path "HKLM:SYSTEMCurrentControlSetServicesPolicyAgent" -Name "AssumeUDPEncapsulationContextOnSendRule" -Type DWORD -Value 2 –Force;

After enabling NAT-T support, you will be able to successfully connect to the VPN server from the client through NAT (including double NAT).

Multiple L2TP VPN Connections from the same LAN

There is another interesting VPN bug. If your local network has several Windows computers, you cannot establish more than one simultaneous connection to an external L2TP/IPSec VPN server. If you try to connect to the same VPN server from another computer (with an active VPN tunnel from different device), error code 809 or 789 will appear:

Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remove computer.

Interestingly, this problem only occurs on Windows devices. On Linux/MacOS/Android devices on the same local network, there are no such problems. You can easily connect to the VPN L2TP server from multiple devices at the same time.

According to TechNet, the issue is related to incorrect implementation of the L2TP/IPSec client on Windows (not fixed for many years).

To fix this bug, you need to change two registry parameters in the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesRasManParameters registry key and restart your computer:

• AllowL2TPWeakCrypto – change to 00000001 (allows weak encryption algorithms, for L2TP/IPSec the MD5 and DES algorithms are used);
• ProhibitIPSec – change to 00000000 (enables IPsec encryption, which is often disabled by some VPN clients or system tools).

Run the following command to change apply these registry changes:

reg add "HKEY_LOCAL_MACHINESystemCurrentControlSetServicesRasmanParameters" /v AllowL2TPWeakCrypto /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINESystemCurrentControlSetServicesRasmanParameters" /v ProhibitIpSec /t REG_DWORD /d 0 /f

This enables support for concurrent L2TP/IPSec VPN connections on Windows through a shared public IP address (works on all versions from Windows XP to Windows 10).