Secure network con­nec­tions with IPsec

IP packets, the fun­da­men­tal unit of internet data transfer, consist of two parts: the actual payload (voice, text, images) and header data con­tain­ing sender and recipient addresses. The core problem: IP itself provides no con­fi­den­tial­i­ty or au­then­ti­ca­tion at the network layer – re­gard­less of any ap­pli­ca­tion-level en­cryp­tion the payload may already carry. Without ad­di­tion­al mech­a­nisms, packets can be read or ma­nip­u­lat­ed in transit, leaving three security prop­er­ties un­pro­tect­ed: con­fi­den­tial­i­ty, integrity, and au­then­ti­ca­tion (the security services defined in the IPsec ar­chi­tec­ture, RFC 4301).

IPsec (Internet Protocol Security) was developed to close this gap. The protocol suite operates at the network layer and secures IP com­mu­ni­ca­tions in­de­pen­dent­ly of the ap­pli­ca­tion being used – which is why it remains the backbone of most site-to-site VPN con­nec­tions today.

What is IPsec?

IPsec is a protocol suite stan­dard­ized by the Internet En­gi­neer­ing Task Force (IETF). Early IPsec spec­i­fi­ca­tions (from RFC 1825, 1995, onward) addressed both IPv4 and IPv6. The key dis­tinc­tion today is that IPsec support is mandatory for IPv6 (RFC 6434) and optional for IPv4, where it is deployed as an add-on. In practice, IPsec is widely used in both protocol versions. The suite divides into three func­tion­al groups:

• Transfer protocols: Au­then­ti­ca­tion Header (AH, RFC 4302), En­cap­su­lat­ing Security Payload (ESP, RFC 4303)
• Key man­age­ment: Internet Key Exchange v2 (IKEv2, RFC 7296); ISAKMP (RFC 2408, now Historic) is a legacy pre­de­ces­sor used with IKEv1
• Databases: Security As­so­ci­a­tion Database (SAD), Security Policy Database (SPD) – defined in the IPsec ar­chi­tec­ture RFC 4301

Com­po­nents of the protocol suite

The two main transfer protocols differ sig­nif­i­cant­ly in what they protect and how:

ESP (En­cap­su­lat­ing Security Payload) is the standard for all modern IPsec de­ploy­ments. It provides en­cryp­tion, au­then­ti­ca­tion, and integrity pro­tec­tion in one header. When used with an AEAD cipher like AES-GCM, ESP handles en­cryp­tion and au­then­ti­ca­tion in a single pass – no separate HMAC cal­cu­la­tion is needed.

AH (Au­then­ti­ca­tion Header) provides integrity and origin au­then­ti­ca­tion only – but no en­cryp­tion. More crit­i­cal­ly, AH au­then­ti­cates the entire packet including the outer IP header, making it in­com­pat­i­ble with NAT – a problem in virtually every modern network. ESP with au­then­ti­ca­tion achieves equiv­a­lent pro­tec­tion and works through NAT. Current BSI (TR-02102-3) and NIST guide­lines recommend ESP ex­clu­sive­ly; AH is con­sid­ered ef­fec­tive­ly obsolete.

ESP protects data through four mech­a­nisms:

• Con­fi­den­tial­i­ty – payload content is encrypted
• Integrity – any tampering in transit is detected
• Origin au­then­ti­ca­tion – confirms the packet came from the expected peer
• Replay pro­tec­tion – each packet carries a mo­not­o­n­i­cal­ly in­creas­ing sequence number; the receiver maintains an anti-replay window and rejects any packet whose number falls outside the window or has already been processed

SAD and SPD

The Security Policy Database (SPD) defines which traffic should be protected and with which protocol. The Security As­so­ci­a­tion Database (SAD) holds the active session pa­ra­me­ters ne­go­ti­at­ed by IKEv2: for each SA, the SAD stores the Security Parameter Index (SPI), the chosen al­go­rithms, and the as­so­ci­at­ed key material. Each SA is uni­di­rec­tion­al – a single protected con­nec­tion therefore requires two SAs, one per direction, each with its own SPI and keys.

IKEv1 vs. IKEv2

IKEv2, stan­dard­ized in RFC 7296 (2014), is the current key man­age­ment standard for IPsec. IKEv1 (RFC 2409, 1998) is dep­re­cat­ed and should not be used in new de­ploy­ments – it has known cryp­to­graph­ic weak­ness­es, a complex handshake, and lacks features standard in modern im­ple­men­ta­tions. If your VPN gateway still defaults to IKEv1, update the con­fig­u­ra­tion.

IKEv2 re­designed the handshake from the ground up and adds several critical ca­pa­bil­i­ties over IKEv1:

• Simpler handshake: always 4 messages (IKEv1 Main Mode required 6–9)
• Built-in NAT traversal: automatic ESP-in-UDP en­cap­su­la­tion on port 4500 when NAT is detected
• MOBIKE (RFC 4555): allows a tunnel to survive an IP address change without rene­go­ti­at­ing – essential for devices switching between Wi-Fi and cellular networks
• EAP au­then­ti­ca­tion: native support for cer­tifi­cates, pre-shared keys, and EAP
• Dead Peer Detection: built in via IN­FOR­MA­TION­AL exchanges (IKEv1 required a separate extension)
• Perfect Forward Secrecy: con­fig­urable per child SA – ensures past sessions cannot be decrypted even if a long-term key is later com­pro­mised

The two types of IPsec: Tunnel vs. Transport

There are two existing transfer modes for secure con­nec­tions with IPsec: transport mode, in which two end hosts are connected directly, and tunnel mode, which creates a con­nec­tion between two IP networks.

Transport mode

In transport mode, IPsec inserts its headers between the existing IP header and the payload. The original IP header – including source and des­ti­na­tion addresses – remains visible. Only the payload is protected. This mode is efficient and fast, but does not conceal the com­mu­ni­cat­ing endpoints. The cryp­to­graph­ic and com­mu­nica­tive terminal points are identical.

Typical use: host-to-host con­nec­tions, network man­age­ment traffic, and as the inner tunnel in L2TP/IPsec con­fig­u­ra­tions for remote access clients.

Tunnel mode

In tunnel mode, the entire original IP packet is en­cap­su­lat­ed inside a new packet with a new outer IP header. Both the payload and the original header (with the real source and des­ti­na­tion addresses) are hidden inside the encrypted ESP payload – only the gateway addresses are visible in transit.

Typical use: site-to-site VPNs (gateway to gateway) and road warrior VPN clients where endpoint identity must be concealed. When IKEv2 detects NAT in the path during its handshake, it au­to­mat­i­cal­ly en­cap­su­lates ESP inside UDP on port 4500, allowing IPsec tunnels to pass through routers and firewalls that would otherwise drop raw ESP packets (IP protocol 50).

Cipher suites and security rec­om­men­da­tions

Choosing the right al­go­rithms is as important as deploying IPsec correctly. The following is aligned with BSI Technical Guideline TR-02102-3 (2024) and current NIST rec­om­men­da­tions.

En­cryp­tion (ESP):

• AES-256-GCM – Rec­om­mend­ed. Au­then­ti­cat­ed en­cryp­tion (AEAD) – no separate HMAC needed. Hardware-ac­cel­er­at­ed via AES-NI on modern CPUs.
• AES-128-GCM – Ac­cept­able for most de­ploy­ments.
• AES-256-CBC – Ac­cept­able if combined with HMAC-SHA-256 or SHA-384. Avoid without separate au­then­ti­ca­tion.
• 3DES / DES – Do not use. Dep­re­cat­ed by NIST (2023). Vul­ner­a­ble to the Sweet32 attack (3DES) or trivially broken (DES).

Integrity / Au­then­ti­ca­tion (for non-AEAD ciphers):

• HMAC-SHA-256 – Minimum rec­om­men­da­tion.
• HMAC-SHA-384 / SHA-512 – Preferred for high-security en­vi­ron­ments.
• HMAC-MD5 / HMAC-SHA-1 – Do not use.

Key exchange – IKEv2 Diffie-Hellman groups:

• Group 19 (P-256 / ECDH) – Rec­om­mend­ed default.
• Group 20 / 21 (P-384 / P-521) – Higher security margin for sensitive de­ploy­ments.
• Group 14 (2048-bit MODP) – Minimum ac­cept­able; only for legacy com­pat­i­bil­i­ty.
• Groups 1, 2, 5 (768–1536-bit) – Do not use. Con­sid­ered broken.

Looking ahead: Post-quantum cryp­tog­ra­phy

Quantum computers capable of breaking current asym­met­ric key exchange are not yet op­er­a­tional, but the IETF is already preparing the tran­si­tion. RFC 9370 (2023) defines multiple key exchanges in IKEv2, enabling hybrid classical/post-quantum key agreement. NIST finalized its first post-quantum algorithm standards in 2024 (FIPS 203, 204, 205). High-security and long-lifecycle de­ploy­ments should monitor BSI and NIST guidance on migration timelines.

IPsec: strengths and weak­ness­es

When using VPNs – the most common ap­pli­ca­tion area of the protocol suite – IPsec operates in­de­pen­dent­ly from any ap­pli­ca­tion at network level. Once the con­nec­tion is made, different forms of data (e-mail, file transfer, IP telephony) can be sent without ap­pli­ca­tion-specific tools. It scales to thousands of si­mul­ta­ne­ous tunnels, is supported natively by major operating systems, and benefits from wide­spread hardware ac­cel­er­a­tion (AES-NI).

The trade-offs: IPsec is complex to configure correctly, and mis­con­fig­u­ra­tion is a genuine risk. Its large codebase is difficult to audit fully. Devices that still ship with IKEv1 or weak ciphers enabled by default require manual hardening.

Com­par­i­son with modern al­ter­na­tives:

• IPsec / IKEv2 (network layer): best for site-to-site VPNs, en­ter­prise en­vi­ron­ments, and mobile clients – mature, scales well, complex con­fig­u­ra­tion.
• SSL / TLS VPN (ap­pli­ca­tion layer): best for remote access and client­less browser-based scenarios – easier firewall traversal, per-app tunneling.
• WireGuard (network layer): a modern al­ter­na­tive in­te­grat­ed into the Linux kernel since version 5.6 (2020). Its minimal codebase (~4,000 lines) makes it sig­nif­i­cant­ly easier to audit and it con­sis­tent­ly out­per­forms IPsec in bench­marks on con­strained hardware. WireGuard uses a fixed, modern cipher suite (ChaCha20-Poly1305, Curve25519) and is worth eval­u­at­ing for green­field Linux de­ploy­ments. It lacks the en­ter­prise feature set and vendor in­ter­op­er­abil­i­ty that IPsec provides.

Use cases

Site-to-site VPN: The most common IPsec de­ploy­ment. Two network gateways establish a permanent tunnel in tunnel mode, en­crypt­ing all traffic between two office or dat­a­cen­ter networks. IKEv2 handles key exchange and re-keying; ESP with AES-256-GCM encrypts the payload. Dead Peer Detection au­to­mat­i­cal­ly re-es­tab­lish­es the tunnel if a peer becomes un­reach­able.

Remote access (road warrior): IKEv2/IPsec combined with EAP au­then­ti­ca­tion allows in­di­vid­ual users to connect securely from laptops and mobile devices. MOBIKE ensures the tunnel persists when the device’s IP address changes – for example, when switching from a hotel Wi-Fi to a cellular con­nec­tion. Windows, macOS, iOS, and Android all include a native IKEv2 client, requiring no third-party software in­stal­la­tion.

Host-to-host (transport mode): Used for securing traffic between specific servers – database repli­ca­tion, inter-dat­a­cen­ter syn­chro­niza­tion, or man­age­ment plane com­mu­ni­ca­tion – without routing through a gateway. Transport mode adds minimal overhead but requires IPsec policy con­fig­u­ra­tion on both endpoints.