What is IPSec?

A closer look at the internet's security protocol suite

IPsec and a series of security icon in small circles

IPsec, an extension of the widely adopted internet protocol (IP), encrypts network communications, protecting data against ‌theft‌ ‌and‌ ‌infiltration. Yet, history gives us a better understanding of IPsec.

In the 1970s and 80s, the US Department of Defense (DoD) developed the IP, laying the foundation for ARPANET, a precursor to the internet.

IP‌ ‌enabled data movement in discrete segments called‌ ‌packets‌ ‌from‌ ‌a‌ ‌source‌ ‌computer‌ ‌to‌ ‌a‌ ‌destination‌ ‌computer. Each packet, or datagram, contained control and user ‌information. The former included instructions to deliver the latter, which was known as the payload. Simply put, the IP encapsulates and routes data packets across IP networks.   

IP had its limitations, though.  The protocol lacked a mechanism to prevent data packets from being snooped on. TV shows in the past have repeatedly alluded‌ ‌to‌ ‌this‌ ‌flaw, indicating it wasn’t hidden for very long.

To end growing security concerns and prevent data sniffing across IP networks, the internet engineering task force (IETF) proposed IP security (IPsec) in 1995. The protocol has remained in use ever since.

The following guide provides insights into the inner workings of IPSec, its features, specific use cases, and more.

How does IPSec work?

IPsec is a layered security protocol that supports IP versions 6 and 4. There are three main protocols within IPsec: ‌‌authentication‌ ‌header‌ ‌(AH),‌ ‌encapsulating‌ ‌security‌ ‌payload‌s ‌(ESP),‌ and internet security association and key management protocol (ISAKMP)

Authentication Header

The authentication header (AH) does two things; it validates IP packets to source data origin and provides connectionless data integrity through hash keys. Hash functions allow arbitrary-sized data to be mapped to ‌fixed values, ensuring confidentiality.

Additionally, the AH layer offers protection against replay attacks. In a replay attack, data is retransmitted or delayed by malicious actors. AH‌ eliminates this problem by sequencing IP packets and discarding obsolete ones.

Encapsulating Security Payloads 

Like AH, the encapsulating security payloads (ESP) protocol authenticates sender and recipient identities and detects tampered IP packets, ensuring data integrity. But it is encryption that sets it apart. The ESP layer extends IP packets’ security by encrypting the entire payload.  

Internet Security Association and Key Management Protocol

ISAKMP adds to AH and ESP by defining security associations between two network entities or hosts exchanging information. Before data transmission, ISAKMP enables the sender and recipient hosts to decide what type of cryptographic algorithm to encrypt the IP packets with, session duration, network parameters, symmetric keys for decryption, and more. 

Notably, the IPsec protocol suite encrypts data exchanges between two hosts (host-to-host), networks (network-to-network), or between a security gateway and ‌host‌ ‌(network-to-host).

Methods of operation

IPSec supports two modes of operation: transport and tunnel.

Transport mode

An IP packet consists of two components: the header, which addresses and routes the packet, and the payload, which contains the data.

However, IP packets in transport mode only have their payload encrypted or authenticated. No changes are made to the header, which includes the source and target IP addresses. This is due to the lack of encryption by default in AH.

Transport mode is best suited for host-to-router network connection management.

Tunnel mode

As a stark contrast to transport mode, tunnel mode encrypts the entire IP packet. Essentially, a new IP header is added on top of the original IP header. This is also how private IP addresses or VPNs work. The IP header, which contains the source and the target IP address, is masked to prevent third-party interception. 

IPsec’s tunnel mode is best applied to network-to-network communications, host-to-network communications (remote user access), and host-to-host communications (private chat).


Depending on the operating system, IPSec may use one of three cryptic algorithms to perform authentication. They are:

  1. Rivest–Shamir–Adleman (RSA)
  2. Elliptic curve digital signature algorithm (ECDSA) 
  3. Pre-shared key (PSK)

Let’s go over each in detail.


Rivest–Shamir–Adleman (RSA) is a public-key cryptosystem named after its inventors Ron Rivest, Adi Shamir, and Leonard Adleman.

A public-key cryptosystem has one public encryption key and one‌ ‌private decryption‌ ‌key. Two large prime numbers and an auxiliary value are used to create the RSA public keys. 

Let’s say John wants to send a message to Clara. To encrypt the messages with RSA, John must know Clara's public key, and likewise, to decrypt them, Clara must use her private key. Clara sends her public key to John over a secure pathway so he can reliably send his encrypted messages. Remember, Clara's private key is never disclosed.

Elliptic curve digital signature algorithm

The ECDSA algorithm is a variant of the digital signature algorithm (DSA), a federal information processing standard for digital signatures. 

Technically, the ECDSA algorithm relies on the algebraic structure of elliptic curves over finite fields. Elliptic curves may be used for key agreement, digital signatures, or pseudo-random generators, among other things.

For elliptic curve cryptography to work, all participating devices must possess a private and public key pair. A message is signed with a private key by its sender, and the recipient uses the sender's public key to verify its authenticity.

Messages that have been altered on route to the recipient will not pass the signature verification test, as the signature only applies to the original message.

Pre-shared key 

In cryptography, a pre-shared key (PSK) is a shared secret that two parties have previously shared through a secure channel. 

Wi-Fi encryption standards, including wired equivalent privacy (WEP), Wi-Fi protected access (WPA) and the extensible authentication protocol (EAP) use PSK in their encryption methods.  The wireless access points (AP) and the clients share the same authentication key.

A PSK may be a password, a passphrase, or a hexadecimal string. 

IPsec versus SSL: What’s the difference?

The‌ ‌IPsec‌ ‌protocol‌ ‌suite‌ ‌operates‌ ‌at‌ ‌the‌ ‌network‌ ‌layer‌ ‌in‌ ‌the‌ open systems interconnection

(‌OSI‌) ‌model. The secure sockets layer (SSL), on the other hand, operates at the application layer of the OSI model.  Each serves a very different purpose: IPsec encrypts IP packers, while SSL encrypts HTTP traffic and web-based transactions.

Featured Resources

The definitive guide to warehouse efficiency

Get your free guide to creating efficiencies in the warehouse

Free download

The total economic impact™ of Datto

Cost savings and business benefits of using Datto Integrated Solutions

Download now

Three-step guide to modern customer experience

Support the critical role CX plays in your business

Free download

Ransomware report

The global state of the channel

Download now


What is TCP/IP?
Internet Protocol version 6 (IPv6)

What is TCP/IP?

10 Sep 2021
US 5G data shows urban-rural digital divide
Network & Internet

US 5G data shows urban-rural digital divide

7 Sep 2021
Bad bots account for almost 40% of internet traffic

Bad bots account for almost 40% of internet traffic

2 Sep 2021
NTT taps Celona to enable private 5G for enterprises

NTT taps Celona to enable private 5G for enterprises

18 Aug 2021

Most Popular

Zoom: From pandemic upstart to hybrid work giant
video conferencing

Zoom: From pandemic upstart to hybrid work giant

14 Sep 2021
What are the pros and cons of AI?
machine learning

What are the pros and cons of AI?

8 Sep 2021
Apple patches zero-day flaw abused by infamous NSO exploit

Apple patches zero-day flaw abused by infamous NSO exploit

14 Sep 2021