IT Pro is supported by its audience. When you purchase through links on our site, we may earn an affiliate commission. Learn more

What is IPSec?

A closer look at the internet's security protocol suite

IPsec and a series of security icon in small circles

IPsec, an extension of the widely adopted internet protocol (IP), encrypts network communications, protecting data against ‌theft‌ ‌and‌ ‌infiltration. Yet, history gives us a better understanding of IPsec.

In the 1970s and 80s, the US Department of Defense (DoD) developed the IP, laying the foundation for ARPANET, a precursor to the internet.

IP‌ ‌enabled data movement in discrete segments called‌ ‌packets‌ ‌from‌ ‌a‌ ‌source‌ ‌computer‌ ‌to‌ ‌a‌ ‌destination‌ ‌computer. Each packet, or datagram, contained control and user ‌information. The former included instructions to deliver the latter, which was known as the payload. Simply put, the IP encapsulates and routes data packets across IP networks.   

IP had its limitations, though.  The protocol lacked a mechanism to prevent data packets from being snooped on. TV shows in the past have repeatedly alluded‌ ‌to‌ ‌this‌ ‌flaw, indicating it wasn’t hidden for very long.

To end growing security concerns and prevent data sniffing across IP networks, the internet engineering task force (IETF) proposed IP security (IPsec) in 1995. The protocol has remained in use ever since.

The following guide provides insights into the inner workings of IPSec, its features, specific use cases, and more.

How does IPSec work?

IPsec is a layered security protocol that supports IP versions 6 and 4. There are three main protocols within IPsec: ‌‌authentication‌ ‌header‌ ‌(AH),‌ ‌encapsulating‌ ‌security‌ ‌payload‌s ‌(ESP),‌ and internet security association and key management protocol (ISAKMP)

Authentication Header

The authentication header (AH) does two things; it validates IP packets to source data origin and provides connectionless data integrity through hash keys. Hash functions allow arbitrary-sized data to be mapped to ‌fixed values, ensuring confidentiality.

Additionally, the AH layer offers protection against replay attacks. In a replay attack, data is retransmitted or delayed by malicious actors. AH‌ eliminates this problem by sequencing IP packets and discarding obsolete ones.

Encapsulating Security Payloads 

Like AH, the encapsulating security payloads (ESP) protocol authenticates sender and recipient identities and detects tampered IP packets, ensuring data integrity. But it is encryption that sets it apart. The ESP layer extends IP packets’ security by encrypting the entire payload.  

Internet Security Association and Key Management Protocol

ISAKMP adds to AH and ESP by defining security associations between two network entities or hosts exchanging information. Before data transmission, ISAKMP enables the sender and recipient hosts to decide what type of cryptographic algorithm to encrypt the IP packets with, session duration, network parameters, symmetric keys for decryption, and more. 

Notably, the IPsec protocol suite encrypts data exchanges between two hosts (host-to-host), networks (network-to-network), or between a security gateway and ‌host‌ ‌(network-to-host).

Methods of operation

IPSec supports two modes of operation: transport and tunnel.

Transport mode

An IP packet consists of two components: the header, which addresses and routes the packet, and the payload, which contains the data.

However, IP packets in transport mode only have their payload encrypted or authenticated. No changes are made to the header, which includes the source and target IP addresses. This is due to the lack of encryption by default in AH.

Transport mode is best suited for host-to-router network connection management.

Tunnel mode

As a stark contrast to transport mode, tunnel mode encrypts the entire IP packet. Essentially, a new IP header is added on top of the original IP header. This is also how private IP addresses or VPNs work. The IP header, which contains the source and the target IP address, is masked to prevent third-party interception. 

IPsec’s tunnel mode is best applied to network-to-network communications, host-to-network communications (remote user access), and host-to-host communications (private chat).


Depending on the operating system, IPSec may use one of three cryptic algorithms to perform authentication. They are:

  1. Rivest–Shamir–Adleman (RSA)
  2. Elliptic curve digital signature algorithm (ECDSA) 
  3. Pre-shared key (PSK)

Let’s go over each in detail.


Rivest–Shamir–Adleman (RSA) is a public-key cryptosystem named after its inventors Ron Rivest, Adi Shamir, and Leonard Adleman.

A public-key cryptosystem has one public encryption key and one‌ ‌private decryption‌ ‌key. Two large prime numbers and an auxiliary value are used to create the RSA public keys. 

Let’s say John wants to send a message to Clara. To encrypt the messages with RSA, John must know Clara's public key, and likewise, to decrypt them, Clara must use her private key. Clara sends her public key to John over a secure pathway so he can reliably send his encrypted messages. Remember, Clara's private key is never disclosed.

Elliptic curve digital signature algorithm

The ECDSA algorithm is a variant of the digital signature algorithm (DSA), a federal information processing standard for digital signatures. 

Technically, the ECDSA algorithm relies on the algebraic structure of elliptic curves over finite fields. Elliptic curves may be used for key agreement, digital signatures, or pseudo-random generators, among other things.

For elliptic curve cryptography to work, all participating devices must possess a private and public key pair. A message is signed with a private key by its sender, and the recipient uses the sender's public key to verify its authenticity.

Messages that have been altered on route to the recipient will not pass the signature verification test, as the signature only applies to the original message.

Pre-shared key 

In cryptography, a pre-shared key (PSK) is a shared secret that two parties have previously shared through a secure channel. 

Wi-Fi encryption standards, including wired equivalent privacy (WEP), Wi-Fi protected access (WPA) and the extensible authentication protocol (EAP) use PSK in their encryption methods.  The wireless access points (AP) and the clients share the same authentication key.

A PSK may be a password, a passphrase, or a hexadecimal string. 

IPsec versus SSL: What’s the difference?

The‌ ‌IPsec‌ ‌protocol‌ ‌suite‌ ‌operates‌ ‌at‌ ‌the‌ ‌network‌ ‌layer‌ ‌in‌ ‌the‌ open systems interconnection

(‌OSI‌) ‌model. The secure sockets layer (SSL), on the other hand, operates at the application layer of the OSI model.  Each serves a very different purpose: IPsec encrypts IP packers, while SSL encrypts HTTP traffic and web-based transactions.

Featured Resources

Three ways manual coding is killing your business productivity

...and how you can fix it

Free Download

Goodbye broadcasts, hello conversations

Drive conversations across the funnel with the WhatsApp Business Platform

Free Download

Winning with multi-cloud

How to drive a competitive advantage and overcome data integration challenges

Free Download

Talking to a business should feel like messaging a friend

Managing customer conversations at scale with the WhatsApp Business Platform

Free Download


WatchGuard finds malware volume decreased in Q2, but warns Emotet is resurging

WatchGuard finds malware volume decreased in Q2, but warns Emotet is resurging

28 Sep 2022
What is subnetting?
Network & Internet

What is subnetting?

28 Jul 2022
Static IP vs dynamic IP: What’s the difference?
Network & Internet

Static IP vs dynamic IP: What’s the difference?

27 Jul 2022
Server virtualisation: What is it and what are the benefits?

Server virtualisation: What is it and what are the benefits?

25 Jul 2022

Most Popular

Vodafone UK confirms talks to merge with Three are underway
mergers and acquisitions

Vodafone UK confirms talks to merge with Three are underway

3 Oct 2022
BT's new platform promises to slash AI development time from months to days
artificial intelligence (AI)

BT's new platform promises to slash AI development time from months to days

3 Oct 2022
How to secure your hybrid workforce
Advertisement Feature

How to secure your hybrid workforce

23 Sep 2022