IPsec (IP security) is a suite of protocols for securing Internet Protocol (IP) communications by authenticating and/or encrypting each IP packet in a data stream.

It operates at the network layer (OSI layer 3). Other internet security protocols in widespread use (e.g. SSL, TLS, SSH), operate from the transport layer up (OSI layers 4 - 7). This makes IPsec more flexible, as it can be used for protecting layer 4 protocols, including both TCP and UDP, the most commonly used transport layer protocols. IPsec has an advantage over SSL and other methods that operate at higher layers: it is transparent to end user and applications.

In order to decide what protection is to be provided for an outgoing packet, IPsec uses the Security Parameter Index (SPI), an index to the security association database (SADB), along with the destination address in a packet header, which together uniquely identify a security association for that packet.

There are two modes of IPsec operation:

Transport mode

• only the payload (the data you transfer) of the IP packet is encrypted and/or authenticated
• routing is intact, since the IP header is neither modified nor encrypted
• transport and application layers are always secured by hash, so they cannot be modified in any way
• transport mode is used for host-to-host communications

Tunnel mode

• entire IP packet (data plus the message headers) is encrypted and/or authenticated
• encapsulated into a new IP packet for routing to work
• used for network-to-network communications (secure tunnels between routers, e.g. for VPNs) or host-to-network and host-to-host communications over the internet

Two protocols have been developed to provide packet-level security for both IPv4 and IPv6:

Authentication header (AH)

• intended to guarantee connectionless integrity and data origin authentication of IP datagrams
• AH protects the IP payload and all header fields of an IP datagram except for mutable fields, i.e. those that might be altered in transit
• authentication data contains the integrity check value (ICV) necessary to authenticate the packet
• Hashed Message Authentication Code (HMAC) - incorporates a secret value while creating the ICV (attacker can easily recompute a hash, without the secret value he won't be able to recreate the proper ICV)

Encapsulating Security Payload (ESP)

• provides origin authenticity, integrity, and confidentiality protection of a packet
• supports encryption-only and authentication-only configurations, but using encryption without authentication is strongly discouraged because it is insecure
• unlike AH, the IP packet header is not protected by ESP

Security Association (SA)

• a set of security information that describes a particular kind of secure connection between one device and another.
• "contract" that specifies the particular security mechanisms that are used for secure one-way communication
• each SA only handles either inbound or outbound traffic for a particular device
• this allows different levels of security to be implemented for a flow from device A to device B, than from B to A
• in a bidirectional communication of this sort, both A and B would have two SAs
  A would have SAs we could call "SAdeviceBin" and "SAdeviceBout". Device B would have SAs "SAdeviceAin" and "SAdeviceAout"

Each SA is uniquely identified by following parameters:

• Security Parameter Index (SPI)
  • 32-bit number that is chosen to uniquely identify a particular SA
  • SPI is placed in AH or ESP datagrams and thus links each secure datagram to the security association
• IP Destination Address
  • address of the device for whom the SA is established
• Security Protocol Identifier
  • AH or ESP - if both are in use with device they have separate SAs

Operational details of each SA are defined by following parameters:

• Sequence Number Counter (SNC)
  • 32-bit value used to generate sequence number in AH and EPS headers
  • by one incremented number assigned to each packet to prevent replays and used for proper ordering
• Sequence Counter Overflof
  • flag that indicates overflow of Sequence number counter as only 32 bits are assigned for SNC
• Anti-reply window
  • defines interval of incoming packets that are still under processing
  • prevents replays
• AH data
  • determine authentication algorithm, mode, keys, lifetime and other parameters required by AH
• ESP data
  • determine encryption algorithm, authentication algorithm, mode, keys and their lifetimes and other parameters required by ESP
• SA lifetime
  • time interval after SA is terminated
• IPSec protocol mode
  • tunnel, transport and wild-card mode (either transport or tunnel mode)
• Path MTU
  • maximum packet size that can be send without fragmentation

A device's security associations SAs are contained in its Security Association Database (SADB).

A security policy is a rule that is programmed into the IPSec implementation that tells it how to process different datagrams received by the device. Security policies for a device are stored in the device's Security Policy Database (SPD).

It's often hard to distinguish the SPD and the SAD, since they are similar in concept. The main difference between them is that security policies are general while security associations are more specific. To determine what to do with a particular datagram, a device first checks the SPD. The security policies in the SPD may reference a particular security association in the SAD. If so, the device will look up that security association and use it for processing the datagram.

more on tcpipguide
more on illustrated guide to ipsec
more on wikipedia
@FRI students: more on slides 93-119