IP Security Protocol
IPsec is a pair of protocols, Encapsulating Security Payload (ESP) and
Authentication Header (AH), which provide security services for IP datagrams.
Both protocols may be enabled or disabled using the following
sysctl(3) variables in
/etc/sysctl.conf. By default, both protocols are
- Enable the ESP IPsec protocol
- Enable the AH IPsec protocol
There are four main security properties provided by IPsec:
IPsec provides these services using two new protocols: Authentication Header
(AH), and Encapsulating Security Payload (ESP).
- - Ensure it is hard for anyone but the receiver to understand what data
has been communicated. For example, ensuring the secrecy of passwords when
logging into a remote machine over the Internet.
- - Guarantee that the data does not get changed in transit. If you are on a
line carrying invoicing data you probably want to know that the amounts
and account numbers are correct and have not been modified by a third
- - Sign your data so that others can see that it is really you that sent
it. It is clearly nice to know that documents are not forged.
- Replay protection
- - We need ways to ensure a datagram is processed only once, regardless of
how many times it is received. That is, it should not be possible for an
attacker to record a transaction (such as a bank account withdrawal), and
then by replaying it verbatim cause the peer to think a new message
(withdrawal request) had been received. WARNING: as per the standard's
specification, replay protection is not performed when using manual-keyed
IPsec (e.g. when using
ESP can provide the properties authentication, integrity, replay
protection, and confidentiality of the data (it secures everything in the
packet that follows the IP header). Replay protection requires
authentication and integrity (these two always go together). Confidentiality
(encryption) can be used with or without authentication/integrity.
Similarly, one could use authentication/integrity with or without
AH provides authentication, integrity, and replay protection (but
not confidentiality). The main difference between the authentication
features of AH and ESP is that AH also authenticates portions of the IP
header of the packet (such as the source/destination addresses). ESP
authenticates only the packet payload.
AH works by computing a value that depends on all of the payload data, some of
the IP header data, and a certain secret value (the authentication key). This
value is then sent with the rest of each packet. The receiver performs the
same computation, and if the value matches, he knows no one tampered with the
data (integrity), the address information (authenticity) or a sequence number
(replay protection). He knows this because the secret authentication key makes
sure no active attacker (man-in-the-middle) can recompute the correct value
after altering the packet. The algorithms used to compute these values are
called hash algorithms and are parameters in the SA, just like the
ESP optionally does almost everything that AH does except that it does not
protect the outer IP header but furthermore it encrypts the payload data with
an encryption algorithm using a secret encryption key. Only the ones knowing
this key can decrypt the data, thus providing confidentiality. Both the
algorithm and the encryption key are parameters of the SA.
These protocols require certain parameters for each connection, describing
exactly how the desired protection will be achieved. These parameters are
collected in an entity called a security association, or SA for short. Typical
SA parameters include encryption algorithm, hash algorithm, encryption key,
and authentication key, to name a few. When two peers have established
matching SAs (one at each end), packets protected with one end's SA may be
verified and/or decrypted using the information in the other end's SA. The
only issue remaining is to ensure that both ends have matching SAs. This may
be done manually, or automatically using a key management daemon.
Further information on manual SA establishment is described in
Information on automated key management for IKEv1 can be found in
isakmpd(8) and for IKEv2 in
In order to identify an SA we need to have a unique name for it. This name is a
triplet, consisting of the destination address, security parameter index (aka
SPI) and the security protocol (ESP or AH). Since the destination address is
part of the name, an SA is necessarily a unidirectional construct. For a
bidirectional communication channel, two SAs are required, one outgoing and
one incoming, where the destination address is our local IP address. The SPI
is just a number that helps us make the name unique; it can be arbitrarily
chosen in the range 0x100 - 0xffffffff. The security protocol number should be
50 for ESP and 51 for AH, as these are the protocol numbers assigned by IANA.
IPsec can operate in two modes, either tunnel or transport mode. In transport
mode the ordinary IP header is used to deliver the packets to their endpoint;
in tunnel mode the ordinary IP header just tells us the address of a security
gateway which knows how to verify/decrypt the payload and forward the packet
to a destination given by another IP header contained in the protected
payload. Tunnel mode can be used for establishing virtual private networks
(VPNs), where parts of the networks can be spread out over an unsafe public
network, but security gateways at each subnet are responsible for encrypting
and decrypting the data passing over the public net. An SA will contain
information specifying whether it is a tunnel or transport mode SA, and for
tunnels it will contain values to fill in into the outer IP header.
The SA also holds a couple of other parameters, especially useful for automatic
keying, called lifetimes, which puts a limit on how much we can use an SA for
protecting our data. These limits can be in wall-clock time or in volume of
To better illustrate how IPsec works, consider a typical TCP packet:
[IP header] [TCP header]
If we apply ESP in transport mode to the above packet, we will
[IP header] [ESP header] [TCP header]
Everything after the ESP header is protected by whatever services
of ESP we are using (authentication/integrity, replay protection,
confidentiality). This means the IP header itself is not protected.
If we apply ESP in tunnel mode to the original packet, we would
[IP header] [ESP header] [IP header]
[TCP header] [data...]
Again, everything after the ESP header is cryptographically
protected. Notice the insertion of an IP header between the ESP and TCP
header. This mode of operation allows us to hide who the true source and
destination addresses of a packet are (since the protected and the
unprotected IP headers don't have to be exactly the same). A typical
application of this is in Virtual Private Networks (or VPNs), where two
firewalls use IPsec to secure the traffic of all the hosts behind them. For
Net A <----> Firewall 1 <--- Internet ---> Firewall 2 <----> Net B
Firewall 1 and Firewall 2 can protect all communications between
Net A and Net B by using IPsec in tunnel mode, as illustrated above.
This implementation makes use of a virtual interface,
enc0, which can be used in packet filters to specify
those packets that have been or will be processed by IPsec.
NAT can also be applied to
interfaces, but special care should be taken because of the interactions
between NAT and the IPsec flow matching, especially on the packet output
path. Inside the TCP/IP stack, packets go through the following stages:
UL/R -> [X] -> PF/NAT(enc0) -> IPsec -> PF/NAT(IF) -> IF
UL/R <-------- PF/NAT(enc0) <- IPsec <- PF/NAT(IF) <- IF
With IF being the real interface and UL/R the Upper Layer or
Routing code. The [X] stage on the output path represents the point where
the packet is matched against the IPsec flow database (SPD) to determine if
and how the packet has to be IPsec-processed. If, at this point, it is
determined that the packet should be IPsec-processed, it is processed by the
PF/NAT code. Unless PF drops the packet, it will then be IPsec-processed,
even if the packet has been modified by NAT.
Security Associations can be set up manually with
automatically with the
iked(8) key management
A number of sysctl(8) variables
are relevant to
ipsec. These are generally
net.inet.ip.ipsec-*. Full explanations can be found in
sysctl(3), and variables can be
set using the sysctl(8)
A number of kernel options are also relevant to
The following IP-level
getsockopt(2) options are
options(4) for further
ipsec. A socket can specify security
levels for three different categories:
- Specifies the use of authentication for packets sent or received by the
- Specifies the use of encryption in transport mode for packets sent or
received by the socket.
- Specifies the use of encryption in tunnel mode.
For each of the categories there are five possible levels which
specify the security policy to use in that category:
- Bypass the default system security policy. This option can only be used by
privileged processes. This level is necessary for the key management
- If a Security Association is available it will be used for sending packets
by that socket.
- Use IP Security for sending packets but still accept packets which are not
- Use IP Security for sending packets and also require IP Security for
- The outbound Security Association will only be used by this socket.
When a new socket is created, it is assigned the default system
security level in each category. These levels can be queried with
getsockopt(2). Only a
privileged process can lower the security level with a
For example, a server process might want to accept only
authenticated connections to prevent session hijacking. It would issue the
int level = IPSEC_LEVEL_REQUIRE;
error = setsockopt(s, IPPROTO_IP, IP_AUTH_LEVEL, &level, sizeof(int));
The system does guarantee that it will succeed at establishing the
required security associations. In any case a properly configured key
management daemon is required which listens to messages from the kernel.
A list of all security associations in the kernel tables can be
obtained using the
A socket operation may fail with one of the following errors returned:
- An attempt was made to lower the security level below the system default
by a non-privileged process.
- The length of option field did not match or an unknown security level was
netstat(1) can be
used to obtain some statistics about AH and ESP usage, using the
-p flag. Using the
information about IPsec flows.
information about memory use by IPsec with the
IPsec was originally designed to provide security services for Internet Protocol
IPv6. It has since been engineered to provide those services for the original
Internet Protocol, IPv4.
flag (look for “tdb” and “xform”
The IPsec protocol design process was started in 1992 by John
Ioannidis, Phil Karn, and William Allen Simpson. In 1995, the former wrote
an implementation for BSD/OS. Angelos D. Keromytis
ported it to OpenBSD and
NetBSD. The latest transforms and new features were
implemented by Angelos D. Keromytis and Niels Provos.
The authors of the IPsec code proper are John Ioannidis,
Angelos D. Keromytis, and Niels
Niklas Hallqvist and
Niels Provos are the authors of
Eric Young's libdeslite was used in this
implementation for the DES algorithm.
Steve Reid's SHA-1 code was also used.
interface follows somewhat loosely the draft-mcdonald-simple-ipsec-api
(since expired, but still available from
There's a lot more to be said on this subject. This is just a beginning. At the
moment the socket options are not fully implemented.