Before talking about SYN cookies and how they are used to preventing SYN Flood attack, Let us first take a look at how TCP connections were established until  mid-1990s.

How TCP Connection Are Established:
A TCB(Transmission Control Block) is created when a TCP entity opens a TCP connection, A TCB contains  whole state of connection. The state of the connection contains:

1. Local sequence number.
2. Sequence number sent by the remote client.

Until the mid-1990s, To avoid overflowing the entity’s memory with TCBs there was a limit on the number of ‘half-open’ TCP connections (TCP connections in the SYN RCVD state) which was most commonly at 100. So, a server could only have 100 ‘half-open’ TCP connections. The TCP entity would stop accepting any new SYN segments when limit was reached.

A TCP implementation must maintain a Transmission Control Block (TCB) for every established TCP connection. A TCB must contain all information required to send and receive segments. They are as follows:

1. Local IP address.
2. Remote IP address.
3. Local TCP port number.
4. Remote TCP port number.
5. Current state of the TCP FSM.
6. Maximum segment size (MSS).

The following diagrams show the TCP connection process:

STEP 1:  Client sends an SYN connection request to server 

Initially, the Client sends an SYN connection request

STEP 2: Server sends an SYN connection request and an acknowledgement to client

The server sends an SYN connection request and an acknowledgement

STEP 3:  Client sends an acknowledgement to server

The client sends an acknowledgement

SYN Flood Attack:
An SYN flood attack is a type of denial-of-service attack during which an attacker rapidly initiates a TCP connection with an SYN request to a server and does not respond to SYN+ACK from the server. The server has to spend resources (Creating TCBs for the connection requests) waiting for half-opened connections Since there was a limit on the number of ‘half-open’ TCP connections. The server will no longer accept any new connections. This will make system unresponsive to legitimate traffic. The following steps show how it was carried out.

1. Attacker would send 100s of SYN segments every second to a server.
2. Attacker would not reply to any received SYN+ACK segments.
3. Attacker would send these SYN segments with a different IP address from their own IP address to avoid being caught.
4. Once a server entered the SYN RCVD state, it would remain in that state for several seconds, waiting for an ACK and not accepting any new, possibly genuine connections, thus being rendered unavailable.

Here are some diagrams depicting an SYN flood attack:

STEP 1:  Client sends an SYN connection request to server 

Initially, the Client sends an SYN connection request

STEP 2: Server sends an SYN connection request and an acknowledgement to the client

The server sends an SYN connection request and an acknowledgement

STEP 3:  Client does not respond with an ACK to complete three-way handshake

The client does not respond with an ACK

STEP 4: Attacker creates 100 other half-open connections from various IP addresses

The attacker creates 100 other half-open connections from various IP addresses

STEP 5:  TCP buffer will be full at server’s end and server will be unable to accept any new connections

The TCP buffer will be full at the server’s end

SYN flood attacks can be performed in three different ways:

1. Direct attack-
   A SYN flood attack where IP address of the attacker is not spoofed is called a direct attack. In a direct attack, the attacker uses a single source device with a real IP address, therefore, the attacker can be traced easily and the requests from IP address of the malicious system can be blocked to prevent the attack.
2. Spoofed attack-
   A SYN flood attack where IP address of attacker is spoofed on each SYN packet is called a spoofed attack. Even though the IP address is spoofed on each packet they can be traced back to their source with help of the Internet service providers (ISPs).
3. Distributed attack-
   A SYN flood attack created using botnet is called distributed attack. The chances of tracing these attacks to source are extremely low. The attacker may also spoof the IP address of each distributed device to make it more difficult to trace.

SYN Flood Attack Prevention:
SYN Flood attacks can be prevented in a number of different ways. Some approaches include:

1. Increasing Backlog queue-
   There is a limit on the number of half-open connections on each operating system on a targeted device. One way to handle high volume of SYN packets is to increase the maximum number of half-open connections which will be allowed by the operating system. To increase the maximum backlog, the system must reserve additional memory resources which can handle all the new requests. If the system does not have sufficient memory to handle  increased backlog queue size, the performance of the system will be affected, but it will be better than denial-of-service.
2. Recycling the Oldest Half-Open TCP connection-
   In this strategy, the oldest half-open connection is overwritten once the backlog is filled. This strategy works only when connections can be fully established in less time than the backlog can be filled with malicious SYN packets. It fails when volume of attack is increased or if the backlog size is too small.
3. SYN Cookies-
   SYN cookies is an IP Spoofing attack mitigation technique whereby server replies to TCP SYN requests with crafted SYN-ACKs, without creating a new TCB for the TCP connection. A TCB is created for the respective TCP connection only when the client replies to this crafted response. This technique is used to protect the server’s resources from filling up under TCP SYN floods.

How SYN Cookies are used to preventing SYN Flood attack:
The way SYN Cookies solves this problem(SYN Flood attack) is to use a function that uses some information from the client’s SYN packet and some information from server-side to calculate a random initial sequence number. Let us assume this number as y-1, y-1 is sent to the client in an SYN + ACK message. If an ACK packet is received with a sequence number y, with the help of some packet header fields and some server-side information, a reverse function can verify that acknowledgement number is valid. If it is valid, a TCB is created and a connection is established. If it is invalid, the connection is refused. The advantage of SYN cookies is that the server doesn’t have to create and store a TCB upon reception of the SYN segment.