What are authentication protocols?

An authentication protocol is a method used by a server to verify the authenticity of a request, typically ensuring that the user is valid. There are many different authentication protocols; however, in this blog post, we will focus on New Technology LAN Manager (NTLM) authentication and Kerberos authentication. These two network authentication protocols both serve to verify the authenticity of users but operate in different ways to perform the authorization process.

How does NTLM and Kerberos authentication work?

The NTLM protocol is simpler than Kerberos in how it verifies a user. The figure below illustrates how this authentication protocol works.

  1. The user and server will enter the negotiation phase, during which the user will request authentication from the server. In this phase, the user will first hash the password and then send an NTLM authentication request to the server, concluding the negotiation phase of the protocol.
  2. When the server receives the authentication request, the process enters the challenge phase. During this phase, the server generates a random 16-byte string and sends it to the user, concluding the challenge phase.
  3. After the server sends the random string to the host device, the authentication phase begins. The host device then encrypts the random string using the hash of its password. Once encrypted, the host device sends the result to the server.
  4. The server will then encrypt the challenge using the user’s stored password and compare it with the encrypted response from the host device. If they match, the server will grant access; otherwise, it will deny access to the host device.

The Kerberos authentication protocol is more complex than NTLM, involving many additional steps. It relies on the Key Distribution Center (KDC), a domain service that provides two key functions: the authentication service and the ticket-granting service. Together, these services form the framework for authenticating users within the system. The figure below outlines the steps of the authentication process.

  1. The user will make an authentication request on the authentication server within the Key Distribution Center (KDC). Once the request is made, the authentication server will then verify the user in a database within the KDC.
  2. When the user is verified, the authentication server will provide the user with a ticket-granting ticket (TGT), which will act as an authentication token for the next step of the authentication process.
  3. The user receives the TGT. The host device will then ask for a service ticket from the ticket granting service (TGS) by providing the TGT to the TGS. The TGS is a service that issues service tickets to users that wish to connect to other devices on a domain. The service ticket will be used to establish a connection between the user and service.
  4. Once the TGS receives the TGT, it will verify that you are an authorized user and send the service ticket to the host device to allow it to access the desired service on the server.
  5. The TGS will then provide a service ticket to host device. The host device can now send a request to gain access to the server. The host device will send the request as well as the service ticket to the server.
  6. The server receives the request and service ticket from the host device. The server will then send the host device that they are authorized to access the server.

Which is better?

NTLM is an outdated protocol that has been replaced by Kerberos. However, NTLM is still used as a fallback protocol if Kerberos fails during the authentication process. The key difference between the two protocols lies in how they authenticate a user on a system. NTLM relies on a challenge-response handshake, making it vulnerable to NTLM relay attacks. In an NTLM relay attack, an attacker intercepts the challenge-response handshake between a device and the server, allowing them to gain access as if they were the legitimate device. This type of attack is a man-in-the-middle attack, where the attacker positions themselves between the user and the server, intercepting the user’s request and forwarding it to the server. The attacker then pretends to be the user, obtaining information from the server as shown below.

However, Kerberos does not have this problem. The Kerberos authentication process was designed for untrusted networks, requiring mutual authentication between users. As a result, man-in-the-middle attacks are prevented. Kerberos is a more secure protocol that allows users to authenticate with other devices with minimal risk.