A Guide to HAProxy Access and Error Logs

HAProxy is a widely-used open-source load balancer and reverse proxy that excels in managing high-volume traffic for web applications. Due to its efficiency and scalability, it’s often deployed in production environments to ensure optimal performance and reliability. To fully leverage HAProxy, understanding its logging system is essential for monitoring server performance, diagnosing issues, and enhancing security.

HAProxy maintains two main types of logs:

1. Access Logs – Capture details of incoming client requests.
2. Error Logs – Record server-side errors and issues related to backend connections.

This guide explores the structure, configuration, and practical use of both types of logs in HAProxy.

---

Log & Configuration File Locations

Before diving into the details, it’s essential to understand where HAProxy stores its logs and configurations. HAProxy logs are typically managed by the system’s syslog daemon, such as rsyslog or syslog-ng.

Description	Location
HAProxy Configuration	/etc/haproxy/haproxy.cfg
Access & Error Logs	/var/log/haproxy.log
Syslog Configuration	/etc/rsyslog.conf or /etc/syslog-ng/syslog-ng.conf

Understanding HAProxy Access Logs

What Are Access Logs?

Access logs capture details of all incoming HTTP requests that pass through HAProxy. They are invaluable for analyzing traffic patterns, understanding client behavior, and identifying potential security threats.

Configuring Access Logs

By default, HAProxy logs both access and error messages to the system’s syslog service. To enable logging in HAProxy, you need to configure the haproxy.cfg file:

1 – Enable Logging in the Global Section
Add the following lines to the global section:

global
    log /dev/log local0
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy

2 – Configure Logging for Frontend and Backend
Add the following lines in your frontend and backend sections to enable access logging:

frontend http_front
    bind *:80
    log global
    option httplog
    default_backend http_back

backend http_back
    server web1 192.168.1.100:80 check
    server web2 192.168.1.101:80 check

Access Log Format

HAProxy uses a predefined format for its access logs, known as httplog. This format captures critical information, such as:

<syslog_timestamp> <haproxy_hostname> haproxy[pid]: <client_ip>:<client_port> [<accept_date>] <frontend_name> <backend_name>/<server_name> <Tq> <Tw> <Tc> <Tr> <Tt> <status_code> <bytes_read> <captured_request_headers> <captured_response_headers> <termination_state> <actconn> <feconn> <beconn> <srvconn> <retries> <srv_queue> <backend_queue>

Sample Access Log Entry

Here’s an example of an HAProxy access log entry:

Nov 14 10:15:23 haproxy[1234]: 192.168.1.10:51560 [14/Nov/2024:10:15:23.123] http_front http_back/web1 0/0/5/15/20 200 854 - - ---- 3/3/1/0/0 0/0

Explanation of the Fields:

Field	Description
<client_ip>:<client_port>	IP and port of the client making the request
<accept_date>	Date and time when the request was accepted
<frontend_name>	The frontend that received the request
<backend_name>/<server_name>	The backend and specific server handling the request
<Tq>	Time spent waiting in the queue (ms)
<Tw>	Time spent waiting for a connection to the backend (ms)
<Tc>	Time taken to establish a TCP connection to the server (ms)
<Tr>	Time to receive the full HTTP response (ms)
<Tt>	Total time taken to process the request (ms)
<status_code>	HTTP status code returned by the server
<bytes_read>	Total bytes sent to the client

---

Understanding HAProxy Error Logs

What Are Error Logs?

Error logs capture issues related to server connections, backend health checks, and other operational problems within HAProxy. These logs are crucial for identifying configuration errors, diagnosing server issues, and ensuring high availability.

Configuring Error Logs

Error logs are also managed by the system’s syslog service. To ensure error logs are captured, include the following in your haproxy.cfg file:

global
    log /dev/log local1 err

defaults
    log global
    option httplog
    option dontlognull
    option log-separate-errors

Error Log Levels

HAProxy supports several log levels:

Level	Description
emerg	Emergency – system is unusable
alert	Immediate action required
crit	Critical conditions
err	General errors
warning	Warning conditions
notice	Normal but significant
info	Informational messages
debug	Debugging messages

Sample Error Log Entry

An example of an HAProxy error log entry:

Nov 14 10:16:45 haproxy[1234]: Server http_back/web1 is DOWN, reason: Layer4 timeout, check duration: 2000ms, status: 503 Service Unavailable

Explanation: The log indicates that the server web1 in the http_back backend is down due to a timeout, resulting in an HTTP 503 status.

Common Error Types

Some typical HAProxy error messages include:

Error	Description
Layer4 timeout	Timeout while trying to establish a TCP connection
Layer7 wrong status	Unexpected HTTP status code received from the server
Connection refused	Backend server is not accepting connections
503 Service Unavailable	Backend server is not reachable or overloaded

---

Configuring Logs for Specific Frontends and Backends

If you are using multiple frontends and backends, it’s beneficial to separate logs for each service. This can be done using log directives within your HAProxy configuration:

frontend http_front
    bind *:80
    log global
    option httplog
    default_backend http_back

backend http_back
    log global
    server web1 192.168.1.100:80 check
    server web2 192.168.1.101:80 check

To filter logs by severity:

frontend secure_front
    bind *:443 ssl crt /etc/ssl/certs/mycert.pem
    log global info
    error-log global err

---

Mastering HAProxy logs is crucial for maintaining a high-performance and reliable infrastructure. By effectively configuring and analyzing access and error logs, you can gain insights into traffic behavior, optimize server performance, and quickly diagnose issues.

Use this guide to optimize your HAProxy logging setup and enhance your server management skills.