Troubleshooting Remote Syslog with TCPDUMP: Inspecting and Viewing Content

Forwarding logs to a remote syslog server is a convenient way to centralize log management. It’s agentless and natively supported by most operating systems. If you’re using syslogd or rsyslog, enabling remote logging is as simple as editing the configuration file (/etc/rsyslog.conf or /etc/syslog.conf) and adding a line like this:

*.* @SYSLOGSERVERIP:PORT

This configuration will forward all logs to the specified SYSLOGSERVERIP. But what if your logs aren’t showing up on the remote server? Let’s look at how to use tcpdump to troubleshoot.

Inspecting Syslog Traffic with tcpdump

Syslog uses clear text, making it ideal for packet inspection with tcpdump. To start troubleshooting, you can use tcpdump to check if logs are being sent to the remote server. Here’s a simple command to filter UDP traffic to your syslog server:

$ sudo tcpdump -nnn udp and host y.y.18.158

This listens for UDP traffic on your network interface, specifically targeting the syslog server at y.y.18.158 (port 514 by default). Sample output might look like:

listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
17:54:01.479811 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
17:54:01.479977 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 90

In this example, logs are being sent from x.x.40.29 to the remote syslog server y.y.18.158 on port 514.

Viewing Log Content with tcpdump

To inspect the actual content of the syslog messages, add the -A flag to display the packet data in ASCII format:

$ sudo tcpdump -A -nnn udp and host y.y.18.158

This command provides deeper insight into what’s being sent. Here’s an example output:

listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
18:00:01.708077 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
E.....@.@...k.(......E...wv.<86>May 17 18:00:01 testserver CRON[19937]: pam_unix(cron:session): session opened for user nginx by (uid=0)
18:00:01.711938 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
E.....@.@...k.(......E...wv.<86>May 17 18:00:01 testserver CRON[19939]: pam_unix(cron:session): session opened for user nginx by (uid=0)

In the output above, you can see that the syslog message includes details from a cron job executed on testserver.

Troubleshooting Network Issues

Just because the logs are being sent from the client doesn’t mean they’re being received by the syslog server. There might be firewalls or network policies blocking the traffic. To verify, you can run the same tcpdump command on the syslog server to check if the packets are arriving:

$ sudo tcpdump -nnn udp port 514

By inspecting both ends (the client and the server), you can identify whether the issue is related to the client configuration, network firewall rules, or the syslog server itself. We provide more TCPDUMP recommends on our Troubleshooting Remote Syslog with TCPDUMP article.