Original submitting IP for syslog (514/udp) - lost or just hard to find?

[gowen]

We have multiple F5 appliances submitting their LTM logs via syslog (514/udp). The logs always have the following line format, with "tmm" or "tmm1" being a generic F5 thing rather than the actual hostname:

Oct 19 15:07:47 tmm1 info tmm1[27478]: ...

Splunk is extracting that first 'tmm1' as the hostname and setting host=tmm1. The problem is, since we have multiple F5s submitting logs, they're all grouped as host=tmm1.

1) I don't see the original IP that submitted the syslog entry stored anywhere, which amazes me, because it implies someone could spoof messages and get them rewritten to belong to another host quite easily. Please tell me where I'm missing the original submitting IP?

2) If I can figure out the original submitting IP*, I can use props and transforms to set the host, but is there a way to simply tell Splunk that the submitting IP is the host and not to try and suss it out from the log entries?

• without knowing the answer to #1, I don't actually know the original submitting IP at this time. The F5 has a dozen IP addresses associated with it, and I don't know which one it is using to send the logs over - it is not, of course, the "main" address that the web GUI sits on.

[jgedeon120]

Instead of using Splunk to listen to the port set up syslog-ng to receive the messages and then splunk read the log directory. You can have syslog-ng write to a location like /var/log/F5/$HOST/syslog-ng.log then define in props.conf host_segment = 4. The added benefit to this is when you need to restart Splunk you aren't missing any logs.

There is plenty of documentation on setting this up and will also find that it is the recommended way to receive syslog to be processed by Splunk.

[kristian_kolb]

Very good answer from sowings - just wanted to clarify that these transforms that are mentioned, are most likely applied automatically. If your sourcetype is 'syslog' that is definitely the case. To avoid this, change the sourcetype name to something different, e.g. 'syslog-f5'.

If needed set up a separate listening port on the splunk server, specifically for the F5 traffic.

If you wish, have a look in /opt/splunk/etc/system/default/props.conf and transforms.conf to see if there are any transformations occuring for your data.

Hope this helps,

Kristian

[gowen]

Can I enter multiple REGEX lines in one stanza, or do I need to adjust the existing line to have a REGEX 'THIS' or 'THAT'? Current entry is "REGEX = ^host::1.2.3.4$", what would be the correct OR, "REGEX = ^host::1.2.3.4|host::1.2.3.5$"?

[gowen]

This seems promising. At least one of our F5s is transformed to SourceType "f5", and the host IP is retained for log messages from that host. I will rewrite this additional F5 to that SourceType and check back in.

[sowings]

You may find that a transform called syslog-host or syslog-host-null is being applied to events arriving on UDP/514. This transform remaps the host field based upon the contents of a syslog-type line, which contains the hostname in the event itself. You can check the connection_host attribute in inputs.conf to specify that the source IP should be kept as the host value. You'll also want to make sure that the syslog hostname transforms listed above aren't being applied to your sourcetype.

[gowen]

There is no such transform - there are syslog transforms that adjust SourceType based on source address, but none adjusting Host. There is no setting for connection_host in inputs.conf, so it should use the default of 'ip'