We are in the midst of implementing Splunk to handle syslog from all of our network devices. I've configured rsyslog to write the logs to a YYYY/MM/DD directory, in a "system-hostname.log" format.
Does anyone have a suggestion on how to handle all of these formats (a dozen+), and allow engineering to add new device types, without intervention on the Splunk side?
What do you mean by "without intervention on the Splunk side"? Also, what are the differences in the formats?
syslog generally defines a generic format which Splunk understands; if all of the logs follow this format, then the
syslog sourcetype should work fine.
Currently, the props files has entries for each device model - netscreens, checkpoint, f5.... I'd like to come up with a way to let engineering route the messages to splunk, and not be required (at least not initially) to add some code to process the files. We don't use a generic syslog sourcetype, as it's not detailed enough. So, we have a f5syslog, netscreensyslog... which actually comes in handy, since there are a number of extracts that only pertain to certain models.
I would re-engineer the rsyslog configuration so that you have a directory structure like
etc. You'd have to set up the device-type subdirs manually and make sure that logs from a certain IP gets written to the correct dir. I believe that you can use the rsyslog variables
$fromhostip to create the hostname directories automatically, but you'll have to maintain the mapping of hostname (or IP) to device-type in the rsyslog conf. However, this is probably fairly static, and can be set up regardless of whether the devices are actually sending any logs (yet).
Then you can have a fairly 'static' Splunk config like so;
[monitor:///var/log/netscreen] host_segment = 4 index = your_index sourcetype = netscreen_syslog ignoreOlderThan = 7d [monitor:///var/log/f5] host_segment = 4 index = your_index sourcetype = f5_syslog ignoreOlderThan = 7d
Hope this helps,