I am using syslog on Splunk currently to capture data from a piece of content-keeper hardware on our network that handles internet usage analysis. It's a fairly straight forward input, but, now I wish to capture more data from a firewall again using the standard UDP:514.
However, I would like for my CK data to go into splunk with the following values:
Host: content-keeper Index: internet
And then my firewall to go in as:
Host: foo Index: firewalls
Currently my inputs.conf segment looks like:
[udp://514] connection_host = none host = content-keeper index = internet sourcetype = content-keeper source = content-keeper
[content-keeper] LOOKUP-domaingroups = common_domains basedomain AS basedomain OUTPUTNEW domaingroup AS domaingroup
[common_domains] filename = group_common_domains.csv
As you can see with props and transforms, I am just doing some basic lookups.
Any help would be very very appreciated.
What you're looking for is documented here:
What you'd need to do would be something like this:
[content-keeper] TRANSFORMS-index = firewalls,hosts
[firewalls] REGEX = <MYFIREWALLREGEX> DEST_KEY = _MetaData:Index FORMAT = firewalls [hosts] REGEX = <(MYFWHOSTREGEX)> DEST_KEY = MetaData:Host FORMAT = host::$1
By the way, at the time you do this, the 'firewalls' index must have already been created.
This is pretty much what I have come up with. The issue I am encountering now is with the regex, I cannot seem to find a way of extracting the hostname or some indicator of the orgin host. When I removed the "host=" over-ride from my inputs.conf, the host value simply defaulted to the fqdn of the splunk indexer...
I am going to simply have to look more into it.
Thanks very very much for you well written response!
Solved with a simple regex that can determine which data to perform the change on and which to not perform changes on.
Create multiple DNS entries, assign them to virtual hosts on your system, configure syslog to look for UDP/514 from each of those IPs and write the data to disk, then configure multiple Splunk imports to look at each of the paths syslog writes to and assign them to the index of choice. Probably sounds complicated but isn't bad really depending on what level of access or turn around you have to things like new DNS entries, OS, etc.
I would suggest setting up a syslog server that captures all the incoming syslog messages and stores them into a unique directory or file for each host. This is pretty straightforward of a syslog setup. I use a directory with the hostname and the hostname-date as the log file for each day.
Then create file monitors for each of the directories. You can make all for determination about which index to put things in using your inputs.conf.