topic Split Syslog (UDP:514) from multi hosts to multi indexes. in Getting Data In

Split Syslog (UDP:514) from multi hosts to multi indexes.

aaronnicoli — Thu, 14 Feb 2013 21:12:23 GMT

Hi there,

I am using syslog on Splunk currently to capture data from a piece of content-keeper hardware on our network that handles internet usage analysis. It's a fairly straight forward input, but, now I wish to capture more data from a firewall again using the standard UDP:514.

However, I would like for my CK data to go into splunk with the following values:

Host: content-keeper
Index: internet

And then my firewall to go in as:

Host: foo
Index: firewalls

Currently my inputs.conf segment looks like:

[udp://514]
connection_host = none
host = content-keeper
index = internet
sourcetype = content-keeper
source = content-keeper

props.conf

[content-keeper]
LOOKUP-domaingroups = common_domains basedomain AS basedomain OUTPUTNEW domaingroup AS domaingroup

transforms.conf

[common_domains]
filename = group_common_domains.csv

As you can see with props and transforms, I am just doing some basic lookups.

Any help would be very very appreciated.

Aaron.

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

jbsplunk — Thu, 14 Feb 2013 22:32:05 GMT

What you're looking for is documented here:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

What you'd need to do would be something like this:

props.conf:

[content-keeper]
TRANSFORMS-index = firewalls,hosts

transforms.conf:

[firewalls]
REGEX = <MYFIREWALLREGEX>
DEST_KEY = _MetaData:Index
FORMAT = firewalls

[hosts]
REGEX = <(MYFWHOSTREGEX)>
DEST_KEY = MetaData:Host
FORMAT = host::$1

By the way, at the time you do this, the 'firewalls' index must have already been created.

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

aaronnicoli — Thu, 14 Feb 2013 22:49:32 GMT

Thanks JB,

This is pretty much what I have come up with. The issue I am encountering now is with the regex, I cannot seem to find a way of extracting the hostname or some indicator of the orgin host. When I removed the "host=" over-ride from my inputs.conf, the host value simply defaulted to the fqdn of the splunk indexer...
I am going to simply have to look more into it.

Thanks very very much for you well written response!
Aaron.

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

Runals — Fri, 15 Feb 2013 06:36:44 GMT

Create multiple DNS entries, assign them to virtual hosts on your system, configure syslog to look for UDP/514 from each of those IPs and write the data to disk, then configure multiple Splunk imports to look at each of the paths syslog writes to and assign them to the index of choice. Probably sounds complicated but isn't bad really depending on what level of access or turn around you have to things like new DNS entries, OS, etc.

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

aaronnicoli — Mon, 04 Mar 2013 23:25:13 GMT

Solved with a simple regex that can determine which data to perform the change on and which to not perform changes on.

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

bstimely — Fri, 05 Oct 2018 19:11:51 GMT

I would suggest setting up a syslog server that captures all the incoming syslog messages and stores them into a unique directory or file for each host. This is pretty straightforward of a syslog setup. I use a directory with the hostname and the hostname-date as the log file for each day.
Then create file monitors for each of the directories. You can make all for determination about which index to put things in using your inputs.conf.