Getting Data In
Highlighted

Split Syslog (UDP:514) from multi hosts to multi indexes.

Path Finder

Hi there,

I am using syslog on Splunk currently to capture data from a piece of content-keeper hardware on our network that handles internet usage analysis. It's a fairly straight forward input, but, now I wish to capture more data from a firewall again using the standard UDP:514.

However, I would like for my CK data to go into splunk with the following values:

Host: content-keeper
Index: internet

And then my firewall to go in as:

Host: foo
Index: firewalls

Currently my inputs.conf segment looks like:

[udp://514]
connection_host = none
host = content-keeper
index = internet
sourcetype = content-keeper
source = content-keeper

props.conf

[content-keeper]
LOOKUP-domaingroups = common_domains basedomain AS basedomain OUTPUTNEW domaingroup AS domaingroup

transforms.conf

[common_domains]
filename = group_common_domains.csv

As you can see with props and transforms, I am just doing some basic lookups.

Any help would be very very appreciated.

Aaron.

Highlighted

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

Influencer

What you're looking for is documented here:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

What you'd need to do would be something like this:

props.conf:

[content-keeper]
TRANSFORMS-index = firewalls,hosts

transforms.conf:

[firewalls]
REGEX = <MYFIREWALLREGEX>
DEST_KEY = _MetaData:Index
FORMAT = firewalls

[hosts]
REGEX = <(MYFWHOSTREGEX)>
DEST_KEY = MetaData:Host
FORMAT = host::$1

By the way, at the time you do this, the 'firewalls' index must have already been created.

View solution in original post

Highlighted

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

Path Finder

Thanks JB,

This is pretty much what I have come up with. The issue I am encountering now is with the regex, I cannot seem to find a way of extracting the hostname or some indicator of the orgin host. When I removed the "host=" over-ride from my inputs.conf, the host value simply defaulted to the fqdn of the splunk indexer...
I am going to simply have to look more into it.

Thanks very very much for you well written response!
Aaron.

0 Karma
Highlighted

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

Path Finder

Solved with a simple regex that can determine which data to perform the change on and which to not perform changes on.

0 Karma
Highlighted

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

Motivator

Create multiple DNS entries, assign them to virtual hosts on your system, configure syslog to look for UDP/514 from each of those IPs and write the data to disk, then configure multiple Splunk imports to look at each of the paths syslog writes to and assign them to the index of choice. Probably sounds complicated but isn't bad really depending on what level of access or turn around you have to things like new DNS entries, OS, etc.

0 Karma
Highlighted

Re: Split Syslog (UDP:514) from multi hosts to multi indexes.

New Member

I would suggest setting up a syslog server that captures all the incoming syslog messages and stores them into a unique directory or file for each host. This is pretty straightforward of a syslog setup. I use a directory with the hostname and the hostname-date as the log file for each day.
Then create file monitors for each of the directories. You can make all for determination about which index to put things in using your inputs.conf.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.