Getting Data In

Split Syslog (UDP:514) from multi hosts to multi indexes.

Path Finder

Hi there,

I am using syslog on Splunk currently to capture data from a piece of content-keeper hardware on our network that handles internet usage analysis. It's a fairly straight forward input, but, now I wish to capture more data from a firewall again using the standard UDP:514.

However, I would like for my CK data to go into splunk with the following values:

Host: content-keeper
Index: internet

And then my firewall to go in as:

Host: foo
Index: firewalls

Currently my inputs.conf segment looks like:

[udp://514]
connection_host = none
host = content-keeper
index = internet
sourcetype = content-keeper
source = content-keeper

props.conf

[content-keeper]
LOOKUP-domaingroups = common_domains basedomain AS basedomain OUTPUTNEW domaingroup AS domaingroup

transforms.conf

[common_domains]
filename = group_common_domains.csv

As you can see with props and transforms, I am just doing some basic lookups.

Any help would be very very appreciated.

Aaron.

1 Solution

Splunk Employee
Splunk Employee

What you're looking for is documented here:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

What you'd need to do would be something like this:

props.conf:

[content-keeper]
TRANSFORMS-index = firewalls,hosts

transforms.conf:

[firewalls]
REGEX = <MYFIREWALLREGEX>
DEST_KEY = _MetaData:Index
FORMAT = firewalls

[hosts]
REGEX = <(MYFWHOSTREGEX)>
DEST_KEY = MetaData:Host
FORMAT = host::$1

By the way, at the time you do this, the 'firewalls' index must have already been created.

View solution in original post

New Member

I would suggest setting up a syslog server that captures all the incoming syslog messages and stores them into a unique directory or file for each host. This is pretty straightforward of a syslog setup. I use a directory with the hostname and the hostname-date as the log file for each day.
Then create file monitors for each of the directories. You can make all for determination about which index to put things in using your inputs.conf.

0 Karma

Motivator

Create multiple DNS entries, assign them to virtual hosts on your system, configure syslog to look for UDP/514 from each of those IPs and write the data to disk, then configure multiple Splunk imports to look at each of the paths syslog writes to and assign them to the index of choice. Probably sounds complicated but isn't bad really depending on what level of access or turn around you have to things like new DNS entries, OS, etc.

0 Karma

Splunk Employee
Splunk Employee

What you're looking for is documented here:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

What you'd need to do would be something like this:

props.conf:

[content-keeper]
TRANSFORMS-index = firewalls,hosts

transforms.conf:

[firewalls]
REGEX = <MYFIREWALLREGEX>
DEST_KEY = _MetaData:Index
FORMAT = firewalls

[hosts]
REGEX = <(MYFWHOSTREGEX)>
DEST_KEY = MetaData:Host
FORMAT = host::$1

By the way, at the time you do this, the 'firewalls' index must have already been created.

View solution in original post

Path Finder

Solved with a simple regex that can determine which data to perform the change on and which to not perform changes on.

0 Karma

Path Finder

Thanks JB,

This is pretty much what I have come up with. The issue I am encountering now is with the regex, I cannot seem to find a way of extracting the hostname or some indicator of the orgin host. When I removed the "host=" over-ride from my inputs.conf, the host value simply defaulted to the fqdn of the splunk indexer...
I am going to simply have to look more into it.

Thanks very very much for you well written response!
Aaron.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!