Getting Data In
Highlighted

routing remote syslog data to different indexes

Path Finder

We have a number of machines set up with rsyslog to collect data from various systems. Rsyslog all the data is sent to /var/log/remote/ in to individual log files for the sending host. I've got each of these machines also set up as heavy forwarders so we can do whatever we need to with the incoming data before we send it along to the indexers.

Here's the inputs.conf file that I'm using at the moment:

[monitor:///var/log/remote]
disabled = 0
index = syslog
sourcetype = syslog

What I want to do eventually is change the index that the data is going to based on the host that is sending the initial syslog stream. What I'm looking for is some guidance on going about doing this. Is there a right way and a wrong way?

Thoughts, comments, questions?

colin j.

0 Karma
Highlighted

Re: routing remote syslog data to different indexes

Splunk Employee
Splunk Employee

Here is the exact answer you need :

http://splunk-base.splunk.com/answers/8531/routing-to-index-based-on-host-etc

This is called : Index time field transformation.
You want to setup this props/transforms rules on the indexer (AND heavy forwarder in your case)

  • The props.conf will contain the sourcetype and the transforms to call


    [syslog]
    TRANSFORMS-changeindex=routemyhosttomynewindex

  • the transforms.conf will use a regex to match the host and change the index accordingly.


    [routemyhosttomynewindex]
    SOURCEKEY = MetaData:Host
    REGEX = myhost
    DEST
    KEY = MetaData:Index
    FORMAT = mynewindex

Highlighted

Re: routing remote syslog data to different indexes

Explorer

Hi Yann,

Shouldnt that be "_MetaData:Index" instead of "MetaData:Index" ?

0 Karma
Highlighted

Re: routing remote syslog data to different indexes

Path Finder

I am trying to use the props and transforms as provided in the answer here. In my case, the server, where the HFWD, is contains logs from several sources.
Will this work ? ( the following files will be on the HFWD )

inputs.conf

[monitor:///var/log/*]
sourcetype = syslog
host_segment = 3

props.conf
[syslog]
TRANSFORMS-changeindex=routemyhosttomynewindex

transforms.conf
[routemyhosttomynewindex]
SOURCEKEY = MetaData:Host
REGEX = myhost
DEST
KEY = MetaData:Index
FORMAT = mynewindex

0 Karma