topic Re: routing remote syslog data to different indexes in Getting Data In

routing remote syslog data to different indexes

colinj — Thu, 19 Jul 2012 19:27:18 GMT

We have a number of machines set up with rsyslog to collect data from various systems. Rsyslog all the data is sent to /var/log/remote/ in to individual log files for the sending host. I've got each of these machines also set up as heavy forwarders so we can do whatever we need to with the incoming data before we send it along to the indexers.

Here's the inputs.conf file that I'm using at the moment:

[monitor:///var/log/remote]
disabled = 0
index = syslog
sourcetype = syslog

What I want to do eventually is change the index that the data is going to based on the host that is sending the initial syslog stream. What I'm looking for is some guidance on going about doing this. Is there a right way and a wrong way?

Thoughts, comments, questions?

colin j.

Re: routing remote syslog data to different indexes

yannK — Mon, 28 Sep 2020 12:07:26 GMT

Here is the exact answer you need :

http://splunk-base.splunk.com/answers/8531/routing-to-index-based-on-host-etc

This is called : Index time field transformation.
You want to setup this props/transforms rules on the indexer (AND heavy forwarder in your case)

The props.conf will contain the sourcetype and the transforms to call
```
[syslog]

TRANSFORMS-changeindex=routemyhosttomynewindex
```

the transforms.conf will use a regex to match the host and change the index accordingly.



[routemyhosttomynewindex]

SOURCE_KEY = MetaData:Host

REGEX = myhost

DEST_KEY = MetaData:Index

FORMAT = mynewindex

Re: routing remote syslog data to different indexes

mikeschneider — Thu, 12 Mar 2015 11:08:31 GMT

Hi Yann,

Shouldnt that be "_MetaData:Index" instead of "MetaData:Index" ?

Re: routing remote syslog data to different indexes

rajanala — Tue, 29 Sep 2020 09:02:08 GMT

I am trying to use the props and transforms as provided in the answer here. In my case, the server, where the HFWD, is contains logs from several sources.
Will this work ? ( the following files will be on the HFWD )

inputs.conf

[monitor:///var/log/*]
sourcetype = syslog
host_segment = 3

props.conf
[syslog]
TRANSFORMS-changeindex=routemyhosttomynewindex

transforms.conf
[routemyhosttomynewindex]
SOURCE_KEY = MetaData:Host
REGEX = myhost
DEST_KEY = MetaData:Index
FORMAT = mynewindex