topic Re: How to send syslog data to a specific index based on a string in the log? in Getting Data In

How to send syslog data to a specific index based on a string in the log?

hlarimer — Thu, 12 Feb 2015 03:57:48 GMT

I have syslog data coming to a distributed environment. I am trying to send the data to a specific index based on a string that is in the data. The string is "NetScreen" and the index is juniper. I have the following props.conf and transforms.conf deployed to the forwarder that is collecting these logs but the logs are still coming into index main

Props.conf

[JuniperFW]
Transforms-JuniperFW=JuniperFW

Transforms.conf

[JuniperFW]
DEST_KEY = _MetaData:Index
REGEX = (NetScreen)
FORMAT = index::juniper

Re: How to send syslog data to a specific index based on a string in the log?

acharlieh — Thu, 12 Feb 2015 04:14:06 GMT

"The forwarder that is collecting these logs" -> Is this a Universal Forwarder or a Heavy Forwarder? The UF only does input phase. In which case you would have to set this on your Indexer(s) or Heavy Forwarders. See the Splunk Wiki for a handy reference for what applies where.

Also you probably want to capitalize TRANSFORMS in the props.conf

Re: How to send syslog data to a specific index based on a string in the log?

hlarimer — Thu, 12 Feb 2015 14:00:00 GMT

It is a UF...
Moving those to the Indexer took care of it! I had to change FORMAT = index:juniper to FORMAT = juniper but that got the data going to right location. Thank you for your help!

Re: How to send syslog data to a specific index based on a string in the log?

hlarimer — Mon, 28 Sep 2020 18:57:21 GMT

Do you know if I can also change the sourcetype here. I have tried using another stanza in transforms.conf and props.conf as shown below, but it is not changing the sourcetype, is there another method I should be using?

props.conf
[JuniperFW]
TRANSFORMS-JuniperFW=JuniperFW

[JuniperFW_ST]
TRANSFORMS-JuniperRW_ST=JuniperFW_ST

transforms.conf
[JuniperFW]
DEST_KEY = _MetaData:Index
REGEX = (NetScreen)
FORMAT = juniper

[JuniperFW_ST]
DEST_KEY = MetaData:Sourcetype
REGEX = (NetScreen)
FORMAT = JuniperFW

Re: How to send syslog data to a specific index based on a string in the log?

acharlieh — Fri, 13 Feb 2015 21:28:45 GMT

You should be able to change sourcetype at index time as well... (I'm not sure if the other sourcetype's props would apply or not when you change it, I haven't done much in this space). But I will note that according to the transforms.conf doc setting MetaData:Sourcetype requires the value to be prefixed sourcetype:: so your FORMAT line should be sourcetype::JuniperFW

Alternatively, you can also rename sourcetypes at search time (slightly different configs).

Re: How to send syslog data to a specific index based on a string in the log?

acharlieh — Fri, 13 Feb 2015 21:42:54 GMT

Re: How to send syslog data to a specific index based on a string in the log?

hlarimer — Sun, 15 Feb 2015 04:17:45 GMT

I ended up leaving them in index=main and then setting the sourcetype and index time. My ultimate goal was to normalize the data to follow the CIM so I could create searches across multiple types of firewalls. With the data separated into sourcetypes I am now able to create the search time field extractions I needed, although I struggled with this until I realized that the app that has the field extractions needed to be set to be shared globally in metadata/default.meta.

Everything is working now and now its time to start actually using the data!!