topic Re: How to configure multiple sourcetypes for a single monitored file? in Getting Data In

How to configure multiple sourcetypes for a single monitored file?

KVinodh — Mon, 22 Feb 2016 17:53:42 GMT

Hi.

I have a single very huge file with different formats. So I decided to create 3 different sourcetypes for this single file. I tried the below, but I did not succeed. Can any one point out where am I lagging?

inputs.conf

[monitor:///file/path/file.txt]
disabled = false
followTail = 0
index = main
sourcetype = sourcetype1name

props.conf

[source::/file/path/file.txt]
TRANSFORMS-myfileformats = format1, format2, format3

transforms.conf

[format1]
REGEX =REGEX 1XXXXXXXXXXXX
FORMAT = sourcetype::Sourcetype2name
DEST_KEY = MetaData:Sourcetype

[format2]
REGEX = REGEX2XXXXXXXXXXXXXXXXXXX
FORMAT = sourcetype::Sourcetype3name
DEST_KEY = MetaData:Sourcetype

[format3]
REGEX = REGEX#XXXXXXXXXXXXXXXXXXXXXXXX
FORMAT = sourcetype::Sourcetype4name
DEST_KEY = MetaData:Sourcetype

These are my config files and am not sure what to be done here. I have not created the Sourcetype2, Sourcetype3, Sourcetype4 so far since Sourcetype cannot be created itself in 6.0 version.

Re: How to configure multiple sourcetypes for a single monitored file?

maciep — Sun, 06 Mar 2016 02:12:34 GMT

This should be doable.

What does your infrastructure look like? Are you reading that file from a universal forwarder? Where did you place your props/transforms. They should be on some server that does parsing in your environment, typically an indexer or heavy forwarder.

And of course, are you sure the regex is correct?

If all of that looks good, what if you try placing the TRANSFORMS-myfileformats key into a [sourcetype1name] stanza since you are setting that explicitly on your inputs anyway.

Re: How to configure multiple sourcetypes for a single monitored file?

ddrillic — Mon, 07 Mar 2016 15:22:35 GMT

Our expert said -

Yeah this would have to be done on a heavy indexer,
Which is also good for doing the parsing CPU processing on a heavy forwarder instead of the indexer.

We could send this file(s) through syslog (/etc/rsyslog.conf) to heavy forwarder too, then the heavy forwarder would transform the file.

Only thing I would ask if the timestamps are going to be different. That would propose a new problem to solve. Having three different date formats in one file?

Re: How to configure multiple sourcetypes for a single monitored file?

jgbricker — Mon, 07 Mar 2016 20:00:03 GMT

I had the problem of different time formats in the same file. I used a script to parse the data out into separate files and ingest those, because it wasn't easy to correct the situation at the source. I was doing this on a heavy forwarder.

here was my script: (with edits for understanding)

grep SpecialLineHeader /path/to/file/myfile.txt | awk '{ print $1, $2, $3, $4, $5, $6 }' > /opt/splunk/etc/apps/mainframe/local/parsedSpecialLineSource/SpecialFile.txt