Getting Data In

How to configure multiple sourcetypes for a single monitored file?

New Member


I have a single very huge file with different formats. So I decided to create 3 different sourcetypes for this single file. I tried the below, but I did not succeed. Can any one point out where am I lagging?


disabled = false
followTail = 0
index = main
sourcetype = sourcetype1name


TRANSFORMS-myfileformats = format1, format2, format3


FORMAT = sourcetype::Sourcetype2name
DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype::Sourcetype3name
DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype::Sourcetype4name
DEST_KEY = MetaData:Sourcetype 

These are my config files and am not sure what to be done here. I have not created the Sourcetype2, Sourcetype3, Sourcetype4 so far since Sourcetype cannot be created itself in 6.0 version.

0 Karma

Ultra Champion

Our expert said -

Yeah this would have to be done on a heavy indexer,
Which is also good for doing the parsing CPU processing on a heavy forwarder instead of the indexer.

We could send this file(s) through syslog (/etc/rsyslog.conf) to heavy forwarder too, then the heavy forwarder would transform the file.

Only thing I would ask if the timestamps are going to be different. That would propose a new problem to solve. Having three different date formats in one file?

0 Karma


I had the problem of different time formats in the same file. I used a script to parse the data out into separate files and ingest those, because it wasn't easy to correct the situation at the source. I was doing this on a heavy forwarder.

here was my script: (with edits for understanding)

grep SpecialLineHeader /path/to/file/myfile.txt | awk '{ print $1, $2, $3, $4, $5, $6 }' > /opt/splunk/etc/apps/mainframe/local/parsedSpecialLineSource/SpecialFile.txt

0 Karma


This should be doable.

What does your infrastructure look like? Are you reading that file from a universal forwarder? Where did you place your props/transforms. They should be on some server that does parsing in your environment, typically an indexer or heavy forwarder.

And of course, are you sure the regex is correct?

If all of that looks good, what if you try placing the TRANSFORMS-myfileformats key into a [sourcetype1name] stanza since you are setting that explicitly on your inputs anyway.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!