topic Detecting multiple sourcetypes within a single archive in Getting Data In

Detecting multiple sourcetypes within a single archive

rturk — Wed, 04 Jul 2012 02:28:55 GMT

Hi Splunkers!

Beyond configuring the autodetection of new sourcetypes, is specifying sourcetype detection via a wildcard possible when the log file is within a zip file?

For the sake of illustration, I have a single zip file, with the following two files:

/data/db2.zip
 - AUDIT_AUTHORIZATION_FAILURES.log
 - AUDIT_FIRST_READ.log

Example:

[batch:///data/db2.zip:AUDIT_AUTHORIZATION_FAILURES*]
index = db2
sourcetype = db2_audit_auth_fail
move_policy = sinkhole
queue = parsingQueue

[batch:///data/db2.zip:AUDIT_FIRST_READ*]
index = db2
sourcetype = db2_audit_first_read
move_policy = sinkhole
queue = parsingQueue

Not pretty I know, but I'd rather avoid having to configure automatic sourcetype recognition as the events are multi-lined and painful.

Cheers & Beers,

Re: Detecting multiple sourcetypes within a single archive

Drainy — Thu, 05 Jul 2012 13:56:44 GMT

Are the events within the two log files quite distinct?

Re: Detecting multiple sourcetypes within a single archive

rturk — Mon, 28 Sep 2020 12:05:20 GMT

So I've found the way to fix the issue. Luckily (as Drainy asked) there are only a few distinct filenames in each of the archives, such as:

db2_20120620.zip
- AUDIT_FIRST_READ.TXT
- AUDIT_FIRST_WRITE.TXT
- AUDIT_AUTHORIZATION_FAIL.TXT

So to apply file specific sourcetypeing I did the following:
- In my inputs.conf, I don't specify the sourcetype.
- In my props.conf, I override the sourcetype based on the source. eg:

[source::.../db2_\d+.zip:./AUDIT_FIRST_READ.TXT]
sourcetype = db2_audit_first_read

[source::.../db2_\d+.zip:./AUDIT_FIRST_WRITE.TXT]
sourcetype = db2_audit_first_write

[source::.../db2_\d+.zip:./AUDIT_AUTHORIZATION_FAIL.TXT]
sourcetype = db2_audit_authorization_fail

Then I define my sourcetypes as I would normally.

Hope this helps someone 🙂

Re: Detecting multiple sourcetypes within a single archive

rturk — Sun, 15 Jul 2012 01:16:29 GMT

They are indeed. Thanks for the comment... it twigged how I should do it. Thankfully there are only a few different types of files and they are consistently named.