Detecting multiple sourcetypes within a single archive

[rturk]

Hi Splunkers!

Beyond configuring the autodetection of new sourcetypes, is specifying sourcetype detection via a wildcard possible when the log file is within a zip file?

For the sake of illustration, I have a single zip file, with the following two files:

/data/db2.zip
 - AUDIT_AUTHORIZATION_FAILURES.log
 - AUDIT_FIRST_READ.log

Example:

[batch:///data/db2.zip:AUDIT_AUTHORIZATION_FAILURES*]
index = db2
sourcetype = db2_audit_auth_fail
move_policy = sinkhole
queue = parsingQueue

[batch:///data/db2.zip:AUDIT_FIRST_READ*]
index = db2
sourcetype = db2_audit_first_read
move_policy = sinkhole
queue = parsingQueue

Not pretty I know, but I'd rather avoid having to configure automatic sourcetype recognition as the events are multi-lined and painful.

Cheers & Beers,

RT

[rturk]

They are indeed. Thanks for the comment... it twigged how I should do it. Thankfully there are only a few different types of files and they are consistently named.

[rturk · Accepted solution]

So I've found the way to fix the issue. Luckily (as Drainy asked) there are only a few distinct filenames in each of the archives, such as:

db2_20120620.zip
- AUDIT_FIRST_READ.TXT
- AUDIT_FIRST_WRITE.TXT
- AUDIT_AUTHORIZATION_FAIL.TXT

So to apply file specific sourcetypeing I did the following:
- In my inputs.conf, I don't specify the sourcetype.
- In my props.conf, I override the sourcetype based on the source. eg:

[source::.../db2_\d+.zip:./AUDIT_FIRST_READ.TXT]
sourcetype = db2_audit_first_read

[source::.../db2_\d+.zip:./AUDIT_FIRST_WRITE.TXT]
sourcetype = db2_audit_first_write

[source::.../db2_\d+.zip:./AUDIT_AUTHORIZATION_FAIL.TXT]
sourcetype = db2_audit_authorization_fail

Then I define my sourcetypes as I would normally.

Hope this helps someone 🙂

[Drainy]

Are the events within the two log files quite distinct?