Getting Data In

Detecting multiple sourcetypes within a single archive

Builder

Hi Splunkers!

Beyond configuring the autodetection of new sourcetypes, is specifying sourcetype detection via a wildcard possible when the log file is within a zip file?

For the sake of illustration, I have a single zip file, with the following two files:

``````/data/db2.zip
- AUDIT_AUTHORIZATION_FAILURES.log
``````

Example:

``````[batch:///data/db2.zip:AUDIT_AUTHORIZATION_FAILURES*]
index = db2
sourcetype = db2_audit_auth_fail
move_policy = sinkhole
queue = parsingQueue

index = db2
move_policy = sinkhole
queue = parsingQueue
``````

Not pretty I know, but I'd rather avoid having to configure automatic sourcetype recognition as the events are multi-lined and painful.

Cheers & Beers,

RT

Tags (4)
1 Solution
Builder

So I've found the way to fix the issue. Luckily (as Drainy asked) there are only a few distinct filenames in each of the archives, such as:

db220120620.zip
- AUDIT
- AUDIT
FIRSTWRITE.TXT
- AUDIT
AUTHORIZATION_FAIL.TXT

So to apply file specific sourcetypeing I did the following:
- In my inputs.conf, I don't specify the sourcetype.
- In my props.conf, I override the sourcetype based on the source. eg:

``````[source::.../db2_\d+.zip:./AUDIT_FIRST_READ.TXT]

[source::.../db2_\d+.zip:./AUDIT_FIRST_WRITE.TXT]
sourcetype = db2_audit_first_write

[source::.../db2_\d+.zip:./AUDIT_AUTHORIZATION_FAIL.TXT]
sourcetype = db2_audit_authorization_fail
``````

Then I define my sourcetypes as I would normally.

Hope this helps someone 🙂

Builder

They are indeed. Thanks for the comment... it twigged how I should do it. Thankfully there are only a few different types of files and they are consistently named.

Builder

So I've found the way to fix the issue. Luckily (as Drainy asked) there are only a few distinct filenames in each of the archives, such as:

db220120620.zip
- AUDIT
- AUDIT
FIRSTWRITE.TXT
- AUDIT
AUTHORIZATION_FAIL.TXT

So to apply file specific sourcetypeing I did the following:
- In my inputs.conf, I don't specify the sourcetype.
- In my props.conf, I override the sourcetype based on the source. eg:

``````[source::.../db2_\d+.zip:./AUDIT_FIRST_READ.TXT]

[source::.../db2_\d+.zip:./AUDIT_FIRST_WRITE.TXT]
sourcetype = db2_audit_first_write

[source::.../db2_\d+.zip:./AUDIT_AUTHORIZATION_FAIL.TXT]
sourcetype = db2_audit_authorization_fail
``````

Then I define my sourcetypes as I would normally.

Hope this helps someone 🙂

Champion

Are the events within the two log files quite distinct?