topic Re: How to split a log into multiple sourcetypes on a heavy forwarder? in Getting Data In

How to split a log into multiple sourcetypes on a heavy forwarder?

cdstealer — Tue, 10 Mar 2015 15:24:24 GMT

Hi,
I seem to be struggling in splitting log data from the heavy forwarder into several sourcetypes in an index.

I have a network device sending logdata to the heavyforwarder via syslog which dumps it into a flat file.
The heavyforwarder reads the files and sends to the indexers.. All good so far.
Now what I'm trying to do is have 2 regexes split into 2 sourcetypes and everything else into the default sourcetype for that index.

props.conf

    [asm_log]
    TIME_PREFIX = ^
    TIME_FORMAT= %Y-%m-%dTO%H:%M:%S%:z
    TRANSFORMS-changeSourcetype1 = routeAsmlog, asm-set-sourcetype

    [psm_log]
    TIME_PREFIX = ^
    TIME_FORMAT= %Y-%m-%dTO%H:%M:%S%:z
    TRANSFORMS-changeSourcetype2 = routePsmlog, psm-set-sourcetype

transforms.conf

    [routeAsmlog]
    REGEX = (\sASM:\s)
    DEST_KEY = queue
    FORMAT = indexQueue

    [asm-set-sourcetype]
    DEST_KEY = MetaData:Sourcetype
    REGEX = (\sASM:\s)
    FORMAT = asm_log

also tried FORMAT = sourcetype::f5:asm

    [routePsmlog]
    REGEX = (\sPSM:\s)
    DEST_KEY = queue
    FORMAT = indexQueue

    [psm-set-sourcetype]
    DEST_KEY = MetaData:Sourcetype
    REGEX = (\sPSM:\s)
    FORMAT = psm_log

inputs.conf

    [monitor:///splunk/F5]
    disabled = false
    sourcetype = f5
    index = f5
    blacklist = gz

also tried removing the sourcetype line

Any idea where I'm going wrong?

Thanks in advance
Steve

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

somesoni2 — Tue, 10 Mar 2015 16:36:16 GMT

Try this (put it on Heavy forwarder and restart)

inputs.conf (same as before)

[monitor:///splunk/F5]
disabled = false
sourcetype = f5
index = f5
blacklist = gz

props.conf

[f5]
TIME_PREFIX = ^
TIME_FORMAT= %Y-%m-%dTO%H:%M:%S%:z
TRANSFORMS-changeSourcetype1 = psm-set-sourcetype, asm-set-sourcetype

[psm_log]
[asm_log]

transforms.conf

[asm-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sASM:\s)
FORMAT = asm_log

[psm-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sPSM:\s)
FORMAT = psm_log

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

cdstealer — Tue, 10 Mar 2015 17:14:59 GMT

Thank you yet again for saving my frail sanity 🙂 Worked like a charm!

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

ppablo — Tue, 10 Mar 2015 23:39:16 GMT

Hi @cdstealer

Glad @somesoni2 helped you find a solution 🙂 don't forget to officially accept his answer by clicking on "Accept" directly below the answer to resolve this post and for both of you to receive karma points. Thanks!

Patrick

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

cdstealer — Wed, 11 Mar 2015 05:29:08 GMT

Thanks Patrick.. completely forgot.. doh! 🙂

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

vGreeshma — Mon, 23 Mar 2015 12:35:10 GMT

After splitting in to multiple source types how to set the default(here f5 source type ) to empty

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

markwymer — Tue, 29 Sep 2020 06:57:09 GMT

Sorry to resurrect an old thread due to my not understanding the solution given, but I have a very similar (if not exactly the same!) issue.

I currently have one log file ( ba.mobile.log ) that gets fed into a single index ( ba_com_mobile_logs ) . The log file has custom events that all start with '|HDR1|' which I have created a custom source type call ba.com_activity_log. There is also loads of xml data in the same log file that I would like treat as a separate sourcetype but still have the messages in the same index.

Is this possible or should I split the two source types into two indexes. If it is possible, what the heck do I put into my props and transforms configs?

The $SPLUNK_HOME/etc/apps/search/local/inputs.conf file on my forwarder looks like...

[monitor::///ba/ba.mobile.log]
index=ba_com_mobile_logs
sourcetype=ba.com_activity_log

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

markwymer — Wed, 12 Aug 2015 07:27:17 GMT

Sorry, I also meant to add.....

Thanks for any help.

Re: How to split a log into multiple sourcetypes on a heavy forwarder?

mbschriek — Tue, 29 Sep 2020 11:37:15 GMT

The above suggestion is not correct since it didn't succeeded for me. I changed the following format:
` [asm-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sASM:\s)
FORMAT = sourcetype:asm_log

[psm-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sPSM:\s)
FORMAT = sourcetype:psm_log`