topic Re: Override sourcetype for custom logs in Getting Data In

Override sourcetype for custom logs

sundarrajan — Wed, 30 Sep 2020 04:18:26 GMT

Hi I am trying to override my current sourcetype to create multiple source types based on key matching patterns. But the settings are not working, my settings are as follows, pls let know me where I go wrong,
pros. conf
[transaction:logs]
BREAK_ONLY_BEFORE_DATE = true
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25
TRANSFORMS - sourcetypeoverwrite =receipts, businesstransaction

transforms.conf
[receipts]
DEST_KEY = MetaData:Sourcetype
REGEX = (%retail)
FORMAT = sourcetype::transaction:logs

[businesstransaction]
DEST_KEY = MetaData:Sourcetype
REGEX = (%transaction)
FORMAT = sourcetype::transaction:logs

I also tried rule:: option but its not working as well in my props.conf
[rule::receipts]
sourcetype = receipt
MORE_THAN_0 = (%retail)

[rule::businesstransaction]
sourcetype = businesstransaction
MORE_THAN_0 = (%transaction)

Yet am not getting results in either of methods. Is there any better way to look into this.

Re: Override sourcetype for custom logs

sundarrajan — Wed, 30 Sep 2020 04:18:28 GMT

I also tried with INGEST_EVAL
INGEST_EVAL = sourcetype:=case(sourcetype=="transaction", "businesstransaction", sourcetype=="retail", "receipts", true(), sourcetype), still its not splitting as expected.

Re: Override sourcetype for custom logs

nickhills — Fri, 21 Feb 2020 12:01:13 GMT

You don't say what you want the new sourcetypes called?

I'll assume they should be called transaction:receipts and transaction: businesstransaction ...
transforms.conf

[receipts]
DEST_KEY = MetaData:Sourcetype
REGEX = (%retail)
FORMAT = sourcetype::transaction:receipts

[businesstransaction]
DEST_KEY = MetaData:Sourcetype
REGEX = (%transaction)
FORMAT = sourcetype::transaction:businesstransaction

Im also assuming you are using (%retail) as a placeholder to refer to some regex which matches in your relevant log? %retail is itself not a valid regex expression. so you might need to use something which actually matches. potentially something like:

eventdata=some text from your log which indicates this is a retail log value=23.45 

regex=\w\s(retail)\s

Re: Override sourcetype for custom logs

sundarrajan — Wed, 30 Sep 2020 04:18:31 GMT

Hi @nickhills thanks for the response. %retail% & %transaction are the key words. I tried to use these key words from the logs as condition to override the event to a new sourcetype.
Please do suggest, if we can use keywords (%retail) in place of regex pattern for the key word. I also tried this key word as part of my rules(MORE_THAN_1 line in the event), but it didn't work.

Re: Override sourcetype for custom logs

nickhills — Fri, 21 Feb 2020 13:09:15 GMT

oh so literal %retail is what you mean?

so you are matching something like this:
eventdata=some text from your log blah blah %retail value=23.45
in which case, your regex should be fine 🙂

Re: Override sourcetype for custom logs

nickhills — Fri, 21 Feb 2020 13:14:55 GMT

try my transforms example from above, I think it should work for you.

Re: Override sourcetype for custom logs

sundarrajan — Wed, 30 Sep 2020 04:18:35 GMT

@nickhills I tried the transforms.conf, still its not showing up results for me. The input log is XML file and i have other custom conditions like LINE_BREAKER, BREAK_ONLY_BEFORE, MUST_BREAK_AFTER conditions in my props. I presume that is not making any impact on this transforms.conf settings.