Split sourcetype + indexed fields

michael_vi — Wed, 13 Mar 2024 16:27:44 GMT

I have an issue with adding indexed fields to each of the new (splatted) sourcetype:

Configuration that "duplicated" indexed fields for each sourcetype:

Now I see fields: indexedfileds1, indexedfileds2 and indexedfileds3 as 200%,

For example: indexedfields1 values:

valuie1 150%

value2 50%

props.conf

[MAIN SOURCE] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = {\"time\":\" MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z TRUNCATE = 999999 TRANSFORMS-changesourcetype = sourcetype1, sourcetype2 TRANSFORMS-indexedfields = indexedfield1, indexedfield2, indexedfield3 [sourcetype1] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = {\"time\":\" MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z TRUNCATE = 999999 [sourcetype2] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = {\"time\":\" MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z TRUNCATE = 999999

transforms.conf

[indexedfield1] REGEX= FORMAT= WRITE_META= [indexedfield2] REGEX= FORMAT= WRITE_META= [indexedfield3] REGEX= FORMAT= WRITE_META= [sourcetype1] DEST_KEY-MetaData:Sourcetype REGEX = some regex FORMAT = sourcetype::sourcetype1 [sourcetype2] DEST_KEY-MetaData:Sourcetype REGEX = some regex FORMAT = sourcetype::sourcetype2

I thought to move the indexed fields to each of the new sourcetype but then I see no indexed fields.

Check with | tstats count

props.conf

[MAIN SOURCE] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = {\"time\":\" MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z TRUNCATE = 999999 TRANSFORMS-changesourcetype = sourcetype1, sourcetype2 [sourcetype1] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = {\"time\":\" MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z TRUNCATE = 999999 TRANSFORMS-indexedfields = indexedfield1, indexedfield2, indexedfield3 [sourcetype2] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = {\"time\":\" MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z TRUNCATE = 999999 TRANSFORMS-indexedfields = indexedfield1, indexedfield2, indexedfield3

What is the needed configuration to see indexed fields per sourcetype, w/o showing 200%

Thanks

Re: Split sourcetype + indexed fields

isoutamo — Thu, 14 Mar 2024 07:27:12 GMT

Hi
You have those on indexer(s)/heavy forwarders and your source is probably json?
Do you have on SH side KV_MODE=json definition for those sourcetypes?
r. Ismo

Re: Split sourcetype + indexed fields

michael_vi — Sun, 17 Mar 2024 07:04:17 GMT

Nope,

No JSON. CEF events

Re: Split sourcetype + indexed fields

isoutamo — Mon, 18 Mar 2024 10:21:22 GMT

Can you give some scrambled test events to check this?

topic Re: Split sourcetype + indexed fields in Getting Data In

Split sourcetype + indexed fields

Re: Split sourcetype + indexed fields

Re: Split sourcetype + indexed fields

Re: Split sourcetype + indexed fields