topic Re: Line breaking issue with [source] stanza (with [sourcetype] stanza working fine in Splunk Search

Line breaking issue with [source] stanza (with [sourcetype] stanza working fine

Naa_Win — Thu, 06 Jul 2023 20:13:56 GMT

We are trying to do custom linebreaking for different types of logs under the same sourcetype using the props below.

The linebreaking in first stanza declared for the sourcetype is working fine, but none of the stanzas for [souce://] are breaking the events correctly, the entire file is getting ingested as a single event.

All the files under this sourcetype are coming in from the same directory, we have tried assigning priorities and deploying it on both forwarder and indexer, but it still doesn’t work.

Have any of you faced a similar issue before? Can you please help us resolve this.

[MY_SRCTYPE]

SHOULD_LINEMERGE=false

LINE_BREAKER=(\~|\r\n)ST\*834\*

NO_BINARY_CHECK=true

TRUNCATE=999999

CHARSET=UTF-8

priority = 1

[source::/mysource/ToSplunk/*.xml.*.edi]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n\s]+)\<Policy\>[\r\n\s]+

NO_BINARY_CHECK=true

TRUNCATE=999999

CHARSET=UTF-8

priority = 5

[source::/mysource/ToSplunk/*.COMPARE.xml.*.edi]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n\s]+)\<CompareMissing\>[\r\n\s]+

NO_BINARY_CHECK=true

TRUNCATE=999999

CHARSET=UTF-8

priority = 6

[source::/mysource/ToSplunk/*.xml.edi]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n\s])+\<Policy\s+

NO_BINARY_CHECK=true

TRUNCATE=999999

CHARSET=UTF-8

priority = 7

[source::/mysource/ToSplunk/*.RCNO*.P.OUT.*]

SHOULD_LINEMERGE=true

LINE_BREAKER=([\r\n]+)

NO_BINARY_CHECK=true

TRUNCATE=999999

CHARSET=UTF-8

priority = 8

Re: Line breaking issue with [source] stanza (with [sourcetype] stanza working fine

isoutamo — Thu, 06 Jul 2023 20:45:12 GMT

this is interesting. Source::xyz always overrides sourcetype on props.conf. Then inside type (source,sourcetype,host) priority set the precedence if I understand correctly what is said on docs.

But maybe there is error on docs and priority also overrides over types? Could you drop priority attributes away from props, restart and try again?

r. Ismo

Re: Line breaking issue with [source] stanza (with [sourcetype] stanza working fine

Naa_Win — Thu, 06 Jul 2023 21:24:21 GMT

Hello @isoutamo

Thanks for your reply, I tried removing the "priority" attributes from the props.conf. I see no events are ingesting into splunk now.

Re: Line breaking issue with [source] stanza (with [sourcetype] stanza working fine

Naa_Win — Thu, 06 Jul 2023 21:53:50 GMT

Also,

 [<spec>] stanzas with [source::<source>] patterns take priority over
stanzas with [host::<host>] and [<sourcetype>] patterns, regardless of their
respective priority key values.

Re: Line breaking issue with [source] stanza (with [sourcetype] stanza working fine

isoutamo — Thu, 06 Jul 2023 22:40:47 GMT

Based on this, there is needs to update docs! Can you send notice/question on props.conf documentation based on your experience?

Can you share log file names and also som content of those?