topic Re: Need some help with event breaks in Getting Data In

Need some help with event breaks

Lazarix — Tue, 14 Nov 2017 15:09:11 GMT

Hi all,
I was hoping someone might be able to point me in the right direction for where to set this and how exactly to set it:
I'm consuming some logs from SCCM (log sample below for reference) and they have a distinct pattern for events and event breaks. I know that the regex that i would come up with is supposed to go into either props.conf or transforms.conf but I don't know which.

Sample events:
<![LOG[Active enforcement task with MTC Id a1c5be10-73fe-4d8a-b745-862c63a44faf not found]LOG]!><time="18:31:43.760+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="37360" file="ccitaskmgr.cpp:1054">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/DeploymentType_bbb6820e-f332-4550-91a6-12344b4ed081.3 for target: , consumer {3FC6019A-45DB-4D07-9F1D-E19A7F509482}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/Application_63222d92-85cb-40bd-850a-1f1264d1302c.3 for target: , consumer {E6682089-51BE-4F27-A576-753A8D2B5EF8}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/RequiredApplication_63222d92-85cb-40bd-850a-1f1264d1302c.3 for target: , consumer {901FBA32-A63D-4172-94FA-5C31054BF474}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">

As you can see from the sample above:
Each event "header" starts with <![LOG[ and ends with ]LOG]!>
Each event "detail" starts with < and ends with > and is always preceded by the "header" before it.

So effectively, it's <![LOG[headerhere]LOG]!><detailhere>

My thought was to do a single event as <![LOG[headerhere]LOG]!><detailhere> and from there I could do field extractions for the "header text" and each part of the detail (where something=somethingelse).

Assuming I do the field extractions within splunk instead of props.conf/transforms.conf (best practice now), how should I do the event break?

Should I do something where "the end of every event is "> " and set that as the props.conf LINE_BREAKER ?
Are there any other considerations that I should be aware of, such as SHOULD_LINEMERGE or other things?

Many thanks

Re: Need some help with event breaks

Lazarix — Tue, 14 Nov 2017 15:11:55 GMT

For reference, the REGEX that I had in mind was:
">

Re: Need some help with event breaks

harsmarvania57 — Tue, 14 Nov 2017 15:20:42 GMT

Hi @Lazarix,

Based on your sample data, it looks like you need break event when <! comes in data In that case you can below configuration in props.conf on your Indexer or Heavy Forwarder whichever comes first.

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!

If you want to capture timestamp then you can use below config in props.conf

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
TIME_FORMAT=%H:%M:%S.%3N
TIME_PREFIX=<time="
MAX_TIMESTAMP_LOOKAHEAD=12

Re: Need some help with event breaks

Lazarix — Tue, 14 Nov 2017 15:38:12 GMT

Thanks!!
I'll give this a try now, wait for some events and let you know

Re: Need some help with event breaks

Lazarix — Tue, 29 Sep 2020 16:47:25 GMT

It's not quite working 😕

my props.conf is:
[sccm]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=<!

Re: Need some help with event breaks

somesoni2 — Tue, 14 Nov 2017 16:52:21 GMT

Give this a try as well

props.conf on Indexer/Heavy Forwarder, whichever comes first in data flow.

[YourSourceTypeHere]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)(?=\<\!\[LOG\[)
TIME_PREFIX=time=\"
TIME_FORMAT=%H:%M:%S.%3N%:::z" date="%m-%d-%Y
MAX_TIMESTAMP_LOOKAHEAD=33

Re: Need some help with event breaks

Lazarix — Tue, 14 Nov 2017 17:38:04 GMT

Same issue as above when I try this config 😞

I wish it were possible to specify the beginning and the end of the event instead

Re: Need some help with event breaks

somesoni2 — Tue, 14 Nov 2017 18:42:47 GMT

Does your log entries appear in separate lines or they are merged into big lines?

Re: Need some help with event breaks

Lazarix — Wed, 15 Nov 2017 08:46:27 GMT

https://i.imgur.com/ufsAGkE.png
The common denominator is the fact that the "event break" as opposed to the "line break" (one would assume that they mean the same thing in this case) should always begin with <![LOG

In the screenshot you will see:
An event being correctly identified
An event being split before ]LOG]!>
Two events in the same event

Re: Need some help with event breaks

Lazarix — Wed, 15 Nov 2017 09:07:42 GMT

I've modified the regex slightly to remove the CRLF (technically it may miss the first event as it doesn't preceed a CRLF) and the pattern in a regex tester is matching correctly, but still exhibits the same behaviour:

https://www.regextester.com/?fam=99529

Re: Need some help with event breaks

harsmarvania57 — Wed, 15 Nov 2017 09:16:03 GMT

Ok, I have modified my config slightly, can you please give this try

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
TIME_FORMAT=%H:%M:%S.%3N%:::z" date="%m-%d-%Y
TIME_PREFIX=<time="
MAX_TIMESTAMP_LOOKAHEAD=33

Because I have tried your sample data using Setting->Add Data and while configuring above configuration splunk breaking events perfectly fine.