Getting Data In

Need some help with event breaks

Lazarix
Communicator

Hi all,
I was hoping someone might be able to point me in the right direction for where to set this and how exactly to set it:
I'm consuming some logs from SCCM (log sample below for reference) and they have a distinct pattern for events and event breaks. I know that the regex that i would come up with is supposed to go into either props.conf or transforms.conf but I don't know which.

Sample events:
<![LOG[Active enforcement task with MTC Id a1c5be10-73fe-4d8a-b745-862c63a44faf not found]LOG]!><time="18:31:43.760+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="37360" file="ccitaskmgr.cpp:1054">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/DeploymentType_bbb6820e-f332-4550-91a6-12344b4ed081.3 for target: , consumer {3FC6019A-45DB-4D07-9F1D-E19A7F509482}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/Application_63222d92-85cb-40bd-850a-1f1264d1302c.3 for target: , consumer {E6682089-51BE-4F27-A576-753A8D2B5EF8}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/RequiredApplication_63222d92-85cb-40bd-850a-1f1264d1302c.3 for target: , consumer {901FBA32-A63D-4172-94FA-5C31054BF474}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">

As you can see from the sample above:
Each event "header" starts with <![LOG[ and ends with ]LOG]!>
Each event "detail" starts with < and ends with > and is always preceded by the "header" before it.

So effectively, it's <![LOG[headerhere]LOG]!><detailhere>

My thought was to do a single event as <![LOG[headerhere]LOG]!><detailhere> and from there I could do field extractions for the "header text" and each part of the detail (where something=somethingelse).

Assuming I do the field extractions within splunk instead of props.conf/transforms.conf (best practice now), how should I do the event break?

Should I do something where "the end of every event is "> " and set that as the props.conf LINE_BREAKER ?
Are there any other considerations that I should be aware of, such as SHOULD_LINEMERGE or other things?

Many thanks

0 Karma

somesoni2
Revered Legend

Give this a try as well

props.conf on Indexer/Heavy Forwarder, whichever comes first in data flow.

[YourSourceTypeHere]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)(?=\<\!\[LOG\[)
TIME_PREFIX=time=\"
TIME_FORMAT=%H:%M:%S.%3N%:::z" date="%m-%d-%Y
MAX_TIMESTAMP_LOOKAHEAD=33
0 Karma

Lazarix
Communicator

Same issue as above when I try this config 😞

I wish it were possible to specify the beginning and the end of the event instead

0 Karma

somesoni2
Revered Legend

Does your log entries appear in separate lines or they are merged into big lines?

0 Karma

Lazarix
Communicator

https://i.imgur.com/ufsAGkE.png
The common denominator is the fact that the "event break" as opposed to the "line break" (one would assume that they mean the same thing in this case) should always begin with <![LOG

In the screenshot you will see:
An event being correctly identified
An event being split before ]LOG]!>
Two events in the same event

0 Karma

Lazarix
Communicator

I've modified the regex slightly to remove the CRLF (technically it may miss the first event as it doesn't preceed a CRLF) and the pattern in a regex tester is matching correctly, but still exhibits the same behaviour:

https://www.regextester.com/?fam=99529

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @Lazarix,

Based on your sample data, it looks like you need break event when <! comes in data In that case you can below configuration in props.conf on your Indexer or Heavy Forwarder whichever comes first.

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!

If you want to capture timestamp then you can use below config in props.conf

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
TIME_FORMAT=%H:%M:%S.%3N
TIME_PREFIX=<time="
MAX_TIMESTAMP_LOOKAHEAD=12
0 Karma

Lazarix
Communicator

Thanks!!
I'll give this a try now, wait for some events and let you know

0 Karma

Lazarix
Communicator

It's not quite working 😕

Img

my props.conf is:
[sccm]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=<!

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Ok, I have modified my config slightly, can you please give this try

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
TIME_FORMAT=%H:%M:%S.%3N%:::z" date="%m-%d-%Y
TIME_PREFIX=<time="
MAX_TIMESTAMP_LOOKAHEAD=33

Because I have tried your sample data using Setting->Add Data and while configuring above configuration splunk breaking events perfectly fine.

0 Karma

Lazarix
Communicator

For reference, the REGEX that I had in mind was:
">

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!