Getting Data In

How to edit props.conf to line merge a set of results?

Sayanta_Basak_I
Explorer

Hello

I have below set of line events(repeating) which I want to convert to single event. For every 6 events I want to convert it to 1 event, viz below

07/24/2017 16:16:31 host=myhost SMB1Enabled=0x0
07/24/2017 16:16:31 host=host=myhost PSExecEnabled=0x0
07/24/2017 16:16:31 host=host=myhost SymantecDefsDate=2017-07-24
07/24/2017 16:16:31 host=host=myhost SystemInfected=0x0
07/24/2017 16:16:31 host=host=myhost SymantecRunningStatus=0x1
07/24/2017 16:16:31 host=host=myhost SymantecProductionVersion=12.1.5337.5000

To go ahead with a multi line merge should I go ahead with below config change in props.conf

[my_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE=([\r\n]+)(?=\d{1,2}\/\d{1,2}\/\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}\shost=/^\w{1,20}$/\sSMB1Enabled)
MAX_EVENTS=6

But it does not seem to work, Help would be appreciated
@lguinn @bwooden @kristian.kolb @woodcock

Regards
Sayanta B

0 Karma

bwooden
Splunk Employee
Splunk Employee

It would be easier at search time. From the small data sample we can see it would likely make the most sense to have each line with its own timestamp indexed as a separate event and then stitch transactions together in ad hoc searches or reports using the stats command or transaction command.

0 Karma

woodcock
Esteemed Legend

Do the lines to merge really have host=host=? If so, that will make it very easy.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Try using BREAK_ONLY_BEFORE=^\d{1,2}\/\d{1,2}\/\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}\shost=.*\sSMB1Enabled.
No need to use capture groups or the line feed, if you anchor to the beginning of the line.

0 Karma

Sayanta_Basak_I
Explorer

Thank you @ssievert

Adding the below in props.conf seems to do its work to some extent

[my_sourcetype]
BREAK_ONLY_BEFORE = SMB1Enabled

Now the events are coming in below format. So its considering the default line break post the SMB1* line and then again waiting for the next matching regex. How to do it differently to merge all 6 events as part of 1

alt text

@lguinn @bwooden @kristian.kolb @woodcock

0 Karma

bwooden
Splunk Employee
Splunk Employee

I was able to get it working using this props config, though admittedly I don't have the full data set.

SHOULD_LINEMERGE=FALSE
LINE_BREAKER=([\r\n]+)\d{2}\/\d{2}\/\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}\shost=.*\sSMB\dEnabled
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!