topic Re: how to count loglines without corresponding second loglines? in Splunk Search

how to count loglines without corresponding second loglines?

avanaschen — Tue, 25 Jul 2017 12:43:42 GMT

I generate logline when starting processing 1 object and another logline when ready.
How to find logline1 without a logline2 with SPLUNK?

I have to generate a list of objects, for which I have a logline when starting processing the object, but for which I do not have a logline generated after processing the object.

Can this be done with SPLUNK and how much performance will this cost?

Thanks,
Albert

Re: how to count loglines without corresponding second loglines?

richgalloway — Tue, 25 Jul 2017 13:02:06 GMT

Can you share some sample events?

Re: how to count loglines without corresponding second loglines?

avanaschen — Tue, 25 Jul 2017 13:31:17 GMT

Before I forget to mention, I have to process at least 2.000.000 objects a day. So I will have 2M "start processing" and I have to find how many of those do not have an "end processing" in de logfile.

I Have 1 logfile.
when starting processing the object I write a log entry "2017-04-04 13:00:59 : ID=14; Start processing
When complete work on the object I write a log entry "2017-04-04 13:01:00 ; ID=14: End processing.

If something failes, the "end processing" line will not be generated.

File might look like:
2017-04-04 13:00:59 : ID=14; Start processing
2017-04-04 13:00:59 : ID=15; Start processing
2017-04-04 13:01:00 ; ID=14: End processing
2017-04-04 13:01:15 : ID=16; Start processing
2017-04-04 13:01:20 ; ID=16: End processing

processing ID 15 failed, so not line is being generated. That is what I have to find with SPLUNK.
Ofcourse, I will generated an error in the logfile but I have to prove that all start processing have an end processing. It is functional requirement.

Re: how to count loglines without corresponding second loglines?

woodcock — Tue, 25 Jul 2017 15:15:14 GMT

Like this:

Your Base Search Here
| streamstats count(eval(searchmatch("StringThatMatchesOnlyYourStartEventHereInSPL"))) AS sessionID
| eventstats count(eval(searchmatch("StringThatMatchesOnlyYourStopEventHereInSPL"))) AS numStopEvents
| search numStopEvents!=1

This shows sequences of events that have either 0 or more than 1 Stop Events.
You can replace searchmatch calls with another function that might be easier for you such as match (if you are good with RegEx) or like (if you are good with SQL).

Re: how to count loglines without corresponding second loglines?

DalJeanis — Tue, 25 Jul 2017 21:15:55 GMT

Try this ...

index=foo "ID=*" "processing" ("Start" OR "End") 
| rex "ID=(?<IDNo>\d+)\s+(?<TypeRec>Start|End) processing"
| stats min(_time) as _time values(TypeRec) as TypeRec by IDNo
| where mvcount(TypeRec)<2
| where TypeRec="Start"