how to count loglines without corresponding second loglines?

[avanaschen]

I generate logline when starting processing 1 object and another logline when ready.
How to find logline1 without a logline2 with SPLUNK?

I have to generate a list of objects, for which I have a logline when starting processing the object, but for which I do not have a logline generated after processing the object.

Can this be done with SPLUNK and how much performance will this cost?

Thanks,
Albert

[DalJeanis]

Try this ...

index=foo "ID=*" "processing" ("Start" OR "End") 
| rex "ID=(?<IDNo>\d+)\s+(?<TypeRec>Start|End) processing"
| stats min(_time) as _time values(TypeRec) as TypeRec by IDNo
| where mvcount(TypeRec)<2
| where TypeRec="Start"

[woodcock]

Like this:

Your Base Search Here
| streamstats count(eval(searchmatch("StringThatMatchesOnlyYourStartEventHereInSPL"))) AS sessionID
| eventstats count(eval(searchmatch("StringThatMatchesOnlyYourStopEventHereInSPL"))) AS numStopEvents
| search numStopEvents!=1

This shows sequences of events that have either 0 or more than 1 Stop Events.
You can replace searchmatch calls with another function that might be easier for you such as match (if you are good with RegEx) or like (if you are good with SQL).

[avanaschen]

Before I forget to mention, I have to process at least 2.000.000 objects a day. So I will have 2M "start processing" and I have to find how many of those do not have an "end processing" in de logfile.

I Have 1 logfile.
when starting processing the object I write a log entry "2017-04-04 13:00:59 : ID=14; Start processing
When complete work on the object I write a log entry "2017-04-04 13:01:00 ; ID=14: End processing.

If something failes, the "end processing" line will not be generated.

File might look like:
2017-04-04 13:00:59 : ID=14; Start processing
2017-04-04 13:00:59 : ID=15; Start processing
2017-04-04 13:01:00 ; ID=14: End processing
2017-04-04 13:01:15 : ID=16; Start processing
2017-04-04 13:01:20 ; ID=16: End processing

processing ID 15 failed, so not line is being generated. That is what I have to find with SPLUNK.
Ofcourse, I will generated an error in the logfile but I have to prove that all start processing have an end processing. It is functional requirement.

[richgalloway]

Can you share some sample events?