Find IN does not OUT

[indeed_2000]

Hi
I have log file like this:

 09:04:04.042 module1:  F[6]L: IN
 09:04:01.417 module1:  F[6]L: OUT
 09:04:01.418 module4:  F[6]L: IN
 09:04:01.419 module4:  F[6]L: OUT
 09:04:01.420 module12: F[6]L: IN
 09:04:01.421 module2:  F[6]L: IN
 09:04:01.422 module41: F[6]L: IN
 09:04:01.426 module12: F[6]L: OUT
 09:04:01.427 module50: F[18]L: IN
 09:04:01.428 module52: F[20]L: IN
 09:04:01.429 module50: F[18]L: OUT
 09:04:01.435 module52: F[20]L: OUT

as you see every module had (IN) value after while (OUT).

Now I want to define something to expect (OUT) value for each (IN) due to the Fingerprint and Module.
For example in above log file:
1- group them by F (F value means fingerprint)
2- group them by modules
3- detect any F had (IN) but no (OUT). example module2, module41 with F[6] had IN (input) but never had OUT (output).

Any recommendation?

Thanks,

[niketn · Accepted solution]

@mehrdad_2000 try the following run anywhere example which creates data similar to what you have provided and I have done some extraction required for the solution.

PS: Pipes from | makeresults till | eval _time... is used to generate the data. | dedup 2 fingerprint module is to ensure that we have latest two events for each finderprint and module. If not 2 they bring only one. This may not be required if a fingerprint module can enter the IN state once it has been in the OUT state before. If not then you do not need dedup pipe at all.

| makeresults
| eval data="09:04:04.042 module1:  F[6]L: IN;
09:04:01.417 module1:  F[6]L: OUT;
09:04:01.418 module4:  F[6]L: IN;
09:04:01.419 module4:  F[6]L: OUT;
09:04:01.420 module12: F[6]L: IN;
09:04:01.421 module2:  F[6]L: IN;
09:04:01.422 module41: F[6]L: IN;
09:04:01.426 module12: F[6]L: OUT;
09:04:01.427 module50: F[18]L: IN;
09:04:01.428 module52: F[20]L: IN;
09:04:01.429 module50: F[18]L: OUT;
09:04:01.435 module52: F[20]L: OUT;"
| eval data=replace(data,";(\s+)",";")
| makemv data delim=";"
| mvexpand data
| rename data as _raw
| rex "^(?<Time>[^\s]+)\smodule(?<module>[^\:]+)\:\s+F\[(?<fingerprint>[^\]]+)\]L:\s+(?<state>.*)$"
| eval _time=strptime(Time,"%H:%M:%S.%3N")
| dedup 2 fingerprint module
| stats min(_time) as _time max(_time) as LatestTime values(state) as state by fingerprint module
| eval duration=LatestTime-_time
| search state="IN" AND state!="OUT"

PS: final search filter ensure to pick fingerprint modules which have IN state but not OUT state. You can play around with this search filter to implement other use cases for example finding the duration between State transition from IN to OUT using | search state="IN" AND state="OUT" etc.