expect value

[indeed_2000]

Hi
I have log file like this:

09:04:04.042 module1:  F[6]L: IN
09:04:01.417 module1:  F[6]L: OUT
09:04:01.418 module4:  F[6]L: IN
09:04:01.419 module4:  F[6]L: OUT
09:04:01.420 module12: F[6]L: IN
09:04:01.421 module2:  F[6]L: IN
09:04:01.422 module41: F[6]L: IN
09:04:01.426 module12: F[6]L: OUT
09:04:01.427 module50: F[18]L: IN
09:04:01.428 module52: F[20]L: IN
09:04:01.429 module50: F[18]L: OUT
09:04:01.435 module52: F[20]L: OUT

as you see every module had (IN) value after while (OUT).

Now I want to define something to expect (OUT) value for each (IN) due to the Fingerprint and Module.
For example in above log file:
1- group them by F (F value means fingerprint)
2- group them by modules
3- detect any F had (IN) but no (OUT). example module2, module41 with F[6] had IN (input) but never had OUT (output).

Any recommendation?

Thanks,

[woodcock]

Like this:

| makeresults
| eval _raw="Time,module,fingerprint,direction
9:04:04.042,module1,F[6]L,IN
09:04:01.437,module1,F[6]L,OUT
09:04:01.427,module4,F[6]L,IN
09:04:01.422,module4,F[6]L,OUT
09:04:01.381,module12,F[6]L,IN
09:04:01.371,module2,F[6]L,IN
09:04:01.338,module41,F[6]L,IN
09:04:01.381,module12,F[6]L,OUT
09:04:01.338,module50,F[18]L,IN
09:04:01.381,module52,F[20]L,IN
09:04:01.338,module50,F[18]L,OUT
09:04:01.381,module52,F[20]L,OUT"
| multikv forceheader=1
| eval _time = strptime(Time, "%H:%M:%S.%3N")
| sort 0 - _time

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| stats dc(direction) AS dc values(direction) AS directions BY fingerprint module
| where dc<2

[indeed_2000]

Thank you @woodcock this is exactly what i want.

[indeed_2000]

run your solution on this part of log:
09:04:01.042,module1,F[6]L,IN
09:04:01.437,module1,F[6]L,OUT
09:04:01.438,module4,F[6]L,IN
09:04:01.439,module4,F[6]L,OUT
09:04:01.481,module2,F[6]L,IN
09:04:01.482,module2,F[6]L,IN
09:04:01.483,module2,F[6]L,IN
09:04:01.484,module2,F[6]L,OUT
09:04:01.485,module2,F[6]L,OUT
09:04:01.488,module50,F[18]L,IN
09:04:01.489,module52,F[20]L,IN
09:04:01.490,module53,F[18]L,OUT
09:04:01.491,module52,F[20]L,OUT"

result:
09:04:01.488,module50,F[18]L,IN
09:04:01.490,module53,F[18]L,OUT

expected result:
09:04:01.483,module2,F[6]L,IN
09:04:01.488,module50,F[18]L,IN
09:04:01.490,module53,F[18]L,OUT

[gcusello]

Hi @mehrdad_2000,
you could run something like this:

index=my_index
| rex "^\d+:\d+:\d+\.\d+\s+(?<module>[^:]*):\s+(?<fingerprint>[^:]*):\s+(?<value>\w+)"
| stats dc(value) AS dc_values values(value) AS value BY module fingerprint
| where dc_values<2 AND value="IN"
| table module fingerprint

Ciao and Happy New Year.
Giuseppe

[indeed_2000]

Thank you and Happy New Year 🙂
this is large file I can't define dc value, it has lots of this kind of events.

I expect every single value that hadn't output.

Any idea?

[gcusello]

Hi @mehrdad_2000,
if you have many events it isn't a problem!
if you could have more than 2 events for the same module and fingerprint, you have to use the transaction command that's a very slow command

 index=my_index
 | rex "^\d+:\d+:\d+\.\d+\s+(?<module>[^:]*):\s+(?<fingerprint>[^:]*):\s+(?<value>\w+)"
 | transaction module fingerprint startswith=": IN"
 | where eventcount<2
 | table module fingerprint

Ciao and Happy New Year.
Giuseppe

[indeed_2000]

run your solution on this part of log:

 09:04:01.042,module1,F[6]L,IN
 09:04:01.437,module1,F[6]L,OUT
 09:04:01.438,module4,F[6]L,IN
 09:04:01.439,module4,F[6]L,OUT
 09:04:01.481,module2,F[6]L,IN
 09:04:01.482,module2,F[6]L,IN
 09:04:01.483,module2,F[6]L,IN
 09:04:01.484,module2,F[6]L,OUT
 09:04:01.485,module2,F[6]L,OUT
 09:04:01.488,module50,F[18]L,IN
 09:04:01.489,module52,F[20]L,IN
 09:04:01.490,module53,F[18]L,OUT
 09:04:01.491,module52,F[20]L,OUT"

result:
09:04:01.481,module2,F[6]L,IN
09:04:01.482,module2,F[6]L,IN
09:04:01.488,module50,F[18]L,IN

expected result:
09:04:01.483,module2,F[6]L,IN
09:04:01.488,module50,F[18]L,IN
09:04:01.490,module53,F[18]L,OUT