With 10 blocks of lines, what is the Splunk command to get only 3 line where line starts from keyword ERROR?

[anitha123gnana]

Block:

2022-02-14 02:30:00,046 [Worker-3] DEBUG User job started
2022-02-14 02:30:00,063 [Worker-3] DEBUG Calling importData
2022-02-14 02:30:00,063 [Worker-3] DEBUG Initializing External DB connection
2022-02-14 02:30:00,063 [Worker-3] ERROR Exception occured
2022-02-14 02:30:00,067 [Worker-3] DEBUG url before binding
2022-02-14 02:30:00,560 [Worker-3] DEBUG inside finally...
2022-02-14 02:30:00,567 [Worker-3] DEBUG sending Notification Email
2022-02-14 02:30:00,567 [Worker-3] DEBUG User job ended
2022-02-14 02:30:00,046 [Worker-3] DEBUG User job started
2022-02-14 02:30:00,063 [Worker-3] DEBUG Calling importData
2022-02-14 02:30:00,063 [Worker-3] DEBUG Initializing External DB connection
2022-02-14 02:30:00,067 [Worker-3] DEBUG url before binding
2022-02-14 02:30:00,560 [Worker-3] DEBUG inside finally...
2022-02-14 02:30:00,567 [Worker-3] DEBUG sending Notification Email
2022-02-14 02:30:00,567 [Worker-3] DEBUG User job ended

Expected output:

2022-02-14 02:30:00,063 [Worker-3] ERROR Exception occured
2022-02-14 02:30:00,067 [Worker-3] DEBUG url before binding
2022-02-14 02:30:00,560 [Worker-3] DEBUG inside finally...
2022-02-14 02:30:00,567 [Worker-3] DEBUG sending Notification Email

Thanks in advance

[gcusello]

Hi @anitha123gnana,

this is one of the few situations in which using transaction command (https://docs.splunk.com/Documentation/Splunk/8.2.6/SearchReference/Transaction),

Please try something like this:

index=your_index (ERROR OR DEBUG)
| transaction startswith="ERROR" maxevents=4

Ciao.

Giuseppe

[anitha123gnana]

Thank you for the quick reply.

But, the block of 10 lines is the result of the below query.

index=my_index   sourcetype="debugLog" | transaction Threadnumber startswith="User job started" endswith="User job ended"

 Threadnumber ->  [Worker-3]  

How do I add logic to this query so that only error messages are extracted from this block?

Thanks in advance

[bowesmana]

Just use your original search and search ERROR after the transaction and then filter out the multi-value event lines from the first errro

index=my_index   sourcetype="debugLog"
| transaction Threadnumber startswith="User job started" endswith="User job ended"
| search "ERROR"
| rex max_match=0 "(?<log>.*ERROR|DEBUG.*)"
| eval log=mvindex(log, mvfind(log, "ERROR"), -1)

Might be another way - but this may work

[anitha123gnana]

I tried this query before but it is not working. 

[bowesmana]

Sorry, I edited the query - not sure which one you tried.

The idea behind mvfind/mvindex is that the rex statement will break raw into a multi-value field called 'log' and then you are just selecting the subset of those events from the ERROR to the last entry.

What didn't work?

[anitha123gnana]

Our expected result is 

• the ERROR row with the following three rows

[gcusello]

Hi @anitha123gnana,

you could try to:

• group events to filter them,
• ungroup them to have again separate events (but already filtered
• to group them again.

something like this:

index=my_index   sourcetype="debugLog" (ERROR OR DEBUG)
| transaction Threadnumber startswith="User job started" endswith="User job ended"
| eval myRaw = _raw
| mvexpand myRaw 
| rename myRaw as _raw
| transaction startswith="ERROR" maxevents=4

Ciao.

Giuseppe

[anitha123gnana]

I'm getting the below output for the query that you have given.

2022-02-14 02:30:00,046 [Worker-3] DEBUG User job started

2022-02-14 02:30:00,063 [Worker-3] DEBUG Calling importData

2022-02-14 02:30:00,063 [Worker-3] DEBUG Initializing External DB connection

2022-02-14 02:30:00,063 [Worker-3] ERROR Exception occured

2022-02-14 02:30:00,063 [Worker-3] DEBUG Calling importData

2022-02-14 02:30:00,063 [Worker-3] DEBUG Initializing External DB connection

2022-02-14 02:30:00,067 [Worker-3] DEBUG url before binding

2022-02-14 02:30:00,067 [Worker-3] DEBUG url before binding

2022-02-14 02:30:00,560 [Worker-3] DEBUG inside finally...

2022-02-14 02:30:00,560 [Worker-3] DEBUG inside finally...

2022-02-14 02:30:00,567 [Worker-3] DEBUG sending Notification Email

2022-02-14 02:30:00,567 [Worker-3] DEBUG User job ended

But, the expected output is only the error message from that transaction.

2022-02-14 02:30:00,063 [Worker-3] ERROR Exception occured

Thanks in advance.

[gcusello]

Hi @anitha123gnana,

let me understand, do you want:

• only one row with the ERROR string,
• the ERROR row with the following three rows

?

if the second, my search should be correct (using maxevents=4 option).

If the first you don't need the transaction command.

Ciao.

Giuseppe

[anitha123gnana]

Sorry, Our expected result is 

• the ERROR row with the following three rows

But, the below query is returning the first transaction block as result . The second transaction command is not working.

index=my_index sourcetype="debugLog" (ERROR OR DEBUG)
| transaction Threadnumber startswith="User job started" endswith="User job ended"
| eval myRaw = _raw
| mvexpand myRaw
| rename myRaw as _raw
| transaction startswith="ERROR" maxevents=4

Thanks in advance

[gcusello]

Hi @anitha123gnana,

what does it happen if you use the simple search

index=my_index   sourcetype="debugLog" (ERROR OR DEBUG)
| transaction startswith="ERROR" maxevents=4

Ciao.

Giuseppe

[anitha123gnana]

Simple search does not work as we have many jobs running in multiple threads recorded in the log file. We are targeting a specific job in one thread and try to extract first 4 lines of its error messages.