Regex for multiline events where skipping lines is required.

[TheJagoff]

I have multiline events where it is required to capture the error messages.

The events are separated by "FAILED".

I need to capture "Host key verification failed" from the first event, "scp: /logs/rsyslog/server02/: Not a directory" from the second event.

The events:

FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz
Host key verification failed.
lost connection
FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz
You are attempting to access a system owned by XYZ
Provide proper credentials for access
Contact the system administrator for assistance
---This system is monitored---
Details as follows.
scp: /logs/rsyslog/server02/: Not a directory

I can capture the first message with:

FAILED.+\:\s(?<LogFile>.+)(\n)(?<Message>.+(\n).+)

I don't know how to skip to capture the last line of the second event for the Message field.

Any help is most appreciated.

Thank you

[TheJagoff]

@ITWhisperer and @livehybrid . Both responses helped me understand the overall issue and I thank you both.  
Another method that I worked on is to use 2 Regex expressions in props.conf:

Regex 1

FAILED.+\:\s(?<LogFile>.+)(\n)(?<Reason1>.+(\n).+)         

- that grabs

"Host key verification failed lost connection" OR "You are attempting to access a system owned by XYZ" into the Reason1 field

The second Regex:

 Agreement\sfor\sdetails\.(\n)(?<Reason2>.+)

That grabs:

"scp: /logs/rsyslog/server02/: Not a directory" into the Reason2 field

In the search there is a case statement to make it work

| eval Message=case(like(Reason1,"%You are%"),Reason2,1==1,Reason1)

It sounds a bit inefficient, but it is working for the report.

Thank you both again.

[ITWhisperer]

What distinguishes the first event from the second? Assuming it is a line with "lost connection", you could try something like this

| makeresults
| fields - _time
| eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz
Host key verification failed.
lost connection"
| append [| makeresults
| fields - _time
| eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz
You are attempting to access a system owned by XYZ
Provide proper credentials for access
Contact the system administrator for assistance
---This system is monitored---
Details as follows.
scp: /logs/rsyslog/server02/: Not a directory"]
| rex "(?m)FAILED to copy checksum for:[^\n]+\n([^\n]+\n)*(?!lost connection)(?<line>[^\n]+(\nlost connection|$))"

[TheJagoff]

@livehybrid  - I need the last 2 lines of the first event, and the last line of the second event. I honestly don't know if this is even possible.
The events start with "FAILED to copy checksum for: "

I will work with what you have sent and see what I get for results. Thank you.

[livehybrid]

Hi @TheJagoff 

How about this? 

|makeresults | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz
Host key verification failed.
lost connection"
| append [|makeresults | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz
You are attempting to access a system owned by XYZ
Provide proper credentials for access
Contact the system administrator for assistance
---This system is monitored---
Details as follows.
scp: /logs/rsyslog/server02/: Not a directory"]
| rex max_match=100 field=_raw "(?m)(?<message>[^\n\r]+)$"
| eval last_line = if(typeof(mvfind(message,"Details as follows"))=="Number","", mvindex(message,-2))+" "+mvindex(message, -1)

It joins the last 2 lines by a space for event 1 - might need tweaking to add the linebreak back in.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid]

Hi @TheJagoff 

Im struggling a little to work out the boundaries between the events but I think I might have it now...Just to check - is it the last line in each event that you want to extract? If so the following might work well:

| rex max_match=100 field=_raw "(?m)(?<message>[^\n\r]+)$"
| eval last_line = mvindex(message, -1)

Incase its useful for future responses, below is the full example with some makeresults to emulate your events.

|makeresults | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz
Host key verification failed.
lost connection"
| append [|makeresults | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz
You are attempting to access a system owned by XYZ
Provide proper credentials for access
Contact the system administrator for assistance
---This system is monitored---
Details as follows.
scp: /logs/rsyslog/server02/: Not a directory"]
| rex max_match=100 field=_raw "(?m)(?<message>[^\n\r]+)$"
| eval last_line = mvindex(message, -1)

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing