topic Re: Regex for multiline events where skipping lines is required. in Getting Data In

Regex for multiline events where skipping lines is required.

TheJagoff — Wed, 09 Apr 2025 14:39:58 GMT

I have multiline events where it is required to capture the error messages.

The events are separated by "FAILED".

I need to capture "Host key verification failed" from the first event, "scp: /logs/rsyslog/server02/: Not a directory" from the second event.

The events:

FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz
Host key verification failed.
lost connection
FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz
You are attempting to access a system owned by XYZ
Provide proper credentials for access
Contact the system administrator for assistance
---This system is monitored---
Details as follows.
scp: /logs/rsyslog/server02/: Not a directory

I can capture the first message with:

FAILED.+\:\s(?<LogFile>.+)(\n)(?<Message>.+(\n).+)

I don't know how to skip to capture the last line of the second event for the Message field.

Any help is most appreciated.

Thank you

Re: Regex for multiline events where skipping lines is required.

livehybrid — Wed, 09 Apr 2025 14:57:17 GMT

Hi @TheJagoff

Im struggling a little to work out the boundaries between the events but I think I might have it now...Just to check - is it the last line in each event that you want to extract? If so the following might work well:

| rex max_match=100 field=_raw "(?m)(?<message>[^\n\r]+)$" | eval last_line = mvindex(message, -1)

Incase its useful for future responses, below is the full example with some makeresults to emulate your events.

|makeresults | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz Host key verification failed. lost connection" | append [|makeresults | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz You are attempting to access a system owned by XYZ Provide proper credentials for access Contact the system administrator for assistance ---This system is monitored--- Details as follows. scp: /logs/rsyslog/server02/: Not a directory"] | rex max_match=100 field=_raw "(?m)(?<message>[^\n\r]+)$" | eval last_line = mvindex(message, -1)

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Regex for multiline events where skipping lines is required.

TheJagoff — Wed, 09 Apr 2025 15:34:01 GMT

@livehybrid - I need the last 2 lines of the first event, and the last line of the second event. I honestly don't know if this is even possible.
The events start with "FAILED to copy checksum for: "

I will work with what you have sent and see what I get for results. Thank you.

Re: Regex for multiline events where skipping lines is required.

ITWhisperer — Wed, 09 Apr 2025 15:57:53 GMT

What distinguishes the first event from the second? Assuming it is a line with "lost connection", you could try something like this

| makeresults | fields - _time | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server01.log.gz Host key verification failed. lost connection" | append [| makeresults | fields - _time | eval _raw="FAILED to copy checksum for: /logs/archives/archived-logs/server02.log.gz You are attempting to access a system owned by XYZ Provide proper credentials for access Contact the system administrator for assistance ---This system is monitored--- Details as follows. scp: /logs/rsyslog/server02/: Not a directory"] | rex "(?m)FAILED to copy checksum for:[^\n]+\n([^\n]+\n)*(?!lost connection)(?<line>[^\n]+(\nlost connection|$))"

Re: Regex for multiline events where skipping lines is required.

livehybrid — Wed, 09 Apr 2025 16:11:03 GMT

Hi @TheJagoff

How about this?

It joins the last 2 lines by a space for event 1 - might need tweaking to add the linebreak back in.

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Regex for multiline events where skipping lines is required.

TheJagoff — Wed, 09 Apr 2025 18:29:37 GMT

@ITWhisperer and @livehybrid . Both responses helped me understand the overall issue and I thank you both.
Another method that I worked on is to use 2 Regex expressions in props.conf:

Regex 1

FAILED.+\:\s(?<LogFile>.+)(\n)(?<Reason1>.+(\n).+)

- that grabs

"Host key verification failed lost connection" OR "You are attempting to access a system owned by XYZ" into the Reason1 field

The second Regex:

Agreement\sfor\sdetails\.(\n)(?<Reason2>.+)

That grabs:

"scp: /logs/rsyslog/server02/: Not a directory" into the Reason2 field

In the search there is a case statement to make it work

| eval Message=case(like(Reason1,"%You are%"),Reason2,1==1,Reason1)

It sounds a bit inefficient, but it is working for the report.

Thank you both again.