How do I search by multiple lines in a log?

[blues1990]

Right now, my search looks like this:

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?201,.*)$"  | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5) | eval ContainerID=mvindex(Msg,13) | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest | table _time ActualDest ContainerID

and the log looks like this:

2016 05 09 12:32:29.000 | SDA written: 201,64,5,1,0,8,0,0,0,0,0,16790
2016 05 09 12:32:29.000 | 5,8,04S05577

I can get the destination that I need, but the container ID, 04S05577, doesn't get listed in the table. Help?

Thanks!

[somesoni2 · Accepted solution]

Give this a try
Updated

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$"  | rex "\|\s+([^,]+),([^,]+),(?<ContainerID>.+)$" | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

Update #2

Feeling positive about this one

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$"  | rex max_match=2 "\|\s+(?<temp>.+)$" 
| eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
| eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

Update #3

 index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$" | rex max_match=2 "([\r\n]+)(?<temp>.+)"  | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
    | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

[blues1990]

What I'm thinking now is that we try to isolate by the second line and ignore the first, eliminating the firstword=SDA.
Something like:

index=4_ip_cnv source="*ATL*Pack*" | rex "\s(?201,.*)$"  | rex max_match=2 "\|\s+(?.+)$" 
 | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
 | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDest) as ActualDest values(ContainerID) as ContainerID by _time

This gets closer to working. Now all I would have to do is eliminate any value in ContainerID that does not contain "04S"

[somesoni2]

You can add following ad the end of currently working search

...| eval ContainerID=mvfilter(match(ContainerID,"04S"))

[blues1990]

THANKS!!! If you're interested, it ended up being:

index=4_ip_cnv source="*ATL*Pack*" | rex "\s(?201,.*)$"  | rex max_match=2 "\|\s+(?.+)$" 
 | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
 | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDest) as ActualDest values(ContainerID) as ContainerID by _time| where like(ContainerID,"04%")| eval ContainerID=mvfilter(match(ContainerID,"04"))

[blues1990]

Not quite. It shows

2016-05-10 06:40:14 West 4 1789
2016-05-10 06:40:15 West 4 1790
2016-05-10 06:56:11 West 1 1791
2016-05-10 06:56:12 West 1 1792
2016-05-10 06:56:33 West 1 1793

Which is the correct destination, but does not include the container ID. Rather, it includes the 11th word of the first line.

[blues1990]

Unfortunately, not. Still does not display container ID. I think the misstep lies somewhere in rex "|\d+,\d+,(?\S+)$"

[somesoni2]

Looks like I missed taking the space after the pipe symbol. Just fixed the rex. Give that a shot.

[blues1990]

Nope 😕 same result.

[somesoni2]

😞
Try the updated query now (made changes to regex)

[blues1990]

That will just give me the entire first line after the 201 limit, I think.

[somesoni2]

Ok.. I probably wasn't clear in asking earlier. Does both the lines are part of single event?

e.g.
Event 1
016 05 09 12:32:29.000 | SDA written: 201,64,5,1,0,8,0,0,0,0,0,16790
2016 05 09 12:32:29.000 | 5,8,04S05577

Event2
016 05 09 12:32:29.000 | SDA written: 201,64,5,1,0,8,0,0,0,0,0,16790
2016 05 09 12:32:29.000 | 5,8,04S05577

[blues1990]

Yes, both lines are part of a single event, they just are written down into the log in separate lines.

[somesoni2]

The sample logs that you provided is one log entry OR two? You're using 14th element in the Msg field as ContainerID, but I don't see 14 elements which are separated by comma, So what is the location of ContainerID (is it last element in Msg?).

[blues1990]

It's in a separate line. I want to be able to include it. But you're right, I'm trying to include the 14th element, even though there isn't one. How can I include the second line to display that container ID?

[blues1990]

04S05577 is it, in the second separated line.

[somesoni2]

To be able to group these two events together, we need to find a rule/pattern. I don't see any common field between these two events, so can time be the key using which these can be joined together (along with other metadata fields)?

[blues1990]

Time can absolutely be used; these two messages will always show up at the exact same time.