Splunk Search

How do I search by multiple lines in a log?

blues1990
Explorer

Right now, my search looks like this:

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?201,.*)$"  | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5) | eval ContainerID=mvindex(Msg,13) | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest | table _time ActualDest ContainerID

and the log looks like this:

2016 05 09 12:32:29.000 | SDA written: 201,64,5,1,0,8,0,0,0,0,0,16790
2016 05 09 12:32:29.000 | 5,8,04S05577

I can get the destination that I need, but the container ID, 04S05577, doesn't get listed in the table. Help?

Thanks!

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Give this a try
Updated

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$"  | rex "\|\s+([^,]+),([^,]+),(?<ContainerID>.+)$" | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

Update #2

Feeling positive about this one

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$"  | rex max_match=2 "\|\s+(?<temp>.+)$" 
| eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
| eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

Update #3

 index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$" | rex max_match=2 "([\r\n]+)(?<temp>.+)"  | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
    | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Give this a try
Updated

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$"  | rex "\|\s+([^,]+),([^,]+),(?<ContainerID>.+)$" | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

Update #2

Feeling positive about this one

index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$"  | rex max_match=2 "\|\s+(?<temp>.+)$" 
| eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
| eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

Update #3

 index=4_ip_cnv source="*ATL*Pack*" FirstWord=SDA | rex "\s(?<Msg>201,.*)$" | rex max_match=2 "([\r\n]+)(?<temp>.+)"  | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
    | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDes) as ActualDest values(ContainerID) as ContainerID by _time 

blues1990
Explorer

What I'm thinking now is that we try to isolate by the second line and ignore the first, eliminating the firstword=SDA.
Something like:

index=4_ip_cnv source="*ATL*Pack*" | rex "\s(?201,.*)$"  | rex max_match=2 "\|\s+(?.+)$" 
 | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
 | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDest) as ActualDest values(ContainerID) as ContainerID by _time

This gets closer to working. Now all I would have to do is eliminate any value in ContainerID that does not contain "04S"

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can add following ad the end of currently working search

...| eval ContainerID=mvfilter(match(ContainerID,"04S"))

blues1990
Explorer

THANKS!!! If you're interested, it ended up being:

index=4_ip_cnv source="*ATL*Pack*" | rex "\s(?201,.*)$"  | rex max_match=2 "\|\s+(?.+)$" 
 | eval Msg=split(Msg,",") | eval ActualDest=mvindex(Msg,5)  | eval ActualDest=if(like(SourceName,"%West%"),"West ","East ") . ActualDest 
 | eval temp=mvindex(temp,-1) | eval ContainerID=mvindex(split(temp,","),-1) | stats values(ActualDest) as ActualDest values(ContainerID) as ContainerID by _time| where like(ContainerID,"04%")| eval ContainerID=mvfilter(match(ContainerID,"04"))

blues1990
Explorer

Not quite. It shows

2016-05-10 06:40:14 West 4 1789
2016-05-10 06:40:15 West 4 1790
2016-05-10 06:56:11 West 1 1791
2016-05-10 06:56:12 West 1 1792
2016-05-10 06:56:33 West 1 1793

Which is the correct destination, but does not include the container ID. Rather, it includes the 11th word of the first line.

0 Karma

blues1990
Explorer

Unfortunately, not. Still does not display container ID. I think the misstep lies somewhere in rex "|\d+,\d+,(?\S+)$"

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Looks like I missed taking the space after the pipe symbol. Just fixed the rex. Give that a shot.

0 Karma

blues1990
Explorer

Nope 😕 same result.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

😞
Try the updated query now (made changes to regex)

0 Karma

blues1990
Explorer

That will just give me the entire first line after the 201 limit, I think.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Ok.. I probably wasn't clear in asking earlier. Does both the lines are part of single event?

e.g.
Event 1
016 05 09 12:32:29.000 | SDA written: 201,64,5,1,0,8,0,0,0,0,0,16790
2016 05 09 12:32:29.000 | 5,8,04S05577

Event2
016 05 09 12:32:29.000 | SDA written: 201,64,5,1,0,8,0,0,0,0,0,16790
2016 05 09 12:32:29.000 | 5,8,04S05577

0 Karma

blues1990
Explorer

Yes, both lines are part of a single event, they just are written down into the log in separate lines.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The sample logs that you provided is one log entry OR two? You're using 14th element in the Msg field as ContainerID, but I don't see 14 elements which are separated by comma, So what is the location of ContainerID (is it last element in Msg?).

0 Karma

blues1990
Explorer

It's in a separate line. I want to be able to include it. But you're right, I'm trying to include the 14th element, even though there isn't one. How can I include the second line to display that container ID?

0 Karma

blues1990
Explorer

04S05577 is it, in the second separated line.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

To be able to group these two events together, we need to find a rule/pattern. I don't see any common field between these two events, so can time be the key using which these can be joined together (along with other metadata fields)?

0 Karma

blues1990
Explorer

Time can absolutely be used; these two messages will always show up at the exact same time.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...