Splunk Search

using foreach at the beginning of the query fetching less records than expected

angelinealex
Communicator
index="index1" tdr=tdr1 OR tdr=tdr2
transaction id
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
| eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != 1 Los !=3 Los!=H LoS!=C
....
....
| table .....

The above code returning 140 records. But the actual records are 200.

index="index1" tdr=tdr1 OR tdr=tdr2
transaction id    
| eval _time=strptime(Qtime,"{%Y-%m-%d %H:%M:%S.%3N}") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != {1} Los !={3} Los!={H} LoS!={C}
....
....
| table .....
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]

The above code is returning 200 records.

Why foreach behaves differently in these two cases?
Please help me to understand the issue.

Tags (3)
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Change line 6 in your first example to

| search LoS != 1 OR isnull(LoS)

and see if they reappear.

angelinealex
Communicator

Its not working. It always has value with {1}

0 Karma

rmuraly
Explorer

In Example 1, Los always has a value = 1, since we strip off {} using foreach at the top before applying the condition check.

0 Karma

micahkemp
Champion

I don't have an answer, but I wonder if it would help you (and us) track down the issue if you look at the events that are not returned, perhaps with a search like this:

 index="index1" tdr=tdr1 OR tdr=tdr2
 transaction id
 | foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
 | eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
 | eval LoS=....
 | search LoS = 1
 | table .....

It's worth pointing out that since we can't see your eval expression for LoS, it's tough to see the full picture of why searching for !=1 and !={1} may return different sets of results.

0 Karma

angelinealex
Communicator

Updated my queries. Please have a look.
tdrs always have the values with {} eg: {H} or {1}

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! &#x1f308; In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...