Splunk Search

using foreach at the beginning of the query fetching less records than expected

angelinealex
Communicator
index="index1" tdr=tdr1 OR tdr=tdr2
transaction id
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
| eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != 1 Los !=3 Los!=H LoS!=C
....
....
| table .....

The above code returning 140 records. But the actual records are 200.

index="index1" tdr=tdr1 OR tdr=tdr2
transaction id    
| eval _time=strptime(Qtime,"{%Y-%m-%d %H:%M:%S.%3N}") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != {1} Los !={3} Los!={H} LoS!={C}
....
....
| table .....
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]

The above code is returning 200 records.

Why foreach behaves differently in these two cases?
Please help me to understand the issue.

Tags (3)
0 Karma

DalJeanis
Legend

Change line 6 in your first example to

| search LoS != 1 OR isnull(LoS)

and see if they reappear.

angelinealex
Communicator

Its not working. It always has value with {1}

0 Karma

rmuraly
Explorer

In Example 1, Los always has a value = 1, since we strip off {} using foreach at the top before applying the condition check.

0 Karma

micahkemp
Champion

I don't have an answer, but I wonder if it would help you (and us) track down the issue if you look at the events that are not returned, perhaps with a search like this:

 index="index1" tdr=tdr1 OR tdr=tdr2
 transaction id
 | foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
 | eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
 | eval LoS=....
 | search LoS = 1
 | table .....

It's worth pointing out that since we can't see your eval expression for LoS, it's tough to see the full picture of why searching for !=1 and !={1} may return different sets of results.

0 Karma

angelinealex
Communicator

Updated my queries. Please have a look.
tdrs always have the values with {} eg: {H} or {1}

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...