Splunk Search

using foreach at the beginning of the query fetching less records than expected

Communicator
index="index1" tdr=tdr1 OR tdr=tdr2
transaction id
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
| eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != 1 Los !=3 Los!=H LoS!=C
....
....
| table .....

The above code returning 140 records. But the actual records are 200.

index="index1" tdr=tdr1 OR tdr=tdr2
transaction id    
| eval _time=strptime(Qtime,"{%Y-%m-%d %H:%M:%S.%3N}") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != {1} Los !={3} Los!={H} LoS!={C}
....
....
| table .....
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]

The above code is returning 200 records.

Why foreach behaves differently in these two cases?
Please help me to understand the issue.

Tags (3)
0 Karma

SplunkTrust
SplunkTrust

Change line 6 in your first example to

| search LoS != 1 OR isnull(LoS)

and see if they reappear.

Communicator

Its not working. It always has value with {1}

0 Karma

Explorer

In Example 1, Los always has a value = 1, since we strip off {} using foreach at the top before applying the condition check.

0 Karma

Champion

I don't have an answer, but I wonder if it would help you (and us) track down the issue if you look at the events that are not returned, perhaps with a search like this:

 index="index1" tdr=tdr1 OR tdr=tdr2
 transaction id
 | foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
 | eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
 | eval LoS=....
 | search LoS = 1
 | table .....

It's worth pointing out that since we can't see your eval expression for LoS, it's tough to see the full picture of why searching for !=1 and !={1} may return different sets of results.

0 Karma

Communicator

Updated my queries. Please have a look.
tdrs always have the values with {} eg: {H} or {1}

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!