topic Re: using foreach at the beginning of the query fetching less records than expected in Splunk Search

using foreach at the beginning of the query fetching less records than expected

angelinealex — Fri, 27 Apr 2018 01:27:54 GMT

index="index1" tdr=tdr1 OR tdr=tdr2
transaction id
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
| eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != 1 Los !=3 Los!=H LoS!=C
....
....
| table .....

The above code returning 140 records. But the actual records are 200.

index="index1" tdr=tdr1 OR tdr=tdr2
transaction id    
| eval _time=strptime(Qtime,"{%Y-%m-%d %H:%M:%S.%3N}") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != {1} Los !={3} Los!={H} LoS!={C}
....
....
| table .....
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]

The above code is returning 200 records.

Why foreach behaves differently in these two cases?
Please help me to understand the issue.

Re: using foreach at the beginning of the query fetching less records than expected

micahkemp — Fri, 27 Apr 2018 01:53:32 GMT

I don't have an answer, but I wonder if it would help you (and us) track down the issue if you look at the events that are not returned, perhaps with a search like this:

 index="index1" tdr=tdr1 OR tdr=tdr2
 transaction id
 | foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
 | eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
 | eval LoS=....
 | search LoS = 1
 | table .....

It's worth pointing out that since we can't see your eval expression for LoS, it's tough to see the full picture of why searching for !=1 and !={1} may return different sets of results.

Re: using foreach at the beginning of the query fetching less records than expected

DalJeanis — Fri, 27 Apr 2018 03:09:35 GMT

Change line 6 in your first example to

| search LoS != 1 OR isnull(LoS)

and see if they reappear.

Re: using foreach at the beginning of the query fetching less records than expected

angelinealex — Fri, 27 Apr 2018 16:41:08 GMT

Updated my queries. Please have a look.
tdrs always have the values with {} eg: {H} or {1}

Re: using foreach at the beginning of the query fetching less records than expected

angelinealex — Fri, 27 Apr 2018 17:19:07 GMT

Its not working. It always has value with {1}

Re: using foreach at the beginning of the query fetching less records than expected

rmuraly — Tue, 01 May 2018 16:29:55 GMT

In Example 1, Los always has a value = 1, since we strip off {} using foreach at the top before applying the condition check.