topic Re: How to include events with non existing fields in a search string ? in Splunk Search

How to include events with non existing fields in a search string ?

AmineTN — Tue, 01 Aug 2023 15:09:27 GMT

After fixing filters on some fields that don't exist in all the events, I tried to apply these filters on the graphs and the problem here is that when Splunk reads the search string of a graph, it gets only the events where the fields exist and it excludes the other events. As a result all the statistics and the graphs are wrong !!

Ayone has a solution please ??

Thanks in Advance.

Re: How to include events with non existing fields in a search string ?

ITWhisperer — Tue, 01 Aug 2023 15:39:26 GMT

It is not clear to me what it is you are trying to do - can you give an example of the search you are doing and where the filters are defined and used? If the statistics are incorrect, are the filters being applied at the correct point in the search?

Re: How to include events with non existing fields in a search string ?

AmineTN — Wed, 02 Aug 2023 09:51:26 GMT

I will explain more here with some examples.

For these two JSON events the first contain the field "SuperForceCouponStatus" but the second don't.

Then I made a filter for "SuperForceCouponStatus" as following :

<input type="dropdown" token="filter_SuperForce">
<label>SuperForce</label>
<choice value="*">ALL</choice>
<prefix>SuperForceCouponStatus="</prefix>
<suffix>"</suffix>
<fieldForLabel>SuperForceCouponStatus</fieldForLabel>
<fieldForValue>SuperForceCouponStatus</fieldForValue>
<search>
<query>|tstats count where index=$filter_index$ $filter_backend$ 
by SuperForceCouponStatus</query>
<earliest></earliest>
<latest>now</latest>
</search>
<default>*</default>
</input>

After that I made a graph to display the number of events depending on the filters qs following :

<panel>
<single>
<title>Total number of Transactions</title>
<search>
<query>| tstats count where index=$filter_index$
$filter_backend$
$filter_status$
$filter_SuperForce$ |</query>
<earliest>$time_range.earliest$</earliest>
<latest>$time_range.latest$</latest>
</search>
<option name="colorBy">value</option>
<option name="colorMode">block</option>
<option name="drilldown">none</option>
<option name="numberPrecision">0</option>
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
<option name="rangeValues">[100,10000,1000000,10000000]</option>
<option name="refresh.display">progressbar</option>
<option name="showSparkline">1</option>
<option name="showTrendIndicator">1</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
<option name="trendColorInterpretation">standard</option>
<option name="trendDisplayMode">absolute</option>
<option name="underLabel">transactions</option>
<option name="unitPosition">after</option>
<option name="useColors">1</option>
<option name="useThousandSeparators">1</option>
</single>
</panel>

And the problem is that this graph takes only the events that have the field "SuperForceCouponStatus" because $filter_SuperForce$ is applied in its search string !!

Hope the problem is more clear now 🙂

Re: How to include events with non existing fields in a search string ?

gcusello — Wed, 02 Aug 2023 09:53:23 GMT

Hi @AmineTN,

did yu tried to add a fillnull to give a value (e.g. "N.A. " or "-") to this field when there isn't any value?

| fillnull value="N.A." SuperForceCouponStatus

Ciao.

Giuseppe

Re: How to include events with non existing fields in a search string ?

ITWhisperer — Wed, 02 Aug 2023 10:03:10 GMT

So, essentially, your ALL option is filtering out all stats events with null SuperForceCouponStatus. Try changing the value of the drop down options to include the condition or an empty string when ALL is selected. Something like this

<input type="dropdown" token="filter_SuperForce"> <label>SuperForce</label> <choice value="">ALL</choice> <fieldForLabel>SuperForceCouponStatus</fieldForLabel> <fieldForValue>SuperForceCouponStatusFilter</fieldForValue> <search> <query>|tstats count where index=$filter_index$ $filter_backend$ by SuperForceCouponStatus | eval SuperForceCouponStatusFilter="SuperForceCouponStatus=\"".SuperForceCouponStatus."\""</query> <earliest></earliest> <latest>now</latest> </search> <default></default> </input>

Re: How to include events with non existing fields in a search string ?

AmineTN — Wed, 02 Aug 2023 10:14:30 GMT

Thanks a lot @ITWhisperer , It's working perfectly !!!

Re: How to include events with non existing fields in a search string ?

AmineTN — Wed, 02 Aug 2023 10:16:51 GMT

Yes I tried and the problem the fillnull command is applied after filtering the events, So it's useless !!

Thanks for your try ❤️

Re: How to include events with non existing fields in a search string ?

gcusello — Wed, 02 Aug 2023 10:27:38 GMT

Hi @AmineTN ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉