topic Why does this indexed field search give the wrong results when looking for specific values? in Splunk Enterprise Security

Why does this indexed field search give the wrong results when looking for specific values?

socconsulting — Fri, 07 Dec 2018 08:44:47 GMT

Hi,

We have an heavy forwarder in every location.
At the HF have an indexed field (meta) called "site-id" that gets added to each event via props/transforms (Regex = .*)

If I now do a search:

index="my_index" site-id="*" (verbose, 24 h)

As a result, I get a count of 122565 events and if I clock on the field "site-id" it shows a distribution of 100% and only 1 value "my_value"

Now, the strange behavior starts:

If I click on the field and add it to the search with the one value that exists

index="my_index" site-id="my_value" (verbose, 24 h)

I only get 47 results

If I do

index="my_index" site-id="my_value*" (verbose, 24 h)

I get the 122565 results again

There are no hidden character or anything at that values I exported it and looked at the character coding only "LF"

I even tried the following two searches to see if there is any difference:

index="my_index" site-id="*"| strcat site-id ":TEST" new_site_id | search new_site_id="my_value:TEST" | stats count by new_site_id
gives me the result result: count 122565
index="my_index" site-id="my_value"| strcat site-id ":TEST" new_site_id | search new_site_id="my_value:TEST" | stats count by 
new_site_id

gives me the result: count 47

search ... | fieldsummary site-id gives a count of 122565 a singe value of my_value and a dc=1

Why can't I search for site-id="my_value" and get the 122565 results?

Please any ideas?

Best
Michael

Re: Why does this indexed field search give the wrong results when looking for specific values?

prakash007 — Fri, 07 Dec 2018 20:11:56 GMT

When you run this search, did you see any other additional sourcetypes/sources/hosts...??

index="my_index" site-id="my_value*" (verbose, 24 h)

Re: Why does this indexed field search give the wrong results when looking for specific values?

socconsulting — Tue, 29 Sep 2020 22:22:59 GMT

No just the expected one.
We now even tried to change the added metafield from site-id to site_id to see if the "-" was not accepted by splunk but that did not change anything. We than added a fields.conf for the search head and the indexer cluster like:

[site_id]
INDEXED = true
INDEXED_VALUE = false

[site-id]
INDEXED = true
INDEXED_VALUE = false

We tried with and without the "INDEXED_VALUES" attribute without any difference.

Re: Why does this indexed field search give the wrong results when looking for specific values?

socconsulting — Tue, 29 Sep 2020 22:23:07 GMT

SOLUTION:

The field was not known to the search head in the context of the search app / system
So we added a metadata export description on the search head inside our fields.conf

my_fields_app/metadata/default.meta
[]
access = read : [ * ], write : [ admin ]
export = system

Now everything works like expected