Inconsistent Key-Value Pair Searching in Splunk (Some Events Only Found with Wildcards)

[Aithnave]

Hey Splunk team,

I’m facing an issue where Splunk fails to search for certain key-value pairs in some events unless I use wildcards (*) in the value. Here's an example to illustrate the problem:

{
"xxxx_ID": "78901",
"ERROR": "Apples mangos lemons. Banana blackberry blackcurrant blueberry.",
"yyyy_NUM": "123456",
"PROCESS": "orange",
"timestamp": "yyyy-mm-ddThh:mm:ss"
}

Query Examples:
This works (using wildcards):

index="idx_xxxx"  *apples mangos lemons*

These don’t work:
-> index="idx_xxxx"  ERROR="Apples mangos lemons. Banana blackberry blackcurrant blueberry."
-> index="idx_xxxx"  ERROR=*apples mangos lemons*
-> The query below, using regex, does not include all error values trying to find any value after the ERROR key: 

index="idx_xxxx" 
| rex field=_raw "ERROR:\s*(?<error_detail>.+?)(?=;|$)"
| table error_detail

Observations:
Non-Latin characters are not the issue because of other events, for example, Greek text in the ERROR field is searchable without wildcards.
This behavior is inconsistent: some events allow exact matches, but others don’t.
Questions:
Could this issue stem from inconsistencies in the field extraction process?
Are there common pitfalls or misconfigurations during indexing or source-type assignments that might cause such behavior?
How can I debug and verify that the keys and values are properly extracted/indexed?
Any help would be greatly appreciated! Thank you!  ‌‌

[Aithnave · Accepted solution]

Hi!

Thank you for your response 🙂

I made the change below to my query, including the "ERROR" key using regex, and it works properly:

index="idx_xxxx"
| rex field=_raw "\"ERROR\":\"(?<ERROR>[^\"]+)\""

....

[PickleRick]

It might "work" but it doesn't work properly.

With this search you need to read every single event you have in your specified time range just to find the few matching ones.

You need to

a) Define proper extractions in Splunk's configuration or (even better; assuming your events _are_ well-formed jsons)

b) Configure the sourcetype associated with this type of events to use KV_MODE=json

[mattymo]

Hi!

I think you are on the right track with field extraction and it's behaviours. 

The search that works, does so because the search "looks for any match of your fruit string in the _raw event", whereas the ones you are struggling with look for a field value pair, which actually does not exist int the raw event. (there is no "ERROR="). Splunk would have to extract this to recognize it as a field. 

I would start with, what is the sourcetype of this data? Does it have any JSON parsing happening at search time, index time or both. (HINT: kv_mode =json / props.conf / transforms.conf ) 

Easy way to start is..does the Splunk UI recognize this as properly formed JSON and show you it "pretty printed"? Do you see the JSON kv pairs extracted in "interesting fields"?

If not then we would need to extract them to be able to reference the fields and their values.