Why query with wildcard works, but not with actual value?

[thomasmuellergr]

If I query with a wildcard, I get the expected result, but if I query with the actual field value, I get no results. Example: I get over 1000 results for the query:

index="..."  splunk_server=* <some more conditions>

Many of the results have pod_name="iwg-k8s-deployment-tom-17-aem-author-c4fdb7875-mb4wc". But if I add that condition to the query (either manually or using the UI), I get no results:

index="..."  splunk_server=* <some more conditions> pod_name="iwg-k8s-deployment-tom-17-aem-author-c4fdb7875-mb4wc"

I do get results (same number as without specifying the field in the query), if I use a wildcard at this location or earlier:

index="..."  splunk_server=* <some more conditions> pod_name="iwg-k8s-deployment-tom-17-aem-author-c4fdb7875*"

index="..."  splunk_server=* <some more conditions> pod_name="iwg-k8s-deployment-tom-17-aem-author-c4fdb78*"

But I get no results if I add the wildcard later, for example:

index="..."  splunk_server=* <some more conditions> pod_name="iwg-k8s-deployment-tom-17-aem-author-c4fdb7875-*"

Also, interesting is the following. Both pod_name = <value> and pod_name != <value> return no results, but removing the condition on pod_name returns the expected results (as initially stated).

What could be the reason?

[thomasmuellergr · Accepted solution]

The solution was to change the format for events to what is described in the Splunk documentation, so that "auto-extraction of fields during search" is not needed.

After changing the event format, everything works as expected!

Before, it looks like search was done on the raw input (possibly based on some kind of fulltext index), and not on the extracted fields. The auto-extraction happened correctly (which confused me), but it happened after search, so searching by field value was somewhat hit-and-miss.

[omera]

Hi Thomas, can you give a detailed explanation on how you changed the format for events? It would be superb if you gave us the splunk docs link. We are experiencing the same issue.

[richgalloway]

@thomasmuellergraf If your problem is resolved, please accept the answer to help future readers.

[anthonymelita]

I've seen similar behavior where in a normal search Splunk is auto-extracting the field name. However when you try to specify the field in the search it seems to happen before the auto-extraction and therefore you get no events because the field doesn't exist and you are requiring it by the search command. You may need to configure a field extraction in that case.
The part about positioning the wildcard is odd and I have not suggestion based on that.