I know what the issue is. It is explained here:
https://www.splunk.com/en_us/blog/tips-and-tricks/cannot-search-based-on-an-extracted-field.html
Here is the way I understand it. Splunk automatically indexes word tokens, which are detected using "standard" delimiters like spaces, tabs, commas, etc.
Here is what most people don't know or understand: even though you specify a search like so (in your example):
index=blah Service="examplename"
What Splunk actually does initially is a search like this:
index=blah "examplename"
Under normal circumstances, this works fine - and it normally returns a "super set" of the results you are looking for. Then, Splunk refines the results further with a search like this:
| search Service="examplename"
Which should reduce the data set down to the specific results you are looking for.
What went wrong here? Well, your data doesn't have the standard word boundary delimiters Splunk expects. In your case, there are tilde characters around the word you want to search on, so the "examplename" string doesn't match, but the "*examplename" does.
How can you fix this? I think you have two choices:
1) Don't use tildes as delimiters! Actually, don't use any non-standard delimiters. But we don't always have control over that, so...
2) If you know you have this issue with your data, create a fields.conf file in the same app that contains your props.conf, and set it like this (NOTE: replace "fieldname" below with the actual fieldname!):
[fieldname]
INDEXED_VALUE = false
But don't arbitrarily do this for all of your fields - as it can make your searches less efficient.
... View more