I have some sources that are coming in as json, and I am experiencing odd behavior where I cannot search on a particular field, but I can only find the value when doing a search against the _raw data.
So for example, I have a field let's say "cluster", and I see it is also extracted just fine in the "Interesting fields" on the lefthand side. One of the values we'll say is "cluster-name-A".
If I search in the query bar for:
cluster="cluster-name-A" sourcetype=mysourcetype index=myindex
I get no results, however if I just do a blanket search:
cluster-name-A sourcetype=mysourcetype index=myindex
My expected results come back fine.
What can I investigate here to see why it will not let me use the fieldname in our searches?
How are you indexing the JSON data? Are you indexing the JSON fields individually at index time?
What do your props and transforms look like?
Check that your props.conf has INDEXED_EXTRACTIONS = json
Note: If you set INDEXED_EXTRACTIONS=JSON, check that you have not also set KV_MODE = json for the same source type, which would extract the JSON fields twice, at index time and again at search time.
I have had similar issues when I created a custom indexed field at index time and did not put it in the fields.conf. If you are indexing the fields at index time, you need to tell Splunk they need to be treated as indexed fields.