Splunk Search

Data from extracted fields not showing unless wildcards or where= is used

FelixLeh
Contributor

We had a problem that certain fields weren't searchable. 
index=foo bar=* did not show any result even though interesting fields and the events themselves showed that the fields were extracted as expected.
Searching specific values also didn't help.
Some fields did return results but only if you added wildcards:

 

index=foo bar="*value"

 

It was possible to find the events when using:

 

| where isnontull(foo)

 

 OR 

 

| eval check = if(foo="bar",1,0)
| where check = 1

 

but any other method did not return results.

After some research in the community I found Posts with similar problems.
Some suggested that there could be hidden fields present.
Most of the time the suggestion was to enable indexed extractions with the fields.conf and adding a wildcard there.

 

[MyField]
INDEXED_VALUE = *<VALUE>

 

While trying out whether this option would work for me, I compared existing fields.conf files in the environment and noticed that all of the affected missing field values are part of the fields.conf of another app.

By default fields.conf is global if im not mistaken hence the fields defined in fields.conf as indexed extraction affected other sourcetypes but since the fields in those other apps did not include the necessary parameters to successfully extract at index time this error occurred. 
Setting export to none in the metadata of the app that holds the fields.conf file solved my problem.

local/default.meta:

 

[fields]
export = none

 

As seen in the fields.conf Docs:

* Do not create indexed fields with names that collide with names of fields
  that are extracted at search time.

I was aware of this setting regarding creating new fields myself but forgot to check whether any of the installed apps already do so. 

Hope this helps anyone that may be in a similar situation!

TLDR:

If you have trouble with extracted fields and cant search them even though they are shown inside the events and interesting fields, check your environment for fields.conf and if the affected fields are defined as indexed extraction. If so limit the fields.conf with metadata to the original app.

Labels (2)
0 Karma
1 Solution

FelixLeh
Contributor

Found the solution myself and wanted to post anyway because I haven't found solutions in the community that summarised the issue.
If I missed one please feel free to link it here so people can find it easier.

View solution in original post

FelixLeh
Contributor

Found the solution myself and wanted to post anyway because I haven't found solutions in the community that summarised the issue.
If I missed one please feel free to link it here so people can find it easier.

Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...