Field Extraction index=_internal

[geekf]

I tried to run the Indexing Performance: Instance dashboard but was not getting any data, on exploring the search I found out index=_internal is not doing the field extractions for this data in the log:

group=per_host_thruput, ingest_pipe=1, series="splunkserver.local", kbps=8.451, eps=32.903, kb=261.974, ev=1020, avg_age=2.716, max_age=3	

If I manually extract the fields using rex I can view it in the search but the dashboard still doesn't show the results. Is there a way to extract these fields for the internal index?

Thanks!

[dural_yyz]

I've tried to search internal in several different apps and it all extracted the fields.  The field extractions are clearly marked out in props.conf under the Splun app default directory.  I really can't see how that would have been subverted but a btool outputs from props.conf for stanza splunkd would be good.

[geekf]

Thank you for your response. I am uploading the btool output for splunkd.

[dural_yyz · Accepted solution]

I can't really see anything wrong but I dislike the following.

/opt/splunk/etc/system/local/props.conf     KV_MODE = json

Since I do see it in several of the various splunkd* stanzas it makes me think it was set in local under a default stanza.  I personally would look to remove that but keep in mind if this fixes the internal log extraction it will break something else that needs the json configuration.  I've always tried to create custom apps and place any default overrides in the custom app rather than allow anything to fall into the ./splunk/etc/system/local/*.conf.

[geekf]

We made this change, and it worked fine!

Thank you so much for your help.

[geekf]

We use json for Zeek, if we change that setting, will it impact Zeek logs?

[dural_yyz]

If you put that setting under the specific stanza for that sourcetype then changes to default stanza wont impact.  Anything under default stanza is only considered if the same setting has NOT been set in a more specific stanza.