topic Re: Field Extraction index=_internal in Splunk Search

Field Extraction index=_internal

geekf — Wed, 09 Oct 2024 12:59:39 GMT

I tried to run the Indexing Performance: Instance dashboard but was not getting any data, on exploring the search I found out index=_internal is not doing the field extractions for this data in the log:

group=per_host_thruput, ingest_pipe=1, series="splunkserver.local", kbps=8.451, eps=32.903, kb=261.974, ev=1020, avg_age=2.716, max_age=3

If I manually extract the fields using rex I can view it in the search but the dashboard still doesn't show the results. Is there a way to extract these fields for the internal index?

Thanks!

Re: Field Extraction index=_internal

dural_yyz — Wed, 09 Oct 2024 13:37:37 GMT

I've tried to search internal in several different apps and it all extracted the fields. The field extractions are clearly marked out in props.conf under the Splun app default directory. I really can't see how that would have been subverted but a btool outputs from props.conf for stanza splunkd would be good.

Re: Field Extraction index=_internal

geekf — Wed, 09 Oct 2024 13:56:59 GMT

Thank you for your response. I am uploading the btool output for splunkd.

Re: Field Extraction index=_internal

dural_yyz — Wed, 09 Oct 2024 14:43:55 GMT

I can't really see anything wrong but I dislike the following.

/opt/splunk/etc/system/local/props.conf KV_MODE = json

Since I do see it in several of the various splunkd* stanzas it makes me think it was set in local under a default stanza. I personally would look to remove that but keep in mind if this fixes the internal log extraction it will break something else that needs the json configuration. I've always tried to create custom apps and place any default overrides in the custom app rather than allow anything to fall into the ./splunk/etc/system/local/*.conf.

Re: Field Extraction index=_internal

geekf — Wed, 23 Oct 2024 12:45:45 GMT

We use json for Zeek, if we change that setting, will it impact Zeek logs?

Re: Field Extraction index=_internal

dural_yyz — Wed, 23 Oct 2024 12:57:27 GMT

If you put that setting under the specific stanza for that sourcetype then changes to default stanza wont impact. Anything under default stanza is only considered if the same setting has NOT been set in a more specific stanza.

Re: Field Extraction index=_internal

geekf — Wed, 23 Oct 2024 13:52:08 GMT

We made this change, and it worked fine!

Thank you so much for your help.