Getting Data In
Highlighted

Process of Indexed Extraction Configuration

Path Finder

Hi All!

I'm currently running into a very weird situation with a Splunk instance I inherited. I setup the props.conf through the UI on my dev instance by indexing a small number of events and then using the UI to parse through the data, creating the props.conf. I should mention that my dev instance is a single host.

I then transferred the props.conf to our test environment which consists of 1 forwarder, 2 indexers (in a "fake" cluster since less than 3 indexers), 1 master, and 3 search heads in a search head cluster. Just like my dev instance, the test instance worked properly as the fields were showing up successfully when searching on a search head.

Finally, I transferred this same props.conf to the prod instance which consists of 3 forwarders, 4 indexers in an index cluster, 1 master and 5 search heads in a search head cluster. In this environment, none of the fields get properly extracted like they were in the test/dev instances but the events are still being parsed correctly as JSON. The current fields back I'm getting are these:
alt text

I've exhausted everything I know about how the configuration/field extraction is determined and I still can't figure it out. I'm sure there's something I'm missing, and given that it's an instance that I've inherited I figured I'd post something here to see what this wonderful community could come up with. Here is a snippet from my props.conf which is pretty much how most of the sourcetypes are configured:
alt text

This props.conf lives only on the indexers (as far as I know) and I didn't find any other props.conf files on the search heads (in $SPLUNK_HOME/etc/system/local).

Any help is greatly appreciated.

Thanks!

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Explorer

we need to put indexedextractions = json on Indexer props and kvmode = none on Search head props.

OR

try to put kv_mode=json on search head props and remove all other json related details from props

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Path Finder

Why would it work correctly on my test environment (no props.conf on the search heads) then?

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Path Finder

As for the second solution, we want indexed extractions and not search head extractions which I believe that would cause.

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Esteemed Legend

You have not deployed the props.conf configurations to the correct place. Unlike every other indexing-related configuration which should be deployed on the first full-instance of Splunk that receives it (either the HF or Indexer tier), the INDEXED_EXTRACTIONS configuration must be deployed to the forwarder, the server which possesses the files and and has the inputs.conf that is set to pull in the json. So send it to your forwarder tier, and restart all splunk instances there and it will work when you forward new data in.

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Path Finder

I should've mentioned that we are using the HEC on the forwarders to transfer the data to the indexers. Does that change anything about what you suggested? I'm also still confused about how the configuration works in the test environment without the props.conf on the forwarder.

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Esteemed Legend

If you are using HEC, then you are not using INDEXED_EXTRACTIONS. Which is it?

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Path Finder

Correct me if I'm wrong but I'm not sure why both have to be separate? I was under the impression it went like this:

host -- (through HEC) --> Forwarders ----> Indexers

How would this effect the extractions at all?

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Esteemed Legend

That is an insane configuration. Typically HEC runs directly on the Indexers. If this is really your architecture, you need a reboot.

0 Karma
Highlighted

Re: Process of Indexed Extraction Configuration

Path Finder

I agree the architecture needs to be rethought but according to this article, the HEC can run on either the forwarders or the indexers so I'm not really sure what you're getting at - https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/ScaleHTTPEventCollector

0 Karma