topic Re: Process of Indexed Extraction Configuration in Getting Data In

Process of Indexed Extraction Configuration

mrstrozy — Tue, 05 Mar 2019 16:52:28 GMT

Hi All!

I'm currently running into a very weird situation with a Splunk instance I inherited. I setup the props.conf through the UI on my dev instance by indexing a small number of events and then using the UI to parse through the data, creating the props.conf. I should mention that my dev instance is a single host.

I then transferred the props.conf to our test environment which consists of 1 forwarder, 2 indexers (in a "fake" cluster since less than 3 indexers), 1 master, and 3 search heads in a search head cluster. Just like my dev instance, the test instance worked properly as the fields were showing up successfully when searching on a search head.

Finally, I transferred this same props.conf to the prod instance which consists of 3 forwarders, 4 indexers in an index cluster, 1 master and 5 search heads in a search head cluster. In this environment, none of the fields get properly extracted like they were in the test/dev instances but the events are still being parsed correctly as JSON. The current fields back I'm getting are these:

I've exhausted everything I know about how the configuration/field extraction is determined and I still can't figure it out. I'm sure there's something I'm missing, and given that it's an instance that I've inherited I figured I'd post something here to see what this wonderful community could come up with. Here is a snippet from my props.conf which is pretty much how most of the sourcetypes are configured:

This props.conf lives only on the indexers (as far as I know) and I didn't find any other props.conf files on the search heads (in $SPLUNK_HOME/etc/system/local).

Any help is greatly appreciated.

Thanks!

Re: Process of Indexed Extraction Configuration

tsaikumar009 — Tue, 29 Sep 2020 23:34:09 GMT

we need to put indexed_extractions = json on Indexer props and kv_mode = none on Search head props.

try to put kv_mode=json on search head props and remove all other json related details from props

Re: Process of Indexed Extraction Configuration

mrstrozy — Tue, 05 Mar 2019 17:37:27 GMT

Why would it work correctly on my test environment (no props.conf on the search heads) then?

Re: Process of Indexed Extraction Configuration

mrstrozy — Tue, 05 Mar 2019 18:01:15 GMT

As for the second solution, we want indexed extractions and not search head extractions which I believe that would cause.

Re: Process of Indexed Extraction Configuration

woodcock — Wed, 06 Mar 2019 07:48:47 GMT

You have not deployed the props.conf configurations to the correct place. Unlike every other indexing-related configuration which should be deployed on the first full-instance of Splunk that receives it (either the HF or Indexer tier), the INDEXED_EXTRACTIONS configuration must be deployed to the forwarder, the server which possesses the files and and has the inputs.conf that is set to pull in the json. So send it to your forwarder tier, and restart all splunk instances there and it will work when you forward new data in.

Re: Process of Indexed Extraction Configuration

mrstrozy — Wed, 06 Mar 2019 15:35:50 GMT

I should've mentioned that we are using the HEC on the forwarders to transfer the data to the indexers. Does that change anything about what you suggested? I'm also still confused about how the configuration works in the test environment without the props.conf on the forwarder.

Re: Process of Indexed Extraction Configuration

mrstrozy — Wed, 06 Mar 2019 16:59:14 GMT

Also I've tried this and it did not fix my issue.

Re: Process of Indexed Extraction Configuration

woodcock — Thu, 07 Mar 2019 03:30:58 GMT

If you are using HEC, then you are not using INDEXED_EXTRACTIONS. Which is it?

Re: Process of Indexed Extraction Configuration

mrstrozy — Thu, 07 Mar 2019 03:38:44 GMT

Correct me if I'm wrong but I'm not sure why both have to be separate? I was under the impression it went like this:

host -- (through HEC) --> Forwarders ----> Indexers

How would this effect the extractions at all?

Re: Process of Indexed Extraction Configuration

woodcock — Thu, 07 Mar 2019 04:11:26 GMT

That is an insane configuration. Typically HEC runs directly on the Indexers. If this is really your architecture, you need a reboot.

Re: Process of Indexed Extraction Configuration

mrstrozy — Thu, 07 Mar 2019 04:25:43 GMT

I agree the architecture needs to be rethought but according to this article, the HEC can run on either the forwarders or the indexers so I'm not really sure what you're getting at - https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/ScaleHTTPEventCollector

Re: Process of Indexed Extraction Configuration

woodcock — Thu, 07 Mar 2019 04:31:40 GMT

You have created an completely unnecessary bottleneck with your Intermediate Forwarder tier. What is worse, apparently you are writing HEC to disk there, so that you can do INDEXED_EXTRACTIONS which is nuts, because it defeats the primary benefit of HEC: diskless I/O.

Re: Process of Indexed Extraction Configuration

mrstrozy — Thu, 07 Mar 2019 04:38:07 GMT

I see what you're saying. Yeah after inheriting this instance my goal was to rethink/architect everything but I was swamped with other things and have limited knowledge so I'm all for learning the best practice. You would suggest having the HEC on the indexer cluster, keeping the props.conf there and going from there?

Re: Process of Indexed Extraction Configuration

woodcock — Thu, 07 Mar 2019 04:52:43 GMT

Yes, definitely. This is 100% upside (no downside).

Re: Process of Indexed Extraction Configuration

mrstrozy — Thu, 07 Mar 2019 05:05:04 GMT

Awesome, thanks for your help woodcook it's greatly appreciated

Re: Process of Indexed Extraction Configuration

kundeng — Sat, 25 Apr 2020 02:31:37 GMT

Did you fix this issue? Does HEC support explicit indexed_extractions of CSV or JSON files when it is set to raw mode?