We have a distributed environment with a search head cluster, 2 indexers (not clustered), deployment server and multiple HF.
We decided to install the Splunk TA eStreamer eNcore add-on on one of the Heavy Forwarders. Everything was configured like it said in the manual and we set it up so that eStreamer connects to the FMC and places the log data from the FMC locally on the heavy forwarder.
This data is then forwarded to one of the Indexers which does the indexing.
So far everything seems to be working, but for a few questions regarding the source type:
in inputs.conf in the eStreamer app on the Heavy Forwarder the source type is set to "cisco:estreamer:data".
This source type exists only on the HF where the add on is installed, but the indexer is still able to index the data with the source type set to "cisco:estreamer:data".
When i open the source type "cisco:estreamer:data" i see a lot of configuration like so:
So the question is: Do we have to install the add-on on the Indexer as well as the Heavy Forwarder?
I noticed that on the indexer i can't see fields like "app" and "bytes_in" which are supposed to be created as aliases by the source type. So maybe the answer is "yes, we need this add-on and source type on the Indexer as well", but at the same time it looks like this might be search-related fieldaliases and lookups, and maybe it should be installed on the Search Head Cluster instead?
Or maybe on both the Indexer and the Search Head Cluster?
I'm also a bit worried about time-stamps as i'm not sure Splunk is reading the "Timestam prefix" when indexing the data (because the source type is not available on the Indexer).
The issue with this app is just an example (a very relevant example) when trying to figure out where Source Types need to be configured and defined. Should they go on HF, Indexers og Search Heads? Or maybe they should be on all three instances in a Distributed Environment?
we have an environment with estreamer on a HF too. So its correct to install the TA on your HF, since this is the instances which will pull the events from your estreamer server. Sourcetype definition will be set in your inputs (and props/transforms) on your HF.
What we also did is, install the APP for Estreamer on the SH/SHC. This will add value for field extraction as well.
I am not sure about the TA on the Indexer. Most of the time the developer will tell you where the app has to go, in this case there is no information on that. We did add the TA on the indexer (but make sure to not enable any inputs/disable them). Since the HF is doing all the parsing before the data is send to the Indexer, it could be enough to have the sourcetype definition on the HF.
I hope that helps
Thank you. I think we now understand how this works.
Like you pointed out there is no information on where the app has to go, but you have som good points. We will install the app on the HF and the SHC.
Because, as you said, the HF parses the data first it should extract the timestamp and set the source type before sending it to the Indexer. And as far as i can see there is no Index-extraction in this app and therefore it should not be needed on the Indexer.
In general, in a distributed environment, you can install the add-on (TA-eStreamer) in SH/SHC, Indexers/clusters and in Heavyforwarder or universal forwarder, as per respective add-on/app documentation. Also refer to https://docs.splunk.com/Documentation/AddOns/released/Overview/Distributedinstall
There is no harm in installing in indexer tier, as it will not do any parsing (index time), if the sourcetype already goes through a heavy forwarder layer. The add-on is needed on the Search head/SHC for search time parsing and datamodels (if you see them).
In the case of SH, IN and Heavy forwarder tiers, its enough if you deploy this add-on on SH and HF in your case [ we have done the same in our dist setup]. The encore config resides only in HF, which connects to FMC using certs to pull the data.