Thank you for your suggestion. But it looks like using Firehose would anyway require us to have a service to pull the logs from S3 and push it to Indexer cluster.   Could you please confirm if we go with Heavy Forwarder (HF) wherein we push the logs to HF using "Splunk App for AWS", then do only parsing on HF and ingest the parsed data into Indexer cluster and use log search on the Indexer cluster after that? Later, we can install log-type specific Splunk Apps on HF to get better field extraction and again ingest the data into Indexer cluster. However, in this case, can we use Indexer cluster for Log search? In other words, we index everything on the Indexer cluster and keep HF as a pass-through layer between S3 and the indexer cluster? Purpose of this architecture would be to keep Indexer cluster as light as possible while HF takes the heavy toll of Splunk Apps. ... View more

We were planning to use Splunk AWS app to read data from S3. Now it seems we can't install this app on Universal Forwarder and we must use Heavy Forwarder (HF). But as HF performs parsing and indexing, using HF may cause unnecessary processing before HF forwards the data to the Indexer cluster. We just want to use HF as a forwarder and all the indexing, parsing etc. we want to do at the Indexer cluster level. ... View more

Thanks. One more question: Since we can't use Heavy Forwarders as discussed above for our use-case, it seems we can't either use the Universal forwarders, because we have to pick the data/logs from AWS S3 buckets and ingest into our Indexer cluster. Could you please suggest any other way, that can satisfy our requirements.   In nutshell, we want to fetch data from S3, send it to intermediate forwarder (Heavy/Universal) and then this forward sends same data to the indexer cluster. In other words, is there a way disable indexing at Heavy forwarder level? If answer is no, then what option do we have to handle the above use case?   Thanks again. ... View more

Thanks. Looks like we have to go for UFs as per your suggestion. Could you please elaborate more on your following comment?: "Since you already have your data in S3, the only parts of the apps you need are the ones that don't input data (no inputs.conf, nothing in /bin)." Could you please point us to the document that can provide more details on the above? Being new Splunk users, we don't have complete understanding on the above. If I understand it correctly, you are saying that we don't need to configure these log specific apps to pull data directly from the devices which is the condition for most of the apps.   Thanks again. ... View more