Thank you for your response. Actually the purpose of configuring HF between AWS S3 and the Indexer cluster is that we don't want Indexer cluster to pull data in any case thereby not using any CPU/Memory for this purpose. So, HF is there just to pull data from S3 and forward it to the Indexer cluster. This indexer cluster is in some other AWS account and we have access to its endpoints. We want indexing and all to happen on this indexer cluster. For this reason we don't want to install any logtype (ie. Cylance, AMP etc.) specific add-ons/apps on HF. My other question is, since log specific Apps usually require direct ingestion from the source device to the App, we cannot use these apps. So, can we rely on default indexing that pulls Selected fields and Interesting fields as indexed by Indexer from the data that is ingested into the Indexer cluster?
... View more
I am looking for a solution for my current environment: - Data residing on AWS S3. This data is from various sources and we collect them to AWS S3 buckets - We are planning to install HF under the same AWS account where the data is available on S3. This data should be injected from S3 to Heavy Forwarder (HF) and then from HF, it should get ingested into Indexer cluster - Since we are getting the data from various different sources, do we need to install individual Splunk apps or add-ons for these data types on HF. Data may be Cylance, FireEye etc. data? Since couple of these apps require data ingestion directly from the source device, it seems we cannot use them for our purpose. My question is: Should we directly inject data from S3 to HF and then from HF to Indexer cluster? Here is a flow to show end to end picture: AWS S3 (Data from sources) ->> AWS SQS ->> HF (with Splunk App for AWS to pull data from AWS SQS ) ->> Indexer cluster Thanks.
... View more