Splunk Search

S3 Bucket Logging

bgill0123
Loves-to-Learn

I have a bunch of web servers that are currently streaming their logs (real time) into an S3 bucket.

I have the Splunk AWS add-on installed to collect those logs but have found that it duplicates the data and given the amount of data coming in, this would kill our license very quickly. The owners of these servers will not allow us to install the UF on it.  Setting up a HF is also a no go.
Is there another way to ingest those logs without duplication.
I have seen there might be a way with a custom props.conf file but I am not sure this is way.

Any ideas would be welcome.

 

Thanks

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Vibe-coding, AI, and Splunkcraft: Highlights from the .conf26 Builder Bar

If you stopped by the Builder Bar at .conf26, thank you! This year, we brought ...

Thanks for the Memories: .conf26 Took Learning to New Heights

Thank you, Splunk Community, for making .conf26 in Denver one for the books. From packed Splunk University ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...