All Apps and Add-ons

Splunk Enterprise on REHL 7 in AWS, how to mount S3 storage data for splunk?

Log_wrangler
Builder

Hi All,
I know this question leans more to AWS support but I thought I would poll the community for any insight.
I have an EC2 (RHEL7 OS) rolling in AWS and I am in the midst of installing Splunk Enterprise.
I have some Splunk data saved in an S3 bucket, but I am apprehensive about mounting the S3 data to the EC2 instance. I am worried that if not done properly Splunk might overwrite the data in S3.

Does anyone have experience with this?

Are there any caveats regarding the way S3 needs to be connected to Splunk's OS?
I have seen two ways (so far) "IAM role to access S3 buckets" and "Fuse" to connect the data to the OS.
Has anyone successfully tried either way with Splunk?

Thank you

0 Karma
1 Solution

burakcinar
Path Finder

Im using s3 bucket with awscli tool, it's much better than other tools, you can also remove files after sync to s3 bucket automatically.

sample command usage
aws s3 sync ./folderpath s3://mySplunkBucket/

aws s3 commands ;
https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

View solution in original post

0 Karma

burakcinar
Path Finder

Im using s3 bucket with awscli tool, it's much better than other tools, you can also remove files after sync to s3 bucket automatically.

sample command usage
aws s3 sync ./folderpath s3://mySplunkBucket/

aws s3 commands ;
https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

0 Karma

Log_wrangler
Builder

Thank you for the reply.

I should mention that the data in the S3 bucket is previously indexed data (uploaded via snowball). So I wanted to seamlessly access that data with the aws/ec2/rhel7/splunk enterprise deployment as before. This splunk instance will act as an indexer.

Do you see any issues with this scenario?

I will dig into your links.

Thank you.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...