Getting Data In

How can I monitor S3 buckets with Splunk Light?

alanpotosnak
Engager

I'm testing out Splunk Light. I know that currently there is no app or add-on that let's one easily monitor an S3 bucket. I've tried to use S3FS as a solution, but it only partially works. After adding my mounted directory to Splunk as an input, it doesn't index new files. In order to get new files indexed I have to disable and then re-enable the input.

Configuration issue?

Thanks

mathew_jones
Engager

Hey Alan,
Did you get a fix for this in the end?

I'm having the exact same behaviour and I'm stumped.

marquiselee
Path Finder

Mathew were you able to find a solution to this?

0 Karma

dkoshe_splunk
Splunk Employee
Splunk Employee

First check if when monitoring local/native FS directory, Splunk Light is able to index the newly created files.
If it's not able to it usually may mean that newly generated files are too similar and you are getting a crccheck issue (where the crc of the files is similar and splunk doesnt index because it thinks its the same file. Basically the first 256 bytes of the files are the same, in this case look for crcSalt in inputs.conf)

Of course it's possible that there's some config issue with S3FS. I haven't really used S3FS to give much input on config changes on S3FS side. But quick searching on the web following maybe pertinent -- Are you using local cache of S3FS for example. If you are using local cache, you may need to look at periodically purging "~/.s3fs"

0 Karma

alanpotosnak
Engager

Thanks for responding. Yes - when monitoring a local directory Splunk Light indexes the files. I've done an experiment where I copy newly created files from the mounted directory to a local directory that Splunk is monitoring and the files get indexed.

I'm not using a local cache of S3FS. What's strange is that I see that the number of files associated with the S3FS input increases as new files get created on S3 but the actual content doesn't get indexed in Splunk.

0 Karma

marquiselee
Path Finder

Hi, were you able to find a solution to this?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...