How can I monitor S3 buckets with Splunk Light?

[alanpotosnak]

I'm testing out Splunk Light. I know that currently there is no app or add-on that let's one easily monitor an S3 bucket. I've tried to use S3FS as a solution, but it only partially works. After adding my mounted directory to Splunk as an input, it doesn't index new files. In order to get new files indexed I have to disable and then re-enable the input.

Configuration issue?

Thanks

[mathew_jones]

Hey Alan,
Did you get a fix for this in the end?

I'm having the exact same behaviour and I'm stumped.

[marquiselee]

Mathew were you able to find a solution to this?

[dkoshe_splunk]

First check if when monitoring local/native FS directory, Splunk Light is able to index the newly created files.
If it's not able to it usually may mean that newly generated files are too similar and you are getting a crccheck issue (where the crc of the files is similar and splunk doesnt index because it thinks its the same file. Basically the first 256 bytes of the files are the same, in this case look for crcSalt in inputs.conf)

Of course it's possible that there's some config issue with S3FS. I haven't really used S3FS to give much input on config changes on S3FS side. But quick searching on the web following maybe pertinent -- Are you using local cache of S3FS for example. If you are using local cache, you may need to look at periodically purging "~/.s3fs"

[alanpotosnak]

Thanks for responding. Yes - when monitoring a local directory Splunk Light indexes the files. I've done an experiment where I copy newly created files from the mounted directory to a local directory that Splunk is monitoring and the files get indexed.

I'm not using a local cache of S3FS. What's strange is that I see that the number of files associated with the S3FS input increases as new files get created on S3 but the actual content doesn't get indexed in Splunk.

[marquiselee]

Hi, were you able to find a solution to this?