Splunk Add-On for S3 data inputs question

[pkeller]

In our test environment, we successfully setup the Splunk Add-on for Amazon S3 and pulled buckets so that we could view the data and make sure the props.conf settings were sorted out before we moved to production.

When we setup the same configuration in Production, we're only pulling 'new' buckets. We'd like to ingest all the same buckets that were pulled into our test environment. Is there some setting in the Add-on (or on the S3 side) that keeps track of what has already been pulled, thus preventing a duplicate pull?

Thanks very much,

[pkeller]

Ultimately I just wiped and reinstalled the app and reconfigured the inputs. The form has a place to enter the date that you're choosing to go back to, but after the first collection, the app seems to look somewhere else last_modified=2016-04-07T2 instead of the date that you enter via the UI ... gets it from

index_store.last_modified in s3_mod/aws_s3_data_loader.py

Anyway. Thank you. I'm all caught up now.

[ryandg · Accepted solution]

When you say you moved it from test to production, do they share the same devices that pull from S3 and just different indexers? The only thing I can think of is the pointers were already created so when you switched the outputs to a different indexer cluster you kept the previous pointers. Easiest way is to just clone/recreate the S3 inputs (assuming there aren't a ton of them) and it will reload them.