Splunk Enterprise

Splunk S3 Generic Input not fetching objects in bucket

Meaf
Loves-to-Learn

Hi,

We are running a Splunk Enterprise HWF with a generic s3 input to fetch object from a s3 bucket, however each time we try to move this input onto a new identical HWF we have issues getting the same data from the same bucket.

Both instances are on Splunk 9.2 however the Splunk AWS TA versions are different. Both are pipeline managed so have all the same config / certs. The only difference we can see if that in the aws ta input log the 'broken' input never creates the S3 Connection before fetching the s3 objects and seems to think the bucket is empty.

Working input

2025-01-15 10:25:09,124 level=INFO pid=5806 tid=Thread-6747 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:load:162 | bucket_name="bucketname" datainput="input", start_time=1736918987 job_uid="8888", phase="fetch_key" | message="load credentials succeed" arn="AWSARN" expiration="2025-01-15 11:25:09+00:00"
2025-01-15 10:25:09,125 level=INFO pid=5806 tid=Thread-6747 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_get_bucket:364 | bucket_name="bucketname" datainput="input", start_time=1736918987 job_uid="8888", phase="fetch_key" | message="Create new S3 connection."
2025-01-15 10:25:09,130 level=INFO pid=5806 tid=Thread-6841 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=s3_key_processer.py:_do_index:148 | bucket_name="bucketname" datainput="input" last_modified="2025-01-15T04:00:41.000Z" phase="fetch_key" job_uid="8888" start_time=1736918987 key_name="bucketobject" | message="Indexed S3 files." size=819200 action="index"

Broken input

2025-01-15 12:00:33,369 level=INFO pid=3157753 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:load:217 | datainput="input" bucket_name="bucketname", start_time=1736942432 job_uid="8888", phase="fetch_key" | message="load credentials succeed" arn="AWSARN" expiration="2025-01-15 13:00:33+00:00"
2025-01-15 12:00:33,373 level=INFO pid=3157753 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_fetch_keys:378 | datainput="input" bucket_name="bucketname", start_time=1736942432 job_uid="88888", phase="fetch_key" | message="End of fetching S3 objects." pending_key_total=0

Unsure, where to go from here as we have tried this on multiple new machines. 

Thanks

Meaf

Labels (2)
Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
Have you already checked that both HFs have same IAM roles, permissions etc. if those have created manually with gui or cli it’s quite possible that some configurations are missing.
0 Karma

Meaf
Loves-to-Learn

Hi Isoutamo,

Both the HWF's using the same IAM role and go to the same S3 Bucket. We've initially create both the gen s3 inputs via config files when setting up the HWF via ansible, but have also recreated the inputs manually via the GUI to see if that made any difference.

When looking at it from AWS we have noticed that the HWF's go to different API endpoint when using ListObjects, as the HWF using the old TA (which is the working HWF) uses ListObjects but the HWF with the updated TA (which isnt working) uses ListObjectsV2.

Thanks

Meaf

0 Karma

isoutamo
SplunkTrust
SplunkTrust
And you have updated the IAM role to include that new endpoint?
0 Karma

Meaf
Loves-to-Learn

That endpoint seems to be specific by the TA itself which we cannot edit, we have both servers using the IAM to point to the s3 bucket which doesn't have s3 private endpoints enabled.

0 Karma
Get Updates on the Splunk Community!

Buttercup Games: Further Dashboarding Techniques (Part 2)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Index This | What is the next number in the series? 7,645 5,764 4,576…

February 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Buttercup Games: Further Dashboarding Techniques

Hello! We are excited to kick off a new series of blogs from SplunkTrust member ITWhisperer, who demonstrates ...