Splunk App for AWS: Why the S3 buckets list does not show in the Splunk Add-on

[xiyangyang]

I can get cloudwatch log and description successfull but not S3 buckets.

My splunk EC2 has full access right to the s3 buckets but they are not listed in the Add-on setting page.

I tried to get the s3 buckets from Add data, but I got internal logs like:

"The last data ingestion itertion hasn't been completed yet"

I attcheched the internal logs, hope soemone can help.

---------------internal logs----------------------------------------------2021-01-18 18:57:59,217 level=ERROR pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:91 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="Failed to collect data through generic S3." start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066"
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 86, in index_data
    self._do_index_data()
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 107, in _do_index_data
    self.collect_data()
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 153, in collect_data
    self._discover_keys(index_store)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 224, in _discover_keys
    bucket = self._get_bucket(credentials)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 370, in _get_bucket
    bucket = conn.get_bucket(self._config[asc.bucket_name])
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/s3/connection.py", line 509, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/s3/connection.py", line 528, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/boto2_s3_patch.py", line 12, in wrapper
    response = func(*args, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/s3/connection.py", line 671, in make_request
    retry_handler=retry_handler
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/connection.py", line 1084, in make_request
    retry_handler=retry_handler)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/connection.py", line 1043, in _mexe
    raise ex
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/connection.py", line 956, in _mexe
    request.body, request.headers)
  File "/opt/splunk/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/splunk/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/splunk/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/splunk/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/opt/splunk/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/https_connection.py", line 119, in connect
    sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/opt/splunk/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/opt/splunk/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
socket.timeout: timed out

2021-01-18 18:57:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:57:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:56:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:56:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:55:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:55:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:54:53,444 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:54:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:53:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:53:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:52:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:52:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:51:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:51:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:50:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:50:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:49:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:49:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet."

2021-01-18 18:48:53,545 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_get_bucket:365 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="Create new S3 connection."

2021-01-18 18:48:53,545 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:load:163 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="load credentials succeed" arn="arn:aws:sts::052086164386:assumed-role/splunk_ec2_access/i-000c869a7adf815b0" expiration="2021-01-18 15:23:20+00:00"
2021-01-18 18:48:53,444 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:_load_source_credentials:195 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="fetch ec2 instance credentials"
2021-01-18 18:48:53,444 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:load:156 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="begin loading credentials" aws_account="splunk_ec2_access" aws_iam_role=None
2021-01-18 18:48:53,444 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_discover_keys:220 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="Start of discovering S3 keys."
2021-01-18 18:48:53,443 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:collect_data:143 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066" | message="Start processing" last_modified="2020-12-17T07:30:59.000Z" latest_scanned=""
2021-01-18 18:48:53,442 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:105 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066" | message="Start processing."