All Apps and Add-ons

Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

asbetsplunk
Explorer

Splunk Add-on for AWS: 3.0.0
Splunk App for AWS: 4.1.1

Error Splunk App for AWS S3 Configure Input:

Unexpected error occurs. In handler 'splunk_app_aws_aws_s3buckets': Unexpected error "" from python handler: "BotoClientError: When using SigV4, you must specify a 'host' parameter.". See splunkd.log for more details.

Error at command line: /opt/splunk/var/log/splunk/splunkd.log

04-25-2016 10:40:29.392 +0000 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 529, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_s3buckets_handler.py", line 49, in handleList\n    buckets = au.list_s3_buckets(proxy, aws_account)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 150, in list_s3_buckets\n    proxy_user=proxy.username, proxy_pass=proxy.password)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/__init__.py", line 141, in connect_s3\n    return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/s3/connection.py", line 196, in __init__\n    "When using SigV4, you must specify a 'host' parameter."\nHostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.\n
04-25-2016 10:40:29.393 +0000 ERROR AdminManagerExternal - Unexpected error "<class 'boto.s3.connection.HostRequiredError'>" from python handler: "BotoClientError: When using SigV4, you must specify a 'host' parameter.".  See splunkd.log for more details.

I am using KMS encrypted CloudTrail logs but I have already updated /opt/splunk/etc/splunk-launch.conf with:

S3_USE_SIGV4 = True

I'm not sure why I'm getting this error because all my other S3 buckets are not encrypted.

I would also like to mention that I sent a request to sales@splunk.com to puchase an Annual Term License so I can get support but so far no reply. 😞

alt text

0 Karma

pchen_splunk
Splunk Employee
Splunk Employee

S3_USE_SIGV4 is not supported in AWS app 4.1.1 or before. We have supported it in the coming v4.2, in Frankfurt only.
For your case, I am investigating it. Will update you the progress later.

0 Karma

pchen_splunk
Splunk Employee
Splunk Employee

This problem is fixed in the coming v4.2. Thanks for reporting.

0 Karma

asbetsplunk
Explorer

@phen. Thank you for the info. Is there a tentative date for when it will be released as well as in the Ireland region? Our infrastructure is running out of the EU regions for compliance reasons.

0 Karma

chwang_splunk
Splunk Employee
Splunk Employee

Could you pls check the region of the S3 bucket? Does it locate in Frankfurt?

0 Karma

asbetsplunk
Explorer

Thank you for the quick reply.

These buckets are in the Ireland region.

0 Karma

chwang_splunk
Splunk Employee
Splunk Employee

I added "S3_USE_SIGV4 = True" to splunk-launch.conf then met such an error.
How about removing "S3_USE_SIGV4 = True" from splunk-launch.conf then restart your Splunk? Can it solve your problem temporarily in this case? If so, we will look into the root cause

0 Karma

asbetsplunk
Explorer

Okay, I stopped the splunk service:

sudo /opt/splunk/bin/splunk stop

Modified the splunk-launch.conf

sudo vi /opt/splunk/etc/splunk-launch.conf

Commented out the line and restarted the splunk service.

#S3_USE_SIGV4 = True

After doing so I was able to successfully add the S3 bucket and aws:elb:accesslogs. However being able to add the ELB S3 bucket doesn't seem to populate any new fields.

The number of ELBs still show 0 and the ELB Traffic Analysis dashboard is empty. The ELB Instances dashboard shows metrics EXCEPT for number of ELBs and ELBs by region.

After removing the S3_USE_SIGV4 = True line, /opt/splunk/var/log/splunk/splunk_ta_aws_s3_main.log just shows this:

2016-04-28 05:46:48,377 INFO pid=3260 tid=Thread-12 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:50:39,254 INFO pid=4213 tid=Thread-19 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:51:48,377 INFO pid=3260 tid=Thread-10 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:56:48,379 INFO pid=3260 tid=Thread-7 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:58:59,251 INFO pid=4213 tid=Thread-17 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 06:01:48,377 INFO pid=3260 tid=Thread-11 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet

I waited about 4 hours for new data to populate but nothing. I tried putting the S3_USE_SIGV4 = True line back in, restarting, and checking to see if S3 ELB data would populate but no change.

My organisation will align to the CIS (Center for Internet Security) AWS Benchmark in which KMS encrypted CloudTrail logs is an audit point so disabling S3_USE_SIGV4 = True is not an option for us. However, it doesn't seem that disabling it and adding the S3 bucket for ELB is adding any new data. Appears to be two different issues now. 😞

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...