All Apps and Add-ons
Highlighted

Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Explorer

Splunk Add-on for AWS: 3.0.0
Splunk App for AWS: 4.1.1

Error Splunk App for AWS S3 Configure Input:

Unexpected error occurs. In handler 'splunk_app_aws_aws_s3buckets': Unexpected error "" from python handler: "BotoClientError: When using SigV4, you must specify a 'host' parameter.". See splunkd.log for more details.

Error at command line: /opt/splunk/var/log/splunk/splunkd.log

04-25-2016 10:40:29.392 +0000 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 529, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_s3buckets_handler.py", line 49, in handleList\n    buckets = au.list_s3_buckets(proxy, aws_account)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 150, in list_s3_buckets\n    proxy_user=proxy.username, proxy_pass=proxy.password)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/__init__.py", line 141, in connect_s3\n    return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/s3/connection.py", line 196, in __init__\n    "When using SigV4, you must specify a 'host' parameter."\nHostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.\n
04-25-2016 10:40:29.393 +0000 ERROR AdminManagerExternal - Unexpected error "<class 'boto.s3.connection.HostRequiredError'>" from python handler: "BotoClientError: When using SigV4, you must specify a 'host' parameter.".  See splunkd.log for more details.

I am using KMS encrypted CloudTrail logs but I have already updated /opt/splunk/etc/splunk-launch.conf with:

S3_USE_SIGV4 = True

I'm not sure why I'm getting this error because all my other S3 buckets are not encrypted.

I would also like to mention that I sent a request to sales@splunk.com to puchase an Annual Term License so I can get support but so far no reply. 😞

alt text

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Splunk Employee
Splunk Employee

Could you pls check the region of the S3 bucket? Does it locate in Frankfurt?

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Explorer

Thank you for the quick reply.

These buckets are in the Ireland region.

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Splunk Employee
Splunk Employee

I added "S3USESIGV4 = True" to splunk-launch.conf then met such an error.
How about removing "S3USESIGV4 = True" from splunk-launch.conf then restart your Splunk? Can it solve your problem temporarily in this case? If so, we will look into the root cause

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Explorer

Okay, I stopped the splunk service:

sudo /opt/splunk/bin/splunk stop

Modified the splunk-launch.conf

sudo vi /opt/splunk/etc/splunk-launch.conf

Commented out the line and restarted the splunk service.

#S3_USE_SIGV4 = True

After doing so I was able to successfully add the S3 bucket and aws:elb:accesslogs. However being able to add the ELB S3 bucket doesn't seem to populate any new fields.

The number of ELBs still show 0 and the ELB Traffic Analysis dashboard is empty. The ELB Instances dashboard shows metrics EXCEPT for number of ELBs and ELBs by region.

After removing the S3USESIGV4 = True line, /opt/splunk/var/log/splunk/splunktaawss3main.log just shows this:

2016-04-28 05:46:48,377 INFO pid=3260 tid=Thread-12 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:50:39,254 INFO pid=4213 tid=Thread-19 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:51:48,377 INFO pid=3260 tid=Thread-10 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:56:48,379 INFO pid=3260 tid=Thread-7 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 05:58:59,251 INFO pid=4213 tid=Thread-17 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet
2016-04-28 06:01:48,377 INFO pid=3260 tid=Thread-11 file=aws_s3_data_loader.py:_do_index_data:72 | Previous run is not done yet

I waited about 4 hours for new data to populate but nothing. I tried putting the S3USESIGV4 = True line back in, restarting, and checking to see if S3 ELB data would populate but no change.

My organisation will align to the CIS (Center for Internet Security) AWS Benchmark in which KMS encrypted CloudTrail logs is an audit point so disabling S3USESIGV4 = True is not an option for us. However, it doesn't seem that disabling it and adding the S3 bucket for ELB is adding any new data. Appears to be two different issues now. 😞

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Splunk Employee
Splunk Employee

S3USESIGV4 is not supported in AWS app 4.1.1 or before. We have supported it in the coming v4.2, in Frankfurt only.
For your case, I am investigating it. Will update you the progress later.

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Splunk Employee
Splunk Employee

This problem is fixed in the coming v4.2. Thanks for reporting.

0 Karma
Highlighted

Re: Splunk App for AWS: When trying to configure S3 input for ELB, getting "BotoClientError: When using SigV4, you must specify a 'host' parameter."

Explorer

@phen. Thank you for the info. Is there a tentative date for when it will be released as well as in the Ireland region? Our infrastructure is running out of the EU regions for compliance reasons.

0 Karma