topic Re: Splunk Add-on for AWS Hangs in All Apps and Add-ons

Splunk Add-on for AWS Hangs

grahmorl — Thu, 01 Mar 2018 11:53:01 GMT

Hi,

I'm having issues getting the Splunk Add-on for AWS to work on a completely clean installation of Splunk.

I'm using the latest version of the Add-on from Splunkbase (v4.4.0) and I've tried on both Splunk 6.6.6 and 7.0.2

The problem I'm having is that if you go to either the 'Inputs' or 'Configuration' pages, the web interface just hangs with a spinning 'Loading' icon.

But unfortunately there's no accepted answer for this.

Looking in splunkd.log

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" sourcetype=splunkd ERROR aws

It appears to be a whole host of REST errors.

Has anyone else come across this issue? And perhaps has a solution?

Thanks.

Re: Splunk Add-on for AWS Hangs

deepashri_123 — Mon, 05 Mar 2018 12:11:41 GMT

Hey grahmorl,

What is the hardware specification you are using? Are you using EBS volumes or are you using local storage?

Re: Splunk Add-on for AWS Hangs

klaxdal — Sat, 31 Mar 2018 14:26:21 GMT

I encountered a similar issue - removed everything in the /local - restarted splunk and it stopped hanging .

From there reconfigured my inputs and all was well -

Re: Splunk Add-on for AWS Hangs

grahmorl — Thu, 26 Apr 2018 12:47:11 GMT

After a few months of back and forth with Splunk Support, we got to the root cause.

The credit for this goes to the unnamed (sorry, I don't know their name) AWS Add-on developer, who the support case got escalated to. They flagged that there was a global boto configuration file (/etc/boto.cfg) which was causing a conflict with some assumed defaults which the AWS Add-on relies on.

To summarise:

Problem:
- This problem presented as a hanging / spinning ‘Loading’ icon in the Splunk AWS Add-on Inputs and Configuration views.
- A search in the _internal index for “aws ERROR” showed python stack trace error messages.
- The problem appears to be configuration in the /etc/boto.cfg file. This is a global configuration file, so the settings in here are conflicting with the defaults (if there was no config file), which the Splunk AWS Add-on relies on.
- The file is created due to the default installation of Google’s gcloud / gsutil command line tools. The tools are installed by default on Google Cloud Compute (GCP) hosted servers.
- Our initial AWS integration testing was using GCP account to host the HFs, whilst waiting for access to a different AWS account.

Solution:
- You don't really want remove the /etc/boto.cfg configuration file because those servers might need these tools to access GCP services / APIs.
- In addition, even if we removed it, there is the possibility that other tools could create it in the future.
- Therefore the fix is to do something which will only affect Splunk.
- What we did was, in the $SPLUNK_HOME/etc/splunk-launch.conf file we added the environment variable:

BOTO_CONFIG=/tmp/does_not_exist

- This sets the BOTO_CONFIG variable for just the Splunk process specifically to a file which doesn’t exist.
- Because the files doesn’t exist, the defaults, as opposed to the global /etc/boto.cfg are used.
- Therefore there is now no conflict for the Splunk AWS Add-on, which runs under the Splunk process.

I hope this helps others who may be having similar issues. Or at least points them in a direction to look into.

Graham.

Re: Splunk Add-on for AWS Hangs

richgalloway — Thu, 26 Apr 2018 14:55:51 GMT

@grahmorl If your problem is resolved, please accept an answer to help future readers.

Re: Splunk Add-on for AWS Hangs

grahmorl — Thu, 26 Apr 2018 14:57:55 GMT

@richgalloway - Of course. I was just waiting to my answer to be approved by the moderator. 😉

Re: Splunk Add-on for AWS Hangs

grahmorl — Thu, 26 Apr 2018 14:59:50 GMT

@deepashri_123 - Thanks for your comment. The issue was somewhat more involved and not hardware related (i.e. not a performance issue). But thank your message.

Re: Splunk Add-on for AWS Hangs

grahmorl — Thu, 26 Apr 2018 15:01:16 GMT

@klaxdal - My issue was actually happening on a completely new install of Splunk and the AWS components. So nothing configured or in /local

So I think my problem was a little different to yours.

But many thanks for your comment.

Re: Splunk Add-on for AWS Hangs

girishkalamati — Wed, 25 Nov 2020 23:40:04 GMT

what does ' /local ' means , I can go to this location '$SPLUNK_HOME/etc/' though. Thanks if you can explain how I can reach to /local and empty files if anything persist so that I can fix this loading issue.

Is the also a possible reason : SSL certificatse ?

Re: Splunk Add-on for AWS Hangs

girishkalamati — Thu, 26 Nov 2020 00:00:16 GMT

This did not fix my issue

Re: Splunk Add-on for AWS Hangs

splunkcol — Fri, 30 Apr 2021 14:01:58 GMT

this has fixed the problem, thanks

Re: Splunk Add-on for AWS Hangs

splunkcol — Fri, 16 Jul 2021 14:43:53 GMT

You saved me, at first I applied the process mentioned in this thread, but today for some reason the problem returned

I ran the command in linux df -h and a folder was full.

After deleting junk information and restarting Splunk I have solved it

Thank you