Getting Data In

Input script encounters fatal error when Splunk runs it automatically, but not manually?

drautb
Explorer

We're using the Splunk App for AWS, and have been doing some customizations to better suit our needs. I've run into a strange problem though. One of the scripts connects to an Amazon S3 bucket in order to download some billing information, but when splunk runs the script automatically as scheduled, it fails to index any data and gives this error:

get_bill.py: Traceback (most recent call last):
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/get_bill.py", line 65, in <module>
get_bill.py:     a = conn.create_bucket(s3bucket1)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/s3/connection.py", line 432, in create_bucket
get_bill.py:     data=data)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/s3/connection.py", line 468, in make_request
get_bill.py:     override_num_retries=override_num_retries)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/connection.py", line 910, in make_request
get_bill.py:     return self._mexe(http_request, sender, override_num_retries)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/connection.py", line 872, in _mexe
get_bill.py:     raise e
get_bill.py: socket.error: [Errno 111] Connection refused

But, if I run the script manually on the server, using the following command, it runs perfectly and all the data is printed out:

$SPLUNK_HOME/bin/splunk cmd python $SPLUNK_HOME/etc/apps/SplunkAppforAWS/bin/get_bill.py

Which doesn't make sense to me. Has anyone seen anything like this before?

I know we have been updating our Splunk instances lately, and there have been some network changes that could be affecting this, but I haven't had a chance to see if there is any correlation. I was baffled that when I run the command manually it works, but when splunk tries to do it, it fails.

Thanks!

0 Karma
1 Solution

drautb
Explorer

Well, something must have gotten messed up during our updates I think. Restarting the box resolved the issue.

View solution in original post

0 Karma

drautb
Explorer

Well, something must have gotten messed up during our updates I think. Restarting the box resolved the issue.

0 Karma

grijhwani
Motivator

This sort of behaviour (batch doesn't work; interactive does) is often a PATH issue. When you log in as Splunk there is no guarantee that your PATH is going to be the same as that available to batch jobs/services, particularly not if you have particular fiddles in your /etc/profile or the Splunk interactive account's .profile or any of the possible Bash rc files.

Set yourself a triggered task to run a shell and output its environment to a file (i.e. "set > set.bg") then do something similar in an interactive shell ("set > set.fg") then diff the two to determine any significant differences in PATH or other environment variables.

0 Karma

drautb
Explorer

Thanks for the suggestion, I had verified that the environments were the same.

0 Karma

drautb
Explorer

Splunk runs under the 'splunk' user. I became the 'splunk' user to run the script manually when it succeeded.

0 Karma

linu1988
Champion

Could you check which account the splunk runs under? May you can own the Splunk process, May be some access related issue rather than Network.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...