Getting Data In

Input script encounters fatal error when Splunk runs it automatically, but not manually?

drautb
Explorer

We're using the Splunk App for AWS, and have been doing some customizations to better suit our needs. I've run into a strange problem though. One of the scripts connects to an Amazon S3 bucket in order to download some billing information, but when splunk runs the script automatically as scheduled, it fails to index any data and gives this error:

get_bill.py: Traceback (most recent call last):
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/get_bill.py", line 65, in <module>
get_bill.py:     a = conn.create_bucket(s3bucket1)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/s3/connection.py", line 432, in create_bucket
get_bill.py:     data=data)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/s3/connection.py", line 468, in make_request
get_bill.py:     override_num_retries=override_num_retries)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/connection.py", line 910, in make_request
get_bill.py:     return self._mexe(http_request, sender, override_num_retries)
get_bill.py:   File "/opt/splunk/etc/apps/SplunkAppforAWS/bin/boto/connection.py", line 872, in _mexe
get_bill.py:     raise e
get_bill.py: socket.error: [Errno 111] Connection refused

But, if I run the script manually on the server, using the following command, it runs perfectly and all the data is printed out:

$SPLUNK_HOME/bin/splunk cmd python $SPLUNK_HOME/etc/apps/SplunkAppforAWS/bin/get_bill.py

Which doesn't make sense to me. Has anyone seen anything like this before?

I know we have been updating our Splunk instances lately, and there have been some network changes that could be affecting this, but I haven't had a chance to see if there is any correlation. I was baffled that when I run the command manually it works, but when splunk tries to do it, it fails.

Thanks!

0 Karma
1 Solution

drautb
Explorer

Well, something must have gotten messed up during our updates I think. Restarting the box resolved the issue.

View solution in original post

0 Karma

drautb
Explorer

Well, something must have gotten messed up during our updates I think. Restarting the box resolved the issue.

0 Karma

grijhwani
Motivator

This sort of behaviour (batch doesn't work; interactive does) is often a PATH issue. When you log in as Splunk there is no guarantee that your PATH is going to be the same as that available to batch jobs/services, particularly not if you have particular fiddles in your /etc/profile or the Splunk interactive account's .profile or any of the possible Bash rc files.

Set yourself a triggered task to run a shell and output its environment to a file (i.e. "set > set.bg") then do something similar in an interactive shell ("set > set.fg") then diff the two to determine any significant differences in PATH or other environment variables.

0 Karma

drautb
Explorer

Thanks for the suggestion, I had verified that the environments were the same.

0 Karma

drautb
Explorer

Splunk runs under the 'splunk' user. I became the 'splunk' user to run the script manually when it succeeded.

0 Karma

linu1988
Champion

Could you check which account the splunk runs under? May you can own the Splunk process, May be some access related issue rather than Network.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...