I’m working on streaming telemetry data to Splunk. I use Splunk Universal Forwarder v7 x86_64 to capture and stream data to Splunk Enterprise 8.
I use the
script:// to capture data and run them at certain specified intervals. The data is being successfully streamed to the server. But, intermittently,
splunkd (SUF) crashes, and I see the following error in my
06-02-2020 17:12:27.975 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'. 06-02-2020 17:12:27.993 -0700 INFO WatchedFile - Will begin reading at offset=1182 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'. 06-02-2020 17:12:56.832 -0700 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views 06-02-2020 17:30:37.696 -0700 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 06-02-2020 17:53:37.315 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process: ERROR - Failed opening "": No such file or directory 06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process: terminate called after throwing an instance of 'EventLoopException' 06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process: what(): Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory 06-02-2020 17:53:37.676 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!
I have tried to grok through Splunk answers and on Google; but, I couldn’t find much documentation/articles on what file
ProcessRunner was trying to open? Could someone help me or point me to the right channel to understand how I can fix this issue.
inputs.conf ’s script stanzas:
[script://$SPLUNK_HOME/bin/scripts/<script-one>.py] source = source-one sourcetype = source-one [script://$SPLUNK_HOME/bin/scripts/<script-two>.path] source = source-two sourcetype = source-two interval = 60 [script://$SPLUNK_HOME/bin/scripts/<script-three>.path] source = source-three sourcetype = source-three interval = 1800 [script://$SPLUNK_HOME/bin/scripts/<script-four>.path] source = source-four sourcetype = source-four interval = 1800
Is it possible that you didn't do a proper error handling in the py scripts themselves ?
Are you trying to read some data in the python scripts and they will abort not in a graceful way while not being able to read the data ?
Thank you for your reply, efika!
My initital thought was that there was an unhandled exception in the script. I removed the
.py script and was seeing the same error.
I'm in the process of testing each stanza by itself to see if the culprit is one of our scripts.
I observed this issue occurring in SUF 7.x and SUF 8.x. I have had the same scripts running for SUF 6.x (32-bit) and did not encounter any such error. The SUF 6.x machines have been running for months now using the same scripts.
If my understanding is right - ExecProcessor runs the script stanzas, and it doesn't cause a Splunk crash if an error occurs while executing the script. I wonder what ProcessRunner is and what it's trying to do?