How to resolve error when Splunkd intermittently crashes while streaming telemetry data on Universal Forwarder: "ProcessRunner: No such file or directory"?

[prajnaamey]

Hello!

I’m working on streaming telemetry data to Splunk. I use Splunk Universal Forwarder v7 x86_64 to capture and stream data to Splunk Enterprise 8.

I use the script:// to capture data and run them at certain specified intervals. The data is being successfully streamed to the server. But, intermittently, splunkd (SUF) crashes, and I see the following error in my splunkd.log.

06-02-2020 17:12:27.975 -0700 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
06-02-2020 17:12:27.993 -0700 INFO  WatchedFile - Will begin reading at offset=1182 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
06-02-2020 17:12:56.832 -0700 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
06-02-2020 17:30:37.696 -0700 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
06-02-2020 17:53:37.315 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  ERROR - Failed opening "": No such file or directory
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  terminate called after throwing an instance of 'EventLoopException'
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:    what():  Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
06-02-2020 17:53:37.676 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!

I have tried to grok through Splunk answers and on Google; but, I couldn’t find much documentation/articles on what file ProcessRunner was trying to open? Could someone help me or point me to the right channel to understand how I can fix this issue.

Here’s my inputs.conf ’s script stanzas:

[script://$SPLUNK_HOME/bin/scripts/<script-one>.py]
source = source-one
sourcetype = source-one
[script://$SPLUNK_HOME/bin/scripts/<script-two>.path]
source = source-two
sourcetype = source-two
interval = 60
[script://$SPLUNK_HOME/bin/scripts/<script-three>.path]
source = source-three
sourcetype = source-three
interval = 1800
[script://$SPLUNK_HOME/bin/scripts/<script-four>.path]
source = source-four
sourcetype = source-four
interval = 1800

Thank you!

[efika]

Is it possible that you didn't do a proper error handling in the py scripts themselves ?
Are you trying to read some data in the python scripts and they will abort not in a graceful way while not being able to read the data ?

[prajnaamey]

Thank you for your reply, efika!

My initital thought was that there was an unhandled exception in the script. I removed the .py script and was seeing the same error.

I'm in the process of testing each stanza by itself to see if the culprit is one of our scripts.

I observed this issue occurring in SUF 7.x and SUF 8.x. I have had the same scripts running for SUF 6.x (32-bit) and did not encounter any such error. The SUF 6.x machines have been running for months now using the same scripts.

If my understanding is right - ExecProcessor runs the script stanzas, and it doesn't cause a Splunk crash if an error occurs while executing the script. I wonder what ProcessRunner is and what it's trying to do?