Monitoring Splunk

How to resolve error when Splunkd intermittently crashes while streaming telemetry data on Universal Forwarder: "ProcessRunner: No such file or directory"?



I’m working on streaming telemetry data to Splunk. I use Splunk Universal Forwarder v7 x86_64 to capture and stream data to Splunk Enterprise 8.

I use the script:// to capture data and run them at certain specified intervals. The data is being successfully streamed to the server. But, intermittently, splunkd (SUF) crashes, and I see the following error in my splunkd.log.

06-02-2020 17:12:27.975 -0700 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
06-02-2020 17:12:27.993 -0700 INFO  WatchedFile - Will begin reading at offset=1182 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
06-02-2020 17:12:56.832 -0700 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
06-02-2020 17:30:37.696 -0700 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
06-02-2020 17:53:37.315 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  ERROR - Failed opening "": No such file or directory
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  terminate called after throwing an instance of 'EventLoopException'
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:    what():  Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
06-02-2020 17:53:37.676 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!

I have tried to grok through Splunk answers and on Google; but, I couldn’t find much documentation/articles on what file ProcessRunner was trying to open? Could someone help me or point me to the right channel to understand how I can fix this issue.

Here’s my inputs.conf ’s script stanzas:

source = source-one
sourcetype = source-one
source = source-two
sourcetype = source-two
interval = 60
source = source-three
sourcetype = source-three
interval = 1800
source = source-four
sourcetype = source-four
interval = 1800

Thank you!

Labels (2)
0 Karma


Is it possible that you didn't do a proper error handling in the py scripts themselves ?
Are you trying to read some data in the python scripts and they will abort not in a graceful way while not being able to read the data ?

0 Karma


Thank you for your reply, efika!

My initital thought was that there was an unhandled exception in the script. I removed the .py script and was seeing the same error.

I'm in the process of testing each stanza by itself to see if the culprit is one of our scripts.

I observed this issue occurring in SUF 7.x and SUF 8.x. I have had the same scripts running for SUF 6.x (32-bit) and did not encounter any such error. The SUF 6.x machines have been running for months now using the same scripts.

If my understanding is right - ExecProcessor runs the script stanzas, and it doesn't cause a Splunk crash if an error occurs while executing the script. I wonder what ProcessRunner is and what it's trying to do?

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...