Monitoring Splunk

How to resolve error when Splunkd intermittently crashes while streaming telemetry data on Universal Forwarder: "ProcessRunner: No such file or directory"?



I’m working on streaming telemetry data to Splunk. I use Splunk Universal Forwarder v7 x86_64 to capture and stream data to Splunk Enterprise 8.

I use the script:// to capture data and run them at certain specified intervals. The data is being successfully streamed to the server. But, intermittently, splunkd (SUF) crashes, and I see the following error in my splunkd.log.

06-02-2020 17:12:27.975 -0700 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
06-02-2020 17:12:27.993 -0700 INFO  WatchedFile - Will begin reading at offset=1182 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
06-02-2020 17:12:56.832 -0700 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
06-02-2020 17:30:37.696 -0700 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
06-02-2020 17:53:37.315 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  ERROR - Failed opening "": No such file or directory
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  terminate called after throwing an instance of 'EventLoopException'
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:    what():  Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
06-02-2020 17:53:37.676 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!

I have tried to grok through Splunk answers and on Google; but, I couldn’t find much documentation/articles on what file ProcessRunner was trying to open? Could someone help me or point me to the right channel to understand how I can fix this issue.

Here’s my inputs.conf ’s script stanzas:

source = source-one
sourcetype = source-one
source = source-two
sourcetype = source-two
interval = 60
source = source-three
sourcetype = source-three
interval = 1800
source = source-four
sourcetype = source-four
interval = 1800

Thank you!

Labels (2)
0 Karma


Is it possible that you didn't do a proper error handling in the py scripts themselves ?
Are you trying to read some data in the python scripts and they will abort not in a graceful way while not being able to read the data ?

0 Karma


Thank you for your reply, efika!

My initital thought was that there was an unhandled exception in the script. I removed the .py script and was seeing the same error.

I'm in the process of testing each stanza by itself to see if the culprit is one of our scripts.

I observed this issue occurring in SUF 7.x and SUF 8.x. I have had the same scripts running for SUF 6.x (32-bit) and did not encounter any such error. The SUF 6.x machines have been running for months now using the same scripts.

If my understanding is right - ExecProcessor runs the script stanzas, and it doesn't cause a Splunk crash if an error occurs while executing the script. I wonder what ProcessRunner is and what it's trying to do?

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...