ERROR ExecProcessor - Message from runFlume.sh

[vanand]

Folks, APM_dyantrace runFlume.sh failed to process the events and logging following messages in splunk log. Any insight what causing this.

04-06-2015 03:05:01.811 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" 2015-04-06 03:05:01,811 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
04-06-2015 03:05:01.811 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" org.apache.flume.EventDeliveryException: Failed to process transaction
04-06-2015 03:05:01.811 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" Caused by: java.io.IOException: Stale file handle
= ... 4 more
04-06-2015 03:06:02.819 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" 2015-04-06 03:06:02,819 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
04-06-2015 03:06:02.819 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" org.apache.flume.EventDeliveryException: Failed to process transaction
04-06-2015 03:06:02.819 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
at java.lang.Thread.run(Thread.java:662)
04-06-2015 03:06:02.819 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" Caused by: java.io.IOException: Stale file handle
04-06-2015 03:06:02.819 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh" at java.io.FileOutputStream.writeBytes(Native Method)
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:195)
04-06-2015 03:06:02.820 -0700 ERROR ExecProcessor - message from "/splunk/etc/apps/APM_dynatrace/bin/runFlume.sh"

Thanks,

[Dynatrace · Accepted solution]

Stale file handles usually result when a file or directory is removed while an application is attempting to write to that file. There might be a mismatch in your environment between Flume rolling over new files versus Splunk taking in those files.

The Flume configuration distributed with our Splunk application is a good baseline but might not work for all environments. I would suggest decreasing the sink.rollInterval in /splunk/etc/apps/APM_dynatrace/bin/flume-conf.properties until you no longer see this error. Decreasing this value will cause Flume to roll to a new file more frequently.

hope this helps!

[vanand]

That works. Thanks.