Unable to get estreamer logs processed in Splunk

[mohammed7860]

Hi Splunkers:

I am having an issue with streamer data feed in splunk for a long time. The issue is that the eStreamer for Splunk app is properly configured in Splunk. We are using one of the Heavy Forwarder server (Linux) to get the streamer logs written. The logs are written on the HW server which I attested by checking on the CLI. Checking the streamer debug logs I see that there are no error messages, the last message written I see is 'Entering Event Loop'. There are no errors either on splunkd.log

The logs get displayed on Search Head for a couple of hours and then don't show up at all. I am always required to disable and enable the management console connection, and restart the Splunk Heavy Forwarder server several times to get the streamer data feeds starting to ingest into Splunk.

Can someone guide me what is causing this problem?

Doesn't seem like an streamer agent issue, as logs are regularly getting written on the Heavy Forwarder server. No errors on splunkd logs or Estreamer debug logs.

Any help is greatly appreciated.

Thanks,

Mohammed Mohiuddin

[ehorton_splunk · Accepted solution]

If the eStreamer server is still sending logs to $SPLUNK_HOME/etc/apps/eStreamer/log and you can see them there, it would appear to be some sort of an issue with the Splunk inputs that is causing the problem; not eStreamer specifically.

Since this is a recurring issue, try putting the splunk logs in debug for the associated process on the HWF:
1) Go to $SPLUNK_HOME/etc/
2) Make a backup copy of the log.cfg file
3) Edit the log.cfg file and change the following lines from INFO:
category.TailingProcessor=DEBUG

category.WatchedFile=DEBUG

category.TailReader=DEBUG

It is also possible for there to be issues with other devices sending data to the same port that is being used to receive the eStreamer logs; so check the inputs on the HWF and verify that this is not happening.