Splunk Universal Forwarder is not able to send monitored file's logs to Splunk Indexers though sending internal logs properly

[MousumiChowdhur]

Hi All,

I have six forwarders and two indexers to which these are supposed to send data. The six forwarders have multiple instances of forwarders i.e., each having three instances. There are three active files of 500mb each which are supposed to be monitored. These three 500mb files are distributed among three instances of forwarders in each forwarder. After setting up the forwarders and doing all the configurations, I started the input for all the six forwarders. Out of six forwarders, 4 are sending data properly but 2 are monitoring the files and not sending any data. Internal logs are coming from all six forwarders. There is no internal error that I'm getting. Also at the time of data input, I was able to get the "TailingProcessor" in the internal logs for the sources. But after that the logs never came. I'm not able to find what the issue could be.

Can anybody please help me to solve this issue?

[MousumiChowdhur]

Hi!! Thanks for the quick response.

I tried running the search for index=* and timepicker value set to All time. But I still can't see any data from those two particular sources. Also I checked, the destination index exists on the two indexers as the data from the other four forwarders are showing up in that particular index. There is no internal error also!

[woodcock]

Try running a search for index=* with a timepicker value of All time. Sometimes the events do not go into the right place and sometimes they are not timestamped correctly and so they are sent "to the future" and will only show up in searches way late, as reality creeps towards them. If you do not see the events, they make sure that the destination index for the events exists on your indexers (there are error logs on the indexers when events come for indexes that are not configured so you can search for those with index=_* warn* OR err*).