We noticed that, right after a log rotation, the data is not being indexed until the next log rotation. That is, lets say, one file was rotated at 8 AM (until which the data was already indexed). The next file is written from 8 AM to 7 PM. But this file is not indexed until around 7 PM.
We are on a Universal forwarder 7.0.3
Below is the monitoring stanza
[monitor:///opt/mapr/hadoop/hadoop/logs/*nodemanager*]
sourcetype = my_st
index = my_index
disabled = 0
ignoreOlderThan = 2h
We added ignoreOlderThan = 2h recently to see if it helps. But the issue still persists.
The latest file will be with yarn-mapr-nodemanager-host_name.log and the latest archived file be with yarn-mapr-nodemanager-host_name.log.1 .
What is interesting is intermittently on certain servers, the current file gets indexed only at the time of its roll/archival i.e. (lets say after 10-11 hours) but with actual file name but not archive file name. And the issue of live/current file not getting indexed on time does not happen all the time. The next live file might get indexed on time. There should be an ideal settings to avoid this.
Any insights on this will be helpful.
Whatever Splunk says about handling log rotation files, seems to have some bug. Are we missing anything here? Please suggest.
... View more