I have a log files updated in realtime. From past two years these files are ingested to splunk without issues. Suddenly I found a weird issue, where splunk skipping some messages in a file to ingest here and there . I found around 10 percent of the messages are skipped.
I am not sure where is the root cause. I can understant if it skips complete file, but its skipping messages here and there in a single file. Its happening for all files ingested from that source. No configs are changed.
I cannot search for any field value in the missing message in splunk.
Should I begin troubleshooting for problems on indexer side or forwarder side.
Hi kamlesh ,
Thank you for your reply.
I checked disk space and errors in splunkd.
There are no errors.
I have observed that while searching for data, I can only get data from 17 indexers instead of 20 indexers. Search for current index does not show any results from remaining three indexers exactly from the date we observed data is missing.
But these three indexers are up and healthy and show results for other indexes.