Splunk Dev

Splunk skipping some messages to read from file

ankithreddy777
Contributor

I have a log files updated in realtime. From past two years these files are ingested to splunk without issues. Suddenly I found a weird issue, where splunk skipping some messages in a file to ingest here and there . I found around 10 percent of the messages are skipped.

I am not sure where is the root cause. I can understant if it skips complete file, but its skipping messages here and there in a single file. Its happening for all files ingested from that source. No configs are changed.

I cannot search for any field value in the missing message in splunk.

Should I begin troubleshooting for problems on indexer side or forwarder side.

May I know what might cause such type of issue.

0 Karma

ankithreddy777
Contributor

Hi kamlesh ,
Thank you for your reply.
I checked disk space and errors in splunkd.
There are no errors.
I have observed that while searching for data, I can only get data from 17 indexers instead of 20 indexers. Search for current index does not show any results from remaining three indexers exactly from the date we observed data is missing.
But these three indexers are up and healthy and show results for other indexes.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi Ankithreddy777,

There might be any possibilities for this issue. But I think it should be below:

  • if you have recently started forwarding new events in the different index then check the existence of the index and check splunkd.log of the indexer.
  • It might be disk space or disk related issue.

you can troubleshoot the problem by following below link.

https://wiki.splunk.com/Community:TroubleshootingIndexing

Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...