Getting Data In

The curious case of missing events

timrich66
Communicator

At the beginning of February this year we started ingesting events from Autosys prod and non-prd servers.  All was going well until the end of the month.  At 23:59 on 28th Feb, events stopped appearing in the index to which they are being ingested.  A few days later they started again at midnight.  None of the missed events were collected.  The log files for Autosys roll over at midnight so I would not expect events during the missing time period to be collected as the files would have been renamed.

This has happened every month since.  In March the events started coming in on 3rd, April it was 4th, May on  5th and June they appeared again on the 6th.  The only pattern I can see is that events reappear on the day of the month that equals the number of the month - ie 3/3, 4/4, 5/5 and 6/6.  If that is the case, then events will not appear in July until 7th.

How can this be?  I can see no _internal messages that help.

Can someone shed some light on this?  Thanks

0 Karma

timrich66
Communicator

To add - 

This does not happen on any other ingestion.  The servers are still phoning home and sending infra metrics to ITSI.  I have redeployed inputs.conf to the servers.  One is prod, one non-prd.

0 Karma
Get Updates on the Splunk Community!

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...

Splunk MCP & Agentic AI: Machine Data Without Limits

Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization uses ...