Getting Data In

The curious case of missing events


At the beginning of February this year we started ingesting events from Autosys prod and non-prd servers.  All was going well until the end of the month.  At 23:59 on 28th Feb, events stopped appearing in the index to which they are being ingested.  A few days later they started again at midnight.  None of the missed events were collected.  The log files for Autosys roll over at midnight so I would not expect events during the missing time period to be collected as the files would have been renamed.

This has happened every month since.  In March the events started coming in on 3rd, April it was 4th, May on  5th and June they appeared again on the 6th.  The only pattern I can see is that events reappear on the day of the month that equals the number of the month - ie 3/3, 4/4, 5/5 and 6/6.  If that is the case, then events will not appear in July until 7th.

How can this be?  I can see no _internal messages that help.

Can someone shed some light on this?  Thanks

0 Karma


To add - 

This does not happen on any other ingestion.  The servers are still phoning home and sending infra metrics to ITSI.  I have redeployed inputs.conf to the servers.  One is prod, one non-prd.

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...