Getting Data In

The curious case of missing events

timrich66
Communicator

At the beginning of February this year we started ingesting events from Autosys prod and non-prd servers.  All was going well until the end of the month.  At 23:59 on 28th Feb, events stopped appearing in the index to which they are being ingested.  A few days later they started again at midnight.  None of the missed events were collected.  The log files for Autosys roll over at midnight so I would not expect events during the missing time period to be collected as the files would have been renamed.

This has happened every month since.  In March the events started coming in on 3rd, April it was 4th, May on  5th and June they appeared again on the 6th.  The only pattern I can see is that events reappear on the day of the month that equals the number of the month - ie 3/3, 4/4, 5/5 and 6/6.  If that is the case, then events will not appear in July until 7th.

How can this be?  I can see no _internal messages that help.

Can someone shed some light on this?  Thanks

0 Karma

timrich66
Communicator

To add - 

This does not happen on any other ingestion.  The servers are still phoning home and sending infra metrics to ITSI.  I have redeployed inputs.conf to the servers.  One is prod, one non-prd.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Rounding off the Splunk Dashboard Contest

What does a contest-winning Splunk dashboard look like? In this case, it isn't in a browser tab at all. It ...

A Four Part Event Series: AI + Observability: AI Agents, LLMs, Apps, & Infrastructure

AI + Observability: AI Agents, LLMs, Apps, & Infrastructure The rapid evolution of artificial intelligence ...

Splunk Technical Support Is Moving to Cisco Support Tools

Introduction Splunk technical support is transitioning to Cisco’s support environment. This change brings ...