Getting Data In

The curious case of missing events

timrich66
Communicator

At the beginning of February this year we started ingesting events from Autosys prod and non-prd servers.  All was going well until the end of the month.  At 23:59 on 28th Feb, events stopped appearing in the index to which they are being ingested.  A few days later they started again at midnight.  None of the missed events were collected.  The log files for Autosys roll over at midnight so I would not expect events during the missing time period to be collected as the files would have been renamed.

This has happened every month since.  In March the events started coming in on 3rd, April it was 4th, May on  5th and June they appeared again on the 6th.  The only pattern I can see is that events reappear on the day of the month that equals the number of the month - ie 3/3, 4/4, 5/5 and 6/6.  If that is the case, then events will not appear in July until 7th.

How can this be?  I can see no _internal messages that help.

Can someone shed some light on this?  Thanks

0 Karma

timrich66
Communicator

To add - 

This does not happen on any other ingestion.  The servers are still phoning home and sending infra metrics to ITSI.  I have redeployed inputs.conf to the servers.  One is prod, one non-prd.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...