We are collecting syslog with a syslog collector, and dumping it to text files. Splunk ingests those txt files from the drive using the Splunk Universal Forwarder and everything works perfectly for all syslog events except the switch data from sourcetype cisco:ios. Every night there is a gap in the data from 12a-4a. Meanwhile, all other syslog data is indexed and reporting properly with nothing missing. Every sourcetype is using the same method and source syslog server. Its only this cisco:ios sourcetype during these hours. At 4:00am everything resumes like nothing ever happened. The text files contain data straight through the night, so its not with the syslog server or the data collecting.
I am completely stumped.
Backups dont run at those times.
Has anyone ever seen anything like this? I feel like my sanity is being tested 🙂
Couple of things that could be causing this:
If none of that helps, it would be good to have more information about your setup, OS, syslog service and configuration, forwarder configuration, and so on.