Forwarder cert expired, lost data ingestion during this time period

[sbrice18]

Splunk Forwarder V 6.5.2- Our certs expired at midnight and we renewed them at 10 am. Log ingestion picked back up at 10 am but everything prior is missing. This log file did not role over, shouldn't the forwarder know that there is a chunk of missing data from 12am to 10:00am? Do I need to re-ingest this log file or clean the fishbucket on the forwarder? Seems like this might be a bug? We also have indexer ack=true enabled.

[somesoni2]

Splunk should pickup those old values. Cleaning fishbucket would cause the whole file to be read again, along with all other data monitoring that was happening. Try restarting Splunk on the forwarder. Also, how much data is there on file? If it's a huge file, you can expect some delay till Splunk catches up.

[sbrice18]

Thanks for the reply! After pushing the new cert I restarted the forwarder and everything connected fine. The log file is only 48MB in size. We validated the crc and it looks like the forwarder tracked everything from file creation . Its been a few hours now but splunk still only shows data from 10am onward. I was going to do a one-shot but I don't want to duplicate the events from 10 am 🙂 I was thinking maybe the forwarder buffer ran over but at 30MB it should have retained the data without any issues. Its like the forwarder thinks it sent the data to the indexers. -odd (6.5.2 fwd /v 7.0.1 on indexers)