Deployment Architecture

Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411
Explorer

Hello,

There is a splunkforwarder that was stopped for a week without our knowledge and the data from that server was not indexed. Is there a way to retrieve that missing 7 days of data into splunk?

woodcock
Esteemed Legend

Just restart the forwarder. It will remember where it left off and forward in the missing logs.

pdantuuri0411
Explorer

Hi @woodcock,

That is what I thought was going to happen. Strangely it only retrieved the logs for the last few hours before it restarted(Restarted at 9 AM 01/14, only got logs from 12 AM 01/13)

Is was reading about manually injecting these missing logs using splunk oneshot but the problem is we have one log file with logs from dates 01/05 - 01/14. If I use onshot, I am suspecting there will be multiple entries and will mess up the report that will be generated using this data.

Please Advice

0 Karma

pdantuuri0411
Explorer

The issue is there is no time stamp in the log file for the entries. I counted back hours to check on what date the entries started to log. Now if I use oneshot, how will splunk know the date of the entries? I assume this will not work. Please let me know if there is a work around? Thank you

0 Karma

woodcock
Esteemed Legend

Are you using DATETIME_CONFIG = CURRENT? How is it timestamping them in the normal case?

0 Karma

pdantuuri0411
Explorer

This is the configuration I have for this particular source type. This is from props.conf

DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
SHOULD_LINEMERGE = false
disabled = false

0 Karma

woodcock
Esteemed Legend

You should probably set a custom datetime.xml to get the timestamp from the file/name.

0 Karma

woodcock
Esteemed Legend

So how is splunk setting _time for your events in the normal case?

0 Karma

woodcock
Esteemed Legend

just copy the log and trim it and then use oneshot on the modified file.

0 Karma

pdantuuri0411
Explorer

The issue is there is no time stamp in the log file for the entries. I counted back hours to check on what date the entries started to log. Now if I use oneshot, how will splunk know the date of the entries? I assume this will not work. Please let me know if there is a work around? Thank you

0 Karma

woodcock
Esteemed Legend

Copy the file. Edit it. Oneshot it. Delete the copy.

0 Karma

somesoni2
Revered Legend

If the log files are still there on the servers (with same name/location from where you were monitoring), those would get ingested automatically. If they've been rolled off to different location/name, you could create create a temporary monitoring input with same index/sourcetype and other setting but from different location to ingest those rolled logs. You can also use one shot method. See this for information on oneshot mehtod :
https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/MonitorfilesanddirectoriesusingtheCLI

0 Karma

pdantuuri0411
Explorer

@somesoni2,

Is was reading about manually injecting these missing logs using splunk oneshot but the problem is we have one log file with logs from dates 01/05 - 01/14. If I use onshot, I am suspecting there will be multiple entries and will mess up the report that will be generated using this data.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...