topic Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped. in Deployment Architecture

Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411 — Mon, 14 Jan 2019 18:31:09 GMT

Hello,

There is a splunkforwarder that was stopped for a week without our knowledge and the data from that server was not indexed. Is there a way to retrieve that missing 7 days of data into splunk?

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

somesoni2 — Mon, 14 Jan 2019 20:07:45 GMT

If the log files are still there on the servers (with same name/location from where you were monitoring), those would get ingested automatically. If they've been rolled off to different location/name, you could create create a temporary monitoring input with same index/sourcetype and other setting but from different location to ingest those rolled logs. You can also use one shot method. See this for information on oneshot mehtod :
https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/MonitorfilesanddirectoriesusingtheCLI

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

woodcock — Mon, 14 Jan 2019 20:33:36 GMT

Just restart the forwarder. It will remember where it left off and forward in the missing logs.

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411 — Mon, 14 Jan 2019 21:24:34 GMT

Hi @woodcock,

That is what I thought was going to happen. Strangely it only retrieved the logs for the last few hours before it restarted(Restarted at 9 AM 01/14, only got logs from 12 AM 01/13)

Is was reading about manually injecting these missing logs using splunk oneshot but the problem is we have one log file with logs from dates 01/05 - 01/14. If I use onshot, I am suspecting there will be multiple entries and will mess up the report that will be generated using this data.

Please Advice

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411 — Mon, 14 Jan 2019 21:25:23 GMT

@somesoni2,

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

woodcock — Mon, 14 Jan 2019 21:26:14 GMT

just copy the log and trim it and then use oneshot on the modified file.

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411 — Mon, 14 Jan 2019 22:55:59 GMT

The issue is there is no time stamp in the log file for the entries. I counted back hours to check on what date the entries started to log. Now if I use oneshot, how will splunk know the date of the entries? I assume this will not work. Please let me know if there is a work around? Thank you

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411 — Mon, 14 Jan 2019 22:56:22 GMT

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

woodcock — Tue, 15 Jan 2019 03:10:46 GMT

Copy the file. Edit it. Oneshot it. Delete the copy.

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

woodcock — Tue, 15 Jan 2019 03:12:40 GMT

Are you using DATETIME_CONFIG = CURRENT? How is it timestamping them in the normal case?

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

pdantuuri0411 — Tue, 29 Sep 2020 22:46:25 GMT

This is the configuration I have for this particular source type. This is from props.conf

DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
SHOULD_LINEMERGE = false
disabled = false

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

woodcock — Tue, 15 Jan 2019 22:11:16 GMT

So how is splunk setting _time for your events in the normal case?

Re: Retrieved missing data that was not forwarded when the splunk forwarder is stopped.

woodcock — Tue, 03 Dec 2019 15:12:35 GMT

You should probably set a custom datetime.xml to get the timestamp from the file/name.