topic Re: Lost AWS events after ingestion in Getting Data In

Lost AWS events after ingestion

vishalduttauk — Thu, 12 Sep 2024 14:22:30 GMT

I am in the middle of a Splunk migration. One of the tasks is to moved data from some sourcetypes onto the new servers using the | collect index=aws sourcetype=* command.

The numbers added up after running checks. I run the same checks again a day later and the numbers no longer match up.

Source 1 ->	Old Splunk	New Splunk	Source 2 ->	Old Splunk	New Splunk
August	12,478,853	12,478,853		26,171,911	26,171,911

24 hours later

Source 1 ->	Old Splunk	New Splunk	Source 2 ->	Old Splunk	New Splunk
	12,478,853	12,477,696		26,171,911	3,001,183

I've set the following stanza within the indexes.conf file on the deployment server. Also the index only contains 22gb of data. Can you help?

[aws]

coldPath = $SPLUNK_DB\$_index_name\colddb enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB\$_index_name\db maxTotalDataSizeMB = 512000 thawedPath = $SPLUNK_DB\$_index_name\thaweddb frozenTimePeriodInSecs=94608000

Re: Lost AWS events after ingestion

tej57 — Wed, 18 Jun 2025 07:40:53 GMT

Hello @vishalduttauk,

Can you provide the complete search that you used for migrating the data from one server to other?

Thanks,
Tejas.

Re: Lost AWS events after ingestion

vishalduttauk — Thu, 19 Jun 2025 14:55:01 GMT

index=aws but i ended up logging onto both servers and moving the whole index from "old" Splunk over to "new" Splunk

Re: Lost AWS events after ingestion

PickleRick — Thu, 19 Jun 2025 20:39:00 GMT

Most probably your data was rolled out due to either retention period or index/volume size limits.