Getting Data In

Lost AWS events after ingestion

vishalduttauk
Communicator

I am in the middle of a Splunk migration. One of the tasks is to moved data from some sourcetypes onto the new servers using the | collect index=aws sourcetype=* command.

The numbers added up after running checks. I run the same checks again a day later and the numbers no longer match up.

Source 1 -> Old Splunk New Splunk Source 2 -> Old Splunk New Splunk
August 12,478,853 12,478,853   26,171,911 26,171,911

 

24 hours later

Source 1 -> Old Splunk New Splunk Source 2 -> Old Splunk New Splunk
  12,478,853 12,477,696   26,171,911 3,001,183

 

I've set the following stanza within the indexes.conf file on the deployment server. Also the index only contains 22gb of data. Can you help?

[aws]

coldPath = $SPLUNK_DB\$_index_name\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\$_index_name\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\$_index_name\thaweddb
frozenTimePeriodInSecs=94608000
Labels (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Most probably your data was rolled out due to either retention period or index/volume size limits.

0 Karma

tej57
Builder

Hello @vishalduttauk,

Can you provide the complete search that you used for migrating the data from one server to other?

Thanks,
Tejas. 

0 Karma

vishalduttauk
Communicator

index=aws but i ended up logging onto both servers and moving the whole index from "old" Splunk over to "new" Splunk

0 Karma
Get Updates on the Splunk Community!

UCC Framework: Discover Developer Toolkit for Building Technology Add-ons

The Next-Gen Toolkit for Splunk Technology Add-on Development The Universal Configuration Console (UCC) ...

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...

Splunk App Developers | .conf25 Recap & What’s Next

If you stopped by the Builder Bar at .conf25 this year, thank you! The retro tech beer garden vibes were ...