On-prem to Smartstore migration issues- Why are the event counts not matching?

[bsrikanthreddy5]

Hi, 

We are migrating our cluster from on-prem to a smart-store enabled cluster in AWS, a few indexes at a time, during which process event counts is not matching in some cases.

Case1: Eventcount in aws cluster is less than event count in on-prem

Case2: Eventcount in aws cluster is more than event count in on-prem

Any idea what might cause the event count, not to match? 

[richgalloway]

How are you migrating the data?

How are you counting the data in each location?  Are you sure the data has finished migrating before you count it?

[bsrikanthreddy5]

I have added below stanza on my on-prem indexes.conf and server.conf

indexes.conf

[volume:splunk]
storageType = remote
path = s3://<bucketName>
remote.s3.endpoint = https://s3.us-east-2.amazonaws.com
remote.s3.encryption = sse-kms
remote.s3.kms.key_id =
remote.s3.access_key =
remote.s3.secret_key =

Server.conf
[cachemanager]
max_concurrent_uploads = 2
eviction_policy = noevict
hotlist_recency_secs = 3888000

adding remotePath = volume:splunk/$_index_name specific index which I plan to migrate and rolling restart the indexer cluster. 

I am using below commands to compare  the event count .  
| tstats count where index=<IndexName> by index
| dbinspect index=<IndexName> | dedup bucketId |  stats sum(eventCount) by index

I am monitoring the migration status using  below command on CM.

/opt/splunk/bin/splunk search "|rest /services/admin/cacheman/_metrics |fields splunk_server migration*" -auth admin:$(cat /etc/splunk/password) .