data on indexers disappeared

[gitingua]

I have about 10 indexers, a cluster. For some reason my "master node" turned off and when it turned on. my data has disappeared. there were 18 million data, and it became 9 million for what reason could this happen? I can't find anything in the logs. HELP PLS

[richgalloway]

Indexer clusters can continue to function without a Manager Node/Cluster Manager so nothing should have happened to your cluster while the MN/CM was off.

Please tell us more about the problem.  How long was the MN off?  How did you discover your data had "disappeared"?  Where there any changes on the MN or indexers while the MN was off?

[gitingua]

@richgalloway. Hello ! Thanks for answering. 

SH shows how much data is total and after the master has been turned off and on. I saw that the data was missing 2 times.

I got a message now

Search peer SH has the following message: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.

the problem arose after switching off and on the MN

[richgalloway]

I don't trust the Data Summary, but that error message is very telling.  Have you reviewed the system health?  There should be a red dot in the menu bar.  Clicking it will bring up the system health dashboard.  Click on other red icons to get details.  Also, use the Monitoring Console to check on the indexer queues.  Verify the storage system is healthy, too.

[gitingua]

@richgalloway  how to check the storage system?) Thanks 

[richgalloway]

Ideally, it would be sending its logs and metrics to Splunk so would you just need to craft a query to check the state of the storage system.  The details are specific to your environment.

Failing that, you can talk to the admin of the storage system.