Cluster Master Error or indexer error - Splunk 7.2.3

[halbeisendv]

We started Splunk and while the messages were scrolling it stopped on the windows index. It just sits, no additional error messages in splunkd.log Not certain what's happening here.

07-29-2019 16:20:10.473 +0000 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/splunk/threathunting/db
07-29-2019 16:20:10.485 +0000 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=webserverlog
07-29-2019 16:20:10.494 +0000 INFO CMBucketId - CMIndexId: New indexName=webserverlog inserted, mapping to id=27
07-29-2019 16:20:12.798 +0000 INFO DatabaseDirectoryManager - idx=webserverlog Writing a bucket manifest in hotWarmPath='/splunk/webserverlog/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
07-29-2019 16:20:12.867 +0000 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/splunk/webserverlog/db
07-29-2019 16:20:12.877 +0000 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=wiki
07-29-2019 16:20:12.884 +0000 INFO CMBucketId - CMIndexId: New indexName=wiki inserted, mapping to id=28
07-29-2019 16:20:13.662 +0000 INFO DatabaseDirectoryManager - idx=wiki Writing a bucket manifest in hotWarmPath='/splunk/wiki/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
07-29-2019 16:20:13.684 +0000 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/splunk/wiki/db
07-29-2019 16:20:13.692 +0000 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=windows
07-29-2019 16:20:13.696 +0000 INFO CMBucketId - CMIndexId: New indexName=windows inserted, mapping to id=29

[harsmarvania57]

Can you please provide more information ? What is happening, splunk crashed on CM or Indexers or any other issue ?

[halbeisendv]

Nothing is happening. The log file stops logging at the precise location listed above on multiple restarts.

[harsmarvania57]

I am still not getting what happens after logs stop updating, once logs stopped updating Splunk process crashed on Indexer or Cluster Master ? Have you checked permission of $SPLUNK_HOME/var/lib/splunk/windows directory and sub-directories on Indexers ?

Any ERROR or WARN log messages on Cluster Master ?