Deployment Architecture

Why does the Monitoring Console not update with new indexer information?

kairisrj
Explorer

I've replaced 2 or my 4 indexers in the cluster on an AWS deployment. The Indexer Cluster Status shows the right list of indexers but the main monitoring console overview shows the original 4 with 2 as down. How can I get the monitoring console to refresh so it sees the right instances?

1 Solution

lguinn2
Legend

I've had old servers remain in the Monitoring Console before. If the Indexer Clustering dashboard on the cluster master is correct, then try this: simply go to the Settings tab in the Monitoring Console and choose General Setup. (Not the overall Settings menu.)
Take a look at the servers listed. If everything looks reasonable, just click Save. You might also do something similar for Forwarder Setup within the MC.

This may clear older entries from the monitoring tables.

View solution in original post

lguinn2
Legend

I've had old servers remain in the Monitoring Console before. If the Indexer Clustering dashboard on the cluster master is correct, then try this: simply go to the Settings tab in the Monitoring Console and choose General Setup. (Not the overall Settings menu.)
Take a look at the servers listed. If everything looks reasonable, just click Save. You might also do something similar for Forwarder Setup within the MC.

This may clear older entries from the monitoring tables.

DBattisto
Communicator

I'm here from the future to say that this also fixed the issue I had.

I couldn't figure out how to Google this issue. I completely rebuilt and replaced my indexer, and spent about an hour doing a bunch of grep commands on my search head to see if the IP address for the old indexer still remained in one of the config files.

Nope. Just gotta click 'save'. ...how intuitive 🙂

0 Karma

kairisrj
Explorer

That did the trick

0 Karma

nawazns5038
Builder

I just added a new indexer to the indexer cluster and the cluster master shows the updated list but not the monitoring console. It shows that the state of the new indexer as new . As a result does not have the updated list on the number of instances.

How do I change it to configured ?

0 Karma

lguinn2
Legend

To update the list of names in the Monitoring Console, you need to go to Monitoring Console >> Settings >> General Setup.
You should see your "new" server in the list. You may need to edit its roles, but you might not need to edit anything.
But ALWAYS click on Apply Changes. This will ensure that the Monitoring Console (MC) updates its configuration to the same settings that you see on the screen. When you enter the General Setup windows, the MC will show you all the servers that it finds, but the new servers (or settings) are not actually configured in the MC until you click Apply Changes.

cblanton
Communicator

+1, my settings were correct, but Apply Changes was apparently missing. fixed it immediately

0 Karma

nawazns5038
Builder

Yes.. It worked .. Thanks !!!

0 Karma

lguinn2
Legend

Do the old indexers appear when you look at the Indexer Clustering dashboard on the cluster master? If yes, then that is why the old indexers are still appearing in the monitoring console. The cluster master will not remove the old indexers from all of its internal tables until you run the following command:

splunk remove cluster-peers -peers guid1,guid2

To get the guid of an indexer, go to the Indexer Clustering dashboard and choose the Indexers tab. Use the small arrow to the left of the indexer name to view its details, including its guid. You will definitely want to cut-and-paste this information into the remove cluster-peers command!

0 Karma

kairisrj
Explorer

No, index clustering shows the correct list. I ran the command you suggested, but Monitoring Console is not updated.

0 Karma

lguinn2
Legend

To update the list of names in the Monitoring Console, you need to go to Monitoring Console >> Settings >> General Setup.
You should see your "new" server in the list. You may need to edit its roles, but you might not need to edit anything.
But ALWAYS click on Apply Changes. This will ensure that the Monitoring Console (MC) updates its configuration to the same settings that you see on the screen. When you enter the General Setup windows, the MC will show you all the servers that it finds, but the new servers (or settings) are not actually configured in the MC until you click Apply Changes.

0 Karma

sherm77
Path Finder

I had to edit the ~/etc/apps/splunk_monitoring_console/lookups/assets.csv file to remove a decomm'd indexer, and also remove it from ~/etc/apps/splunk_monitoring_console/local/splunk_monitoring_console_assets.conf. Once I did that, it disappeared from the monitoring console without restart.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...