Monitoring Splunk
Highlighted

Monitoring Console shows inconsistent max memory on Indexer

Contributor

Hi,
on a 7.2.4 Cluster my Indexers show memory usage of more than 80% in the initial screen of the monitoring console.
When I then go into the Resource Usage: Machine screen, I see that Splunk knows it has 7GB memory. in the Machine information on top, which also corresponds to the free command on the box.
But when I look at the snapshot or memory usage graph, I see nonsense:
I see 35.000 of 40.000 MB used.
Where does this come from and how can it be fixed?

thx
afx

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

SplunkTrust
SplunkTrust

I am looking at 7.2.7 Monitoring Console and in Resource Usage: Machine -> Snapshot -> Memory Usage (MB) is running REST API search | rest splunk_server=indexer_host /services/server/status/resource-usage/hostwide . Can you please try to run this search and check what it returns.

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

Contributor

This returns nonsense for mem as well:
mem: 40892.957
men_used: 36003.289

thx
afx

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

SplunkTrust
SplunkTrust

Have you tried to restart splunk on Indexer ? If yes and it will not solve problem then I'll suggest to open case with splunk.

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

Contributor

A rolling restart fixed this, thx!
But now I get
Some Data is Not Searchable
Search Factor is Not Met
Replication Factor is Not Met
And it seems that this comes from _audit.
Had that last year, it seemd to have resolved itself.

cheers
afx

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

SplunkTrust
SplunkTrust

Yes it will resolve automatically, that is due to bucket fixup activites.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.