Monitoring Splunk
Highlighted

Monitoring Console shows inconsistent max memory on Indexer

Contributor

Hi,
on a 7.2.4 Cluster my Indexers show memory usage of more than 80% in the initial screen of the monitoring console.
When I then go into the Resource Usage: Machine screen, I see that Splunk knows it has 7GB memory. in the Machine information on top, which also corresponds to the free command on the box.
But when I look at the snapshot or memory usage graph, I see nonsense:
I see 35.000 of 40.000 MB used.
Where does this come from and how can it be fixed?

thx
afx

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

SplunkTrust
SplunkTrust

I am looking at 7.2.7 Monitoring Console and in Resource Usage: Machine -> Snapshot -> Memory Usage (MB) is running REST API search | rest splunk_server=indexer_host /services/server/status/resource-usage/hostwide . Can you please try to run this search and check what it returns.

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

Contributor

This returns nonsense for mem as well:
mem: 40892.957
men_used: 36003.289

thx
afx

0 Karma

Re: Monitoring Console shows inconsistent max memory on Indexer

SplunkTrust
SplunkTrust

Have you tried to restart splunk on Indexer ? If yes and it will not solve problem then I'll suggest to open case with splunk.

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

Contributor

A rolling restart fixed this, thx!
But now I get
Some Data is Not Searchable
Search Factor is Not Met
Replication Factor is Not Met
And it seems that this comes from _audit.
Had that last year, it seemd to have resolved itself.

cheers
afx

0 Karma
Highlighted

Re: Monitoring Console shows inconsistent max memory on Indexer

SplunkTrust
SplunkTrust

Yes it will resolve automatically, that is due to bucket fixup activites.

0 Karma