Monitoring Splunk

Monitoring Console shows inconsistent max memory on Indexer

afx
Contributor

Hi,
on a 7.2.4 Cluster my Indexers show memory usage of more than 80% in the initial screen of the monitoring console.
When I then go into the Resource Usage: Machine screen, I see that Splunk knows it has 7GB memory. in the Machine information on top, which also corresponds to the free command on the box.
But when I look at the snapshot or memory usage graph, I see nonsense:
I see 35.000 of 40.000 MB used.
Where does this come from and how can it be fixed?

thx
afx

0 Karma

harsmarvania57
Ultra Champion

I am looking at 7.2.7 Monitoring Console and in Resource Usage: Machine -> Snapshot -> Memory Usage (MB) is running REST API search | rest splunk_server=indexer_host /services/server/status/resource-usage/hostwide . Can you please try to run this search and check what it returns.

0 Karma

afx
Contributor

This returns nonsense for mem as well:
mem: 40892.957
men_used: 36003.289

thx
afx

0 Karma

harsmarvania57
Ultra Champion

Have you tried to restart splunk on Indexer ? If yes and it will not solve problem then I'll suggest to open case with splunk.

0 Karma

afx
Contributor

A rolling restart fixed this, thx!
But now I get
Some Data is Not Searchable
Search Factor is Not Met
Replication Factor is Not Met
And it seems that this comes from _audit.
Had that last year, it seemd to have resolved itself.

cheers
afx

0 Karma

harsmarvania57
Ultra Champion

Yes it will resolve automatically, that is due to bucket fixup activites.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...