topic Re: Indexer in Splunk Search

Indexer

SN1 — Tue, 11 Mar 2025 06:43:31 GMT

I want to get total memory allocated on 1 indexer and how much memory it is using. so that i could get remaining disk space left.

Re: Indexer

kiran_panchavat — Tue, 11 Mar 2025 07:15:11 GMT

@SN1

Splunk indexers store data on disk in indexes, and the "total memory allocated" could refer to the total disk space available on the partition where Splunk stores its data (typically under $SPLUNK_HOME/var/lib/splunk). The "memory it is using" would then be the disk space consumed by the indexes, and the "remaining disk space left" would be the free space on that partition.

| rest /services/server/status/partitions-space splunk_server=*
| eval totalGB = round(capacity/1024/1024, 2)
| eval freeGB = round(free/1024/1024, 2)
| eval usedGB = round((capacity - free)/1024/1024, 2)
| table splunk_server, totalGB, usedGB, freeGB

To get the total memory allocated on an indexer and its current usage (which is different from disk space), you can use the following Splunk commands:

For memory information:

| rest /services/server/status/resource-usage/hostwide splunk_server=*

This will show you key metrics including:

Total physical memory on the system
Memory currently in use
Available memory

If you're specifically interested in Splunk's memory usage:

For disk space information (which seems to be what you're actually asking about):

For specific index volume usage:

Note that memory usage and disk space are different resources. Memory refers to RAM available for processing, while disk space refers to storage capacity for data. Your question mentions memory but ends with disk space, so I've provided commands for both.

Re: Indexer

gcusello — Tue, 11 Mar 2025 07:44:31 GMT

Hi @SN1 ,

in addition to the perfect answer of @kiran_panchavat ,

you could install the Splunk_TA_nix add-on ( https://splunkbase.splunk.com/app/833 ) and extract additional information from the linux system you're using.

Ciao.

Giuseppe

Re: Indexer

PickleRick — Tue, 11 Mar 2025 09:18:19 GMT

Well... TA_nix without careful tweaking what it reports can be a handful. It's just a bunch of ziptie and duct-tape connected scripts giving you some relatively unfriendly output. And if you just install it and enable all inputs, it can get noisy.

Re: Indexer

PickleRick — Tue, 11 Mar 2025 09:21:49 GMT

The difference between disk usage and memory has already been pointed out.

There is also one more thing worth noting - the disk utilization on indexers is usually managed by adjusting retention parameters (you might also get some additional usage from knowledge bundles and intermediate results but they are rarely very significant). And the memory usage can vary greatly depending on the current load at the time of checking since memory is used mostly for searching. So the more more complicated searches you're running at any given moment, the higher memory usage.

Re: Indexer

livehybrid — Tue, 11 Mar 2025 14:08:36 GMT

Hi @SN1

Some good answers here, its worth noting that for me

| rest /services/server/status/partitions-space

doesnt give me the right data, and it can depend on how your partitions are configured (e.g. multiple partitions for hot/warm/cold etc)

If you're using Linux then its worth also checking something as simple as in the linux command line

df -h

This will list all the filesystems on the server and show you the size, used and available disk space.

I'd definitely recommend setting up some proper monitoring using the Splunk TA for *Nix to cover your servers and cover all partitions and filesystems.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

Re: Indexer

SN1 — Wed, 12 Mar 2025 07:46:53 GMT

when i am running this search it is giving 16GB as total_GB while our total size is 16Tb.

Re: Indexer

livehybrid — Wed, 12 Mar 2025 08:18:23 GMT

Hi @SN1

This is because the values from the endpoint are in MB but are being divided by 1024 twice in this search hence they become in TB.
try switching 1024/1024 for just 1024 in each occurrence and see if that resolves for you 🙂

Will

Re: Indexer

livehybrid — Wed, 12 Mar 2025 08:44:54 GMT

You can also use an mstats query to query to _metrics index:

| mstats latest(_value) as val WHERE index=_metrics AND metric_name=spl.intr.disk_objects.Partitions.data.* by data.mount_point, metric_name | rename data.mount_point as mount_point | eval metric_name=replace(metric_name,"spl.intr.disk_objects.Partitions.data.","") | eval {metric_name}=val | stats latest(*) as * by mount_point | eval free = if(isnotnull(available), available, free) | eval usage = round((capacity - free) / 1024, 2) | eval capacity = round(capacity / 1024, 2) | eval compare_usage = usage." / ".capacity | eval pct_usage = round(usage / capacity * 100, 2) | stats first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point | rename mount_point as "Mount Point", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will