Replicate Splunk dashboard using internal logs

[_pravin]

Hi All,

I have a use case where I want to check the total number of files in Linux that Splunk uses. I was able to get it done with a script on Linux and read the output via a forwarder.

I need to replicate the same using _introspection logs and get similar results, but the code below is the most I could get.

index=_introspection sourcetype=splunk_resource_usage component=PerProcess host=* 
| timechart span=1m sum(data.fd_used) AS total_open_files_by_FD

Has anyone faced the same issue to replicate the same Linux features in Splunk using internal logs?

Thanks,

Pravin

[bowesmana]

What exactly are you trying to measure, an 'fd' is a file descriptor, not a file, and it applies to many things other than "files", e.g. a socket connections also has an 'fd'.

Also, each introspection log records a point in time state of a process, so the same process may log many events, so just summing the fds will not point to a total count of files used.

Can you give a better explanation of what you're trying to replicate.

[_pravin]

Hi @bowesmana ,

Thanks for the answer and the detailed explanation about the file descriptors.

 I am currently using an internal bash script to count all the processes used by Splunk on the server directly. Code attached below

splunk_process=0
for pid in $(pgrep -u autoengine splunk); do
    if [ -d "/proc/$pid/fd" ]; then
        count=$(ls "/proc/$pid/fd" 2>/dev/null | wc -l)
        splunk_process=$((splunk_process + count))
    fi
done

I was trying to replicate the same using introspection, or at least get the closest to this script functionality.

Thanks,

Pravin

[inventsekar]

Hi @_pravin 

lets try this (pls replace the host value):

index="_introspection" host=your_splunk_host sourcetype=splunk_resource_usage component=PerProcess | stats count(data.args) as data.args.count by data.args

this is listing out the processes and their counts. but it you just wanted the process count(unique processes count):

index="_introspection" host=your_splunk_host sourcetype=splunk_resource_usage component=PerProcess | dedup data.args | table data.args

most of these processes will be splunk's internal processes (the introspection log's purpose itself is to monitor the splunk's internals)

----------------------------------------------------------------------------------------------
If this post or any post addressed your question, could you pls:
Give it karma to show appreciation

PS - As of May 2026, my Karma Given is 2312 and my Karma Received is 497, lets revamp the Karma Culture!
Thanks and best regards, Sekar
--------------------------------------------------------------------------------------------