Replicate Splunk dashboard using internal logs

[_pravin]

Hi All,

I have a use case where I want to check the total number of files in Linux that Splunk uses. I was able to get it done with a script on Linux and read the output via a forwarder.

I need to replicate the same using _introspection logs and get similar results, but the code below is the most I could get.

index=_introspection sourcetype=splunk_resource_usage component=PerProcess host=* 
| timechart span=1m sum(data.fd_used) AS total_open_files_by_FD

Has anyone faced the same issue to replicate the same Linux features in Splunk using internal logs?

Thanks,

Pravin

[isoutamo]

I haven’t look the latest ta for Unix and Linux to see what it’s collecting. Maybe this is one of those things and then you should use it.

Anyhow in UF introspection index is not collecting all those same values than it’s collecting on full enterprise instance. If you want to collect same kind of data set you need to add it into collections. There are instructions in help/docs how to do it. I can’t remember if those sets are some (probably not) or not, but you could try it.

[bowesmana]

What exactly are you trying to measure, an 'fd' is a file descriptor, not a file, and it applies to many things other than "files", e.g. a socket connections also has an 'fd'.

Also, each introspection log records a point in time state of a process, so the same process may log many events, so just summing the fds will not point to a total count of files used.

Can you give a better explanation of what you're trying to replicate.

[_pravin]

Hi @bowesmana ,

Thanks for the answer and the detailed explanation about the file descriptors.

 I am currently using an internal bash script to count all the processes used by Splunk on the server directly. Code attached below

splunk_process=0
for pid in $(pgrep -u autoengine splunk); do
    if [ -d "/proc/$pid/fd" ]; then
        count=$(ls "/proc/$pid/fd" 2>/dev/null | wc -l)
        splunk_process=$((splunk_process + count))
    fi
done

I was trying to replicate the same using introspection, or at least get the closest to this script functionality.

Thanks,

Pravin

[inventsekar]

Hi @_pravin 

lets try this (pls replace the host value):

index="_introspection" host=your_splunk_host sourcetype=splunk_resource_usage component=PerProcess | stats count(data.args) as data.args.count by data.args

this is listing out the processes and their counts. but it you just wanted the process count(unique processes count):

index="_introspection" host=your_splunk_host sourcetype=splunk_resource_usage component=PerProcess | dedup data.args | table data.args

most of these processes will be splunk's internal processes (the introspection log's purpose itself is to monitor the splunk's internals)

----------------------------------------------------------------------------------------------
If this post or any post addressed your question, could you pls:
Give it karma to show appreciation

PS - As of May 2026, my Karma Given is 2312 and my Karma Received is 497, lets revamp the Karma Culture!
Thanks and best regards, Sekar
--------------------------------------------------------------------------------------------