Blacklist Question for local proc directory

[taskmaster]

I'm new to Splunk... I'm currently running Splunk on an Ubuntu system.  I've noticed that the /proc directory is showing as using 10TB out of my 14TB disk and in the Splunk GUI under “Overview > Disk Usage” my disk is reflecting this usage. 

From my understanding, /proc is a virtual filesystem and doesn’t actually consume real disk space. However, Splunk appears to be interpreting it as used space, which is concerning. I'm worried that this might eventually lead Splunk to report 100% disk usage, even though there's actually plenty of space available.

I'd like to hide or exclude /proc from being monitored in Disk Usage so that Splunk reflects the actual disk consumption more accurately.

My questions are:

1. If I stop the Splunk service and update the inputs.conf file to exclude /proc, will that prevent it from being counted in the Disk Usage stats?
2. Once the change is made, how long does it typically take for the Splunk GUI to reflect the update?

Looking at doing something like this

/opt/splunk/etc/system/local/inputs.conf

[monitor:///]

blacklist = ^/proc

Appreciate any guidance on this.

[richgalloway · Accepted solution]

Don't do that.  Putting [monitor:///] in inputs.conf will tell Splunk to ingest EVERY FILE on the system (except /proc), which is not desirable.  It also won't solve the problem.

Disk usage data doesn't come from monitor inputs; it comes from the Linux TA.  Go to that add-on and disable or modify the df script.

[PickleRick]

1. Yes, the remark about monitor stanza is 100% spot-on.

2. I suppose we're talking about different spot where disk usage is presented. @richgalloway is talking about what is retrieved from forwarders with a scripted input from Add-on for Unix and Linux but I think @taskmaster is refering to what's presented in Monitoring Console's instance resource screen. The latter is retrieved from the OS by splunkd itself and - to be honest - I don't recall ever seeing anything other than Splunk's filesystems. Maybe that's something specific to Ubuntu? (I don't think I've ever run Splunk on anything else than RH-derived systems)

[taskmaster]

Yes @PickleRick you are correct. Here's an example from a test VM I set up, running Windows, just to illustrate the issue. Under Overview > Resource Usage: Instance > Disk Usage on the Splunk Server, it shows the total and used disk space. In this case, the C: drive is 63.3 GB, with about 31 GB in use, as shown in the screenshot below.

My concern with our Ubuntu build (noted in the original question) is around /proc, which is a virtual filesystem and doesn’t actually take up real disk space. However, Splunk seems to include it under resource usage, which could lead to inaccurate reporting especially if Splunk is interpreting virtual usage as real disk usage.

That’s why I wanted to blacklist /proc: to ensure only actual disk utilization is shown, excluding virtual filesystems that don’t reflect real storage use. I understand now that this is not a valuable solution but wanted to hopefully find something.  I hope this makes sense and thanks everyone for the feedback.

[taskmaster]

Thanks for the feedback, I’ll definitely give that a try