Excellent! Runs straight out without any changes. This gives you a table view listing each server, OS, Apps installed, and what Serverclass they belong to. Not sure if it's possible, but it doesn't take into account if any server is 'Blacklist' in an App. THAT would be nice to know too. Already saved it as a Report. Thank you. ... View more

@harsmarvania57 - spot on search. Saweeet! Run this on your Deployment Server. I substituted the last lines 'hostname' and dropped in 'applications' and got a count of servers that have downloaded the Apps. ... View more

No. It didn't take zero time to execute. First/Original run: 6,273,099 events in 142.291 seconds Second run: 6,273,099 events in 56.79 seconds. ... View more

Using the exact same time range: earliest=1/14/2019:00:00:00 latest=01/13/2020:24:00:00 From your note, I would be seeing the effect of the OS caching tsidx files from the onPRem cluster. ... View more

Which it is/was. Identical search run repeatedly produced different execution times. ... View more

Unfortunately, I'm unable to upload an image from my desktop to share. It's ok. With the answers, I understand. ... View more

@martin_mueller This also helps complete my question. Your link to the dashboard caching mechanism will help others as well. You too get an 'Accept', but I can only give out one. But in my heart you deserve one. ... View more

Thanks, @wmyersas That's what I was looking to confirm. Appreciate your guidance and sharing the link for others to read up on. ... View more

I disabled the script in the ../local/inputs.conf file. ... View more

Found it in the inputs.conf [script:///emat/splunk/etc/apps/SplunkDeploymentMonitor/bin/scripted_inputs/dm_backfill_factory.py] disabled = false interval = 15 passAuth = admin Turning to disabled = true ... View more

I am trying to find a way to disable the script but haven't found a way yet. I disabled the dm_backfill.conf - disabled=1 but that failed. 33d22d46-a353-4972-9682-3cf55625e013] dedup = 0 earliest = 1340776800.0 latest = 1341986400.0 maxjobs = 1 namespace = SplunkDeploymentMonitor reverse = 1 saved_search = All indexers - regenerator seed = 1342017493.05 status = 2 totaljobs = None disabled = 1 ....followed by a restart of Splunk, but it is still running. I want to have the Deployment Monitor running, just not this script, so I need to leave this enabled in the app.conf. Nothing to turn it off in the UI. Any other thoughts on turning this script off short of uninstalling and reinstalling? ... View more

...more data has come out after my initial posting. It turns out our Asset Management tool is supplying the hosts for the LOOKUP Table in "short names" while the syslog is being indexed as FQDN AND short names. When the lookup table tries to match a FQDN it fails and produces an "unknown" status. Once I figure out how to strip off FQDN to short names my searches should be successful. ... View more

Taking a view from the other side, I would say yes. You absolutely can install Splunk on NAS and have your Indexers Search heads reside there. The only caution I have is to make sure your storage team has a very "tuned" environment and your servers mounting NFS with the correct options. Case in point. I have just completed an upgrade of (4) separate Splunk environments running SuSE LINUX and all the storage requirements were met by using NetApps Filers, to a consolidated shared environment for 1) Platform; and a second Shared environment for 2) Applications. Again, all based on NetApp 6210 Filers. I confirmed any concern of not handling throughput to the filer by running Bonnie++ testing against the NFS mounts. My results came back like this: Sequential Output: 86.6MB/sec Sequential Input (random): 104.2MB/sec When compared to the measured indexing rate I see on a single Indexing server I believe the NAS NFS can stand on its own. This server by itself sees in excess of 15-20GB/day of data with the following rates: Events/Sec: 1350 Indexing rate(KBps): 370 The server in question is setup with the following: (1) 10GB interface; 48GB Memory; (24) 1600MHz CPU's NFS mount options: nfs rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=60,nfsvers=3 0 0 With this knowledge and the understanding of our future growth rates (which are large), I won't hesitate to continue using Tier I & Tier II NAS as the backend for both hotdb/warmdb (Tier I NAS), and colddb (Tier II NAS). But let me reiterate, don't walk into this blind. Make sure your NAS can handle the load Splunk will put on it when indexing larger amounts of data. Do some extensive testing with Bonnie++ and compare it against what you can get against local or SAN disk. Good luck. ... View more

This finally corrected the issue of not filling in the server host list found in the SoS App while using SoSv2.1. ... View more

Nice feature to add on in order to get more data on the environment. Thanks for the upgrade. ... View more

Ok....we finally have a winner. Upgrading to S.o.S. v2.2 not only fixed my issue on seeing the Indexers, but also allows me to run the "| btool inputs | search "/var/log/splunk". And reports on all (4) additional Indexers. Thanks for your patience and persistance, HEXX. A check goes in the WIN column. I can't vote for my own, so you will have to vote for your answer to my question. ... View more

Correct...and as administrator I still get his Permissions error when I run any | btool command. I tried multiple attempts and quoting in case something misfired. Anything else I can try for this troubleshooting exercise? ... View more

Here is the HOST information: splunk cmd btool inputs list | grep -A20 "///emat/splunk/var/log/splunk" [monitor:///emat/splunk/var/log/splunk] _rcvbuf = 1572864 host = c111xbz - or the same as "abc123" To answer how |btool input failed At the search line I add this: | btool inputs | search "/var/log/splunk" and the error that comes back is this: "Search operation 'btool' is unknown. You might not have permission to run this operation." I tried adding a single quote around |btool input' -- to no avail. I even tried to send just |btool input but that failed too. ... View more

the "abc123" is only a replacement name for the actual server name that was listed....I simply anonymized the entry substituting the real name for "abc123". I also tried to run the | btool inputs | search "/var/log/splunk" on the search line but was unsuccessful, so I substituted for what you appeared to be looking for...any entries from the inputs file that had splunk in it. # splunk cmd btool inputs list | grep splunk [batch:///emat/splunk/var/spool/splunk] [batch:///emat/splunk/var/spool/splunk/...stash_new] [fschange:/emat/splunk/etc] [monitor:///emat/splunk/etc/splunk.version] sourcetype = splunk_version [monitor:///emat/splunk/var/log/splunk] [script:///emat/splunk/etc/apps/sos/bin/lsof_sos.sh] [script:///emat/splunk/etc/apps/sos/bin/ps_sos.sh] [script:///emat/splunk/etc/apps/tagapp/bin/tagapp.pl] [splunktcp] or this: # splunk cmd btool inputs list | grep sos [script:///emat/splunk/etc/apps/sos/bin/lsof_sos.sh] index = sos source = lsof_sos [script:///emat/splunk/etc/apps/sos/bin/ps_sos.sh] index = sos source = ps_sos If neither of these helped I can run some others. ... View more

I only get two columns, one starting with abc123 and the other dns, and nothing else. Should I be seeing a list of the Indexing servers here? Do you need to see a conf file to help explain what might be happening? sos_server count dc server_label server_role abc123 2 2 search-head : abc123 search-head dns 4 2 ... View more