The splunk list monitor command should be run on individual forwarders, not the indexer. The command only lists what is being monitored on the server on which the command is run so running on the indexer will only show you the stuff that Splunk is sending to itself from the indexer to _internal . ... View more

Sounds like you're trying to chart OVER distinct values, use something like this.  | chart max(delay) over foo by bar ... View more

followup - I gave up and wrote a script. ... View more

I setup our primary splunk partition as XFS simply because it allows for file defragging where as ext3 does not. Any kind of file fragmentation will impact performance because of extra seeks. With ext4 offering defrag support as well, perhaps that would be an option worth considering; although it's still a bit new, and apparently the defrag support isn't considered stable yet. Of course, splunk's file usage patterns seem to keep fragmentation fairly low. All the rawdata files can get fragmented, but then get gziped (and thus re-written to disk) which helps keep fragmentation low. Also the *.tsidx files can get fragmented as new content is added to a bucket, but then these files get merged over time, which again triggers the creation of a new file which is much less likely to be fragmented. You can look at some fragmentation of various files on your ext3 files system using the filefrag tool. You have to run this as root, and it only reports one file at a time, so I often run a command like: find /opt/splunk/var/lib/splunk -type f | xargs filefrag ... View more

This is fascinating stuff, but there's a datum missing that would help me know everything for an indexing latency problem I have: _time is timestamp on the event _indextime is time it was indexed at the indexer. need _arrivaltime, the time it arrive at the indexer. Is there such a value? In general where can I find these hidden groovy _* variables listed? Thanks, Dave ... View more

index=_internal sourcetype=splunkd_ui_access | stats count by clientip , user , _time | lookup dnslookup clientip | timechart distinct_count(clienthost) by clienthost span=1d limit=100 This command worked very well ... View more

This is a parameter set by automatic sourcetype learning. There is no value in you setting this on your own. ... View more

If we could, we would, unfortunately we can't aggregate all this data in one place, and that's part of the problem that we're using Splunk to try and solve. On the other note about caching, yes these operations are cacheable (even in v3), but short lived as I understand it, for the reasons you mentioned. Certain things like attribute cache are possible to disable for security. If the read pattern isn't able to benefit from caching much, perhaps there may be some worth in determining which ops are faster without caching effects, and conservatively using those when network filesystems are in use ... View more

this works. thank you ... View more