We are trying to use search head pooling (on Solaris 10/x86) with an NFS share but we have performance issues (Splunk Web is pretty slow, searching is very very slow). We see that splunk does a lot of open syscalls with dtrace: dtrace -n 'syscall::open*:entry /execname == "splunkd"/ {@[copyinstr(arg0)] = count(); } tick-1sec { printa(@); clear(@); }' A lot of the opens are because of the csv files in [path_to_shared_storage]/var/run/splunk/dispatch/[job xy] directories. It looks like splunk is causing a open syscall once per second for those files. I tried changeing the poll.interval.rebuild and poll.interval.check settings in the [pooling] section of server.conf . But that did not seem to have an impact. We have over 1000 directories/jobs below /var/run/splunk/dispatch most of the ttls for the jobs are 600 (60%) & 120 (20%) seconds some are 86400 (1 day). Should we somehow limit the amount of jobs (have multiple sets of search head pools)? Thanks Chris Update: I added dispatch_dir_cleanup_freq = 600 in limits.conf as ewoo suggested, this did not have a noticeable impact. Using another dtrace command: dtrace -n 'syscall::open*:entry /execname == "splunkd"/ {@[copyinstr(arg0), ustack()] = count(); } tick-3sec { printa(@); clear(@); }' one can see what function calls lead to the open syscalls: /mnt/splunkshead/var/run/splunk/dispatch/1321015664.61171/metadata.csv libc.so.1`__open+0xa libc.so.1`open+0x10c libc.so.1`_endopen+0xcc libc.so.1`fopen+0x35 splunkd`_ZNK8Pathname5fopenEPKc+0x17 splunkd`_ZN14ScopedFileStarC1ERK8PathnamePKc+0x9 splunkd`_ZN13SearchResults7fromCSVERK8Pathnamemmb+0x118 splunkd`_ZN22DispatchSearchMetadataC1ERK8PathnameRK4UserRK3Strm+0x2a8 splunkd`_ZN23DispatchDirectoryReaper4tickEv+0x5d6f splunkd`_ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval+0xb4 splunkd`_ZN9EventLoop3runEv+0xbc splunkd`_ZN14DispatchReaper4mainEv+0x553 splunkd`_ZN6Thread8callMainEPv+0x62 libc.so.1`_thr_setup+0x5b libc.so.1`_lwp_start /mnt/splunkshead/var/run/splunk/dispatch/1321017866.61929/info.csv libc.so.1`__open+0xa libc.so.1`open+0x10c libc.so.1`_endopen+0xcc libc.so.1`fopen+0x35 splunkd`_ZNK8Pathname5fopenEPKc+0x17 splunkd`_ZN14ScopedFileStarC1ERK8PathnamePKc+0x9 splunkd`_ZN13SearchResults7fromCSVERK8Pathnamemmb+0x118 splunkd`_ZN17SearchResultsInfo7fromCSVERK8Pathnamebb+0x4f splunkd`_ZN14DispatchSearch16getSearchJobInfoERK3StrR17SearchResultsInfo+0x4c splunkd`_ZN23DispatchDirectoryReaper4tickEv+0x1b2 splunkd`_ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval+0xb4 splunkd`_ZN9EventLoop3runEv+0xbc splunkd`_ZN14DispatchReaper4mainEv+0x553 splunkd`_ZN6Thread8callMainEPv+0x62 libc.so.1`_thr_setup+0x5b libc.so.1`_lwp_start /mnt/splunkshead/var/run/splunk/dispatch/1321017866.61929/metadata.csv libc.so.1`__open+0xa libc.so.1`open+0x10c libc.so.1`_endopen+0xcc libc.so.1`fopen+0x35 splunkd`_ZNK8Pathname5fopenEPKc+0x17 splunkd`_ZN14ScopedFileStarC1ERK8PathnamePKc+0x9 splunkd`_ZN13SearchResults7fromCSVERK8Pathnamemmb+0x118 splunkd`_ZN22DispatchSearchMetadataC1ERK8PathnameRK4UserRK3Strm+0x2a8 splunkd`_ZN23DispatchDirectoryReaper4tickEv+0x5d6f splunkd`_ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval+0xb4 splunkd`_ZN9EventLoop3runEv+0xbc splunkd`_ZN14DispatchReaper4mainEv+0x553 splunkd`_ZN6Thread8callMainEPv+0x62 libc.so.1`_thr_setup+0x5b Does this help any of the splunk guys (ewoo) to narrow down the issue? ... View more

Is there an easy way I can list & export all users that have a certain role or that have access to a certain index or a certain capability? It would be nice if I could do something like: | metadata type=users or index=_users role=xy or any other method would be ok ... View more

Has anyone come across this message in an environment where distributed search is used: Unable to distribute to peer named xy at uri https://xy:8089 because peer has status = "Authentication Failed" I sometimes get this error for all the search peers at the same time. I have alerts that send emails, because the authentication fails on all the search peers the search head does not find any results. When I run the search manually in splunkweb later on the search works. ... View more

We have a setup where Universal Forwarders send data to indexers and dedicated search heads search those indexers. There is a lookup in $SPLUNK_HOME/etc/system/lookups which is used in $SPLUNK_HOME/etc/search/local/props.conf $SPLUNK_HOME/etc/search/local/transforms.conf that adds some fields to all events in Splunk (I moved props.conf and transforms.conf from system/local to the search app because I got errors when searching internal indexes on the search heads) props.conf [default] LOOKUP-table = my_lookup IP_NAME AS host transforms.conf [my_lookup] filename = mylist.csv max_matches = 1 When I search, the fields correctly show up in the list of events in the flashtimeline view. But if I try to use one of the custom fields in my search not all of the events show up: index="x" myOwnField=value earliest=07/28/2011:09:50:0 latest=07/28/2011:09:55:0 If I pipe the results through another search command and use my custom fields in the second search I get the results i expect: index="x" earliest=07/28/2011:09:50:0 latest=07/28/2011:09:55:0 | search myOwnField=value Has anyone else run across similar behaviour? ... View more

For Java GC logs, which deliever events in milisecond steps, we would like to build a dashboard with a graph where you can zoom in. Is there a possibility to add zooming functionality to a timechart? It should be similar to the zoom in/out possibilities in the standard search view. ... View more