I'd like to have some opinions on the following search. We're not thrilled with it's performance, and I'm sure theres a lot of potential to improve it. What is the search supposed to do: This is a kind of "service check", on each server we're checking CPU, memory and the disk space. The data comes from different indexes, needs different result calculation and so on. So our approach was to use append quite heavily, basically constructing different searches with the same result fields, that we could combine and put into the same table. The search is limited to look only for data within the last 5 minutes. It is used within an app that uses the Javascript framework, all $parameters$ defaulting to "*". The search inspector says "This search has completed and has returned 19 results by scanning 25,204 events in 85.338 seconds." The components taking the most time are: 1. dispatch.evaluate.append 2. command.transaction 3. command.search 4. dispatch.stream.remote 5. dispatch.fetch Any general remarks or hints on inspection and improvement strategies for searches are highly appreciated. Thanks! There it is (I cut a lot of evals): search host="$filterHost$" host_category="build-servers" host_group="$filterHostGroup$" index="perfmon" (object="LogicalDisk" (instance = "C:" OR instance ="D:") (counter="% Free Space" OR counter="Free Megabytes")) OR (object="Processor" counter="% Processor Time" instance="_Total") | 2x eval .. | eval transactionKey="" + host + ":" + instance + ":" + _time | transaction maxevents=4 transactionKey | eval dedupKey=host + ":" + instance | dedup dedupKey sortby -_time | 12x eval | append [ search host="$filterHost$" host_category="build-servers" host_group="$filterHostGroup$" index=perfmon sourcetype="base_pagefile" | dedup host sortby -_time | rex field=_raw max_match=0 "AllocatedBaseSize=(?<alloc>.*)" | rex max_match=0 field=_raw "CurrentUsage=(?<usage>.*)" | eval tmpCounter=mvcount(alloc) | stats max(tmpCounter) sum(alloc) as Total, sum(usage) as Used by _time, host | 9x eval ] | append [ search index="os" host="$filterHost$" host_category="build-servers" host_group="$filterHostGroup$" (source=df (MountedOn=/ OR MountedOn=/tmp OR MountedOn=/data OR MountedOn=/net/x OR MountedOn=/mnt/y OR MountedOn=/mnt/z OR MountedOn=/mnt/w)) OR (source=cpu CPU="all") OR (sourcetype="vmstat") | eval 2x | dedup dedupKey sortby -_time | 14x eval ] | append [ search index="perfmon" sourcetype="base_systeminfo" host="$filterHost$" host_category="build-servers" host_group="$filterHostGroup$" | dedup host sortby -_time | eval 9x ] | search $filterStatus$ service="$filterService$" | eval statusNumber=case(status=="CRITICAL", 0, status=="WARNING", 1, status=="OK", 2) | sort statusNumber -host, -service | eval "Last Check"=strftime(_time, "%F %T") | rename host as "Host" service as "Service" status as "Status" value AS "Description" | table Host Service Status "Description" "Last Check" ... View more

I want to create an app that will be deployed on a few forwarders to monitor log files. Problem is with the paths of these log files. They contain the host name (e.g. /path/<hostname>/logfile.log ). I don't want to create an individual app for each host and I can not use wildcards, because defining /path/*/logfile.log in the inputs.conf would bring up a lot of unwanted data - unfortunately, other server's log directories are mounted too. I know that in a configuration stanza I can use $SPLUNK_HOME, so my question is: Is there a way to use variables in the configuration, like [monitor:///path/$HOSTNAME/logfile.log] ... View more

In an inputs.conf I can define a forwarder's host field which I can use in searches. Identifying a single host is not always enough, e.g., we have build servers, source code management servers, filers etc. to monitor - a kind of grouping would come in handy in order to limit searches to a certain group of hosts. We could do that by maintaining lists and using them as look-ups, but I was wondering whether there's the possibility to achieve that on the forwarder with just configuration. Example: inputs.conf on host A: host=hosta group=build_servers inputs.conf on host B: host=hostb group=git_servers I want to be able to search for something like search host=* group="build_servers" sourcetype="df" ... | ... Is there a way to do this? ... View more

We're not sure whether it's safe to use the deployment server feature for all our apps, especially those with lookup files. We want to deploy an app that has several lookup files as csv in its $appname/lookups/ directory. These apps will be rolled out to several search heads. On the search heads the csv files will be updated with data regularly. Will there be a redeployment once the search head phones home to the deployment server, thus overwriting the csv? When and how is the check sum of the deployed app computed? Thanks! ... View more