Using "search" It appears when you just use "search" then whatever first data is in the search field is used as the actual search term. Example: index=blah [search index=blah host=blahadmin | rename EventCode as search] will result in a search like such: litsearch index=blah 538 | fields keepcolorder=t * "*" "host" "index" "source" "sourcetype" "splunk_server" Notice the "538" which is the first result returned in the EventCode field in the subsearch. You can also use "search" to modify the actual search string that gets passed to the outer search. For Example: index=blah [search index=blah host=blahadmin| eval search="hello there " + EventCode] Will result in: litsearch index=blah hello there 538 | fields keepcolorder=t * "*" "host" "index" "source" "sourcetype" "splunk_server" Using "query" "Query" will behave similar to format, but instead of putting out (field1=foo) OR (field1=bar) will just put out the values (foo) OR (bar) . Example: index=blah [search index=blah host=blahadmin | rename EventCode as query | fields query] will result in a search like this: litsearch index=blah ( ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 19 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 538 ) OR ( 540 ) OR ( 2000 ) OR ( 2000 ) ) | fields keepcolorder=t * "*" "host" "index" "source" "sourcetype" "splunk_server" Of course you can dedup EventCode to come up with litsearch index=blah ( ( 538 ) OR ( 540 ) OR ( 10002 ) OR ( 19 ) OR ( 2000 ) OR ( 560 ) OR ( 576 ) OR ( 552 ) ) | fields keepcolorder=t * "*" "host" "index" "source" "sourcetype" "splunk_server" Normal syntax And finally the using the regular syntax with the implied format at the end of the subsearch: index=blah [search index=blah host=blahadmin| fields EventCode] Results: litsearch index=blah ( ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="10002" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) OR ( EventCode="540" ) OR ( EventCode="538" ) ) | fields keepcolorder=t * "*" "host" "index" "source" "sourcetype" "splunk_server" Again a dedup should be used here. Conclusion There are definitely many use cases for each of these methods. " Search" can come in real handy when you need to append some static data or do some evals on the data in your subsearch and then pass it to the outer search. You could build some complex logic here. "Query" comes in handy when you are simply looking for the values in the fields returned from the subsearch, but not in these exact fields. And regular syntax comes in handy when you are looking for the returned values in the returned fields.- On top of all this you can use the format command to apply all kinds of fun formatting to your subsearch results. :)- Note: After running a search you can click the Actions menu and select "Inspect Search", scroll down to the remoteSearch field, and you can see what the actual query that ended up being built by your subsearch was. ... View more

You can use the eval command: your search | eval percent_difference=(difference/max(list(Select))*100) Then set up a custom alert condition that hits when percent_difference > 5. If this doesn't work try renaming your list(Select) to a more friendly name (without parenthesis). [Edit:] Ok, playing around with your query and my data I noticed that the list(Select) didn't return a numerical value list for me, which caused the calculation of percent_difference to fail. Try adding a tonumber() (added in 4.1.4 or 4.1.5, so you need a recent version for this to work) into your query as such: sourcetype="datapath-device" earliest=-2h| rex "^DEV#:\s+(?<dev_no>\d+)\s+DEVICE NAME:\s+(?<device_name>\S+)\s+TYPE:\s+(?<type>\d+)\s+POLICY:\s+(?<policy>\S+)" | rex "SERIAL:\s+(?<serial>\S+)" | multikv | stats list(Select) as listSelect, list(Disk), range(Select) as difference by device_name, host | eval percent_difference=((difference/max(tonumber(listSelect)))*100) If that doesn't work, use eventstats instead of regular stats as such: sourcetype="datapath-device" earliest=-2h| rex "^DEV#:\s+(?<dev_no>\d+)\s+DEVICE NAME:\s+(?<device_name>\S+)\s+TYPE:\s+(?<type>\d+)\s+POLICY:\s+(?<policy>\S+)" | rex "SERIAL:\s+(?<serial>\S+)" | multikv | eventstats list(Select) as listSelect, list(Disk), range(Select) as difference by device_name, host | eval percent_difference=((difference/max(tonumber(listSelect)))*100) [Edit2] What about this: sourcetype="datapath-device" earliest=-2h| rex "^DEV#:\s+(?<dev_no>\d+)\s+DEVICE NAME:\s+(?<device_name>\S+)\s+TYPE:\s+(?<type>\d+)\s+POLICY:\s+(?<policy>\S+)" | rex "SERIAL:\s+(?<serial>\S+)" | multikv | stats list(Select) as listSelect, list(Disk), range(Select) as difference by device_name, host | eventstats max(listSelect) as maxSelect | eval percent_difference=((difference/maxSelect)*100) [EDIT 3:] how about streamstats? sourcetype="datapath-device" earliest=-2h| rex "^DEV#:\s+(?<dev_no>\d+)\s+DEVICE NAME:\s+(?<device_name>\S+)\s+TYPE:\s+(?<type>\d+)\s+POLICY:\s+(?<policy>\S+)" | rex "SERIAL:\s+(?<serial>\S+)" | multikv | stats list(Select) as listSelect, list(Disk), range(Select) as difference by device_name, host | streamstats max(listSelect) as maxSelect window=1 | eval percent_difference=((difference/maxSelect)*100) ... View more

There are two chart types that will display bars: Column: This chart type will show veritcal bars (columns) Bar: This chart type will show horizontal bars.- ... View more

You can do sourcetype="bcoat_proxysg" |top limit=100 Domain by Client_IP, Client_User More info on top: http://www.splunk.com/base/Documentation/latest/SearchReference/Top ... View more

I don't see anything wrong with that approach per se. If you are putting everything in the main index then your app will be able to access that data. If you do not want your new app to index the same data, just don't add any inputs to it. In regards to index segmentation, the best practice really does depend on your environment and requirements. Separating indexes will provide you with the means of easily restricting people to certain data sets and specifying different retention periods per index. For example you may want to keep your web logs for five years, your top stats for three months, and your audit logs for one year online and then archive them to disk. Additionally, you can sign events or blocks of events in indexes in order to detect tampering. By setting up different retention periods and cold to frozen actions you can more efficiently use your storage. I recommend the About Managing Indexes chapter in the docs for further reading. ... View more

This regex will work: (?<=\[\d+-\d+-\d+\s\d+:\d+:\d+,\d+\]\[\w+\].*\[\S+\]\s)(\w+\(\)) [Edit: rex usage] To use this in rex you can simply change the lookbehind to a non-capturing group and give a field name (in this case "function") to your capturing group: search terms | rex "(?:\[\d+-\d+-\d+\s\d+:\d+:\d+,\d+\]\[\w+\].*\[\S+\]\s)(?<function>\w+\(\))" ... View more

I believe that unless you specify earliest and latest , subsearches are run over the time range chosen in the time picker, but will only run for as long as the maxtime setting in limits.conf specifies. Section of limits.conf: [subsearch] maxout = <integer> * Maximum number of results to return from a subsearch. * Defaults to 100. maxtime = <integer> * Maximum number of seconds to run a subsearch before finalizing * Defaults to 60. ttl = <integer> * Time to cache a given subsearch's results. * Defaults to 300. ... View more

The following gives you a total for all adhoc searches run in the past 24 hours: index=_internal sourcetype="searches" earliest=-24h | stats count And this one will give you a total for all searches, including saved searches run in the past 24 hours: index=_internal (sourcetype="searches" OR SavedSplunker) earliest=-24h | stats count And individual counts by user: index=_internal (sourcetype="searches" OR SavedSplunker) earliest=-24h | stats count by user ... View more