I have read the documentation about breaker characters, but within our organization there is disagreement about when they actually come into play in the main search. The docs don't say anything about it either way, but some say we must use quotes around sourcetype, for example: index=iis sourcetype="http_err_logs" status=500 ...etc It goes without saying that they're needed within literal search phrases; the text of a specific error message, for example. But do they really also apply to comparisons for standard fields like index or sourcetype? As another example, we have sourcetypes with names like "WinEventLog:Application" and "WinEventLog:System" and some are saying that colon becomes a breaker which leads to a search of the entire raw event data. We also have index names with underscores, and so on. As a result, at this point we're playing it safe and quoting anything that has breaker characters, but is there any documentation that describes where they're actually applied or not? ... View more

We have alerts for high Windows Server CPU usage, and we have automated vulnerability scanners which can trip these alerts, which we'd like to ignore. The existing alert has just one nested search to check CPU usage, and the main search uses that host name to figure out the processes with high CPU, roughly like this:     ```analyze processes on high-cpu hosts``` index=perf source=process cpu>25 ```find high-CPU hosts``` [search index=perf source=cpudata cpu>95 |stats count by host |search count > 2 |fields host ] ```omitted: other macros, formatting, etc.```     In order to ignore the vulnerability scans, we have to check the IIS index for requests from servers that start with a particular name (scanner*) -- this works fine:     ```analyze processes on high-cpu hosts``` index=perf source=process cpu>25 ```disregard hosts taking scanner traffic``` [search index=iis |lookup dnslookup clientip OUTPUT clienthost |eval scanner=(if(like(clienthost,"scanner%"),1,0) |stats sum(scanner) as count by host |search count<10 |fields host ```find high-CPU hosts``` |search [search index=perf source=cpudata cpu>95 |stats count by host |search count > 2 |fields host ]] ```omitted: other macros, formatting, etc.```     The problem is that not all servers use IIS. For example a SQL Server will never appear in the IIS index. So I'm trying to find a way to have the host value passed to the main search when the IIS sub-search doesn't have any hits at all. I kind of suspect I should combine the IIS search into the high-CPU subsearch (reference both indexes) but I'm having a hard time wrapping my head around how that would work. As a side note, performance seems pretty bad. Each subsearch runs subsecond, stand-alone (and our indexes like IIS have 1 million plus events per minute), but the multi-subsearch version takes a little over 1 minute -- this surprises me since only a few host values are passed out of the innermost search. Performance suggestions are very welcome but we can live with the processing time if I can solve this other issue. ... View more

My webhook endpoint needs to retrieve the results of the alert that was triggered. Am I correct in thinking that the payload's "sid" value is the same as the Enterprise REST API's {search_id} value in the search/jobs/{search_id}/results endpoint? I'm a little surprised the webhook docs don't say anything about this since it seems like the logical next step. Normally I'd just try it myself, but we're in a gigantic corporate environment, there's tons of paperwork to get permission to do anything, etc. etc. -- much faster to just ask if I'm on the right track. And, I guess, the other obvious question is, if I'm not on the right track, how do I retrieve the search results based on a webhook payload? Thanks in advance! ... View more

I have two sets of IIS data (two sourcetypes) in a single index. One sourcetype logs web service requests, the other is a once-daily dump of the available services and their base URIs (app pools and apps). I have two working queries as shown below, and I'm trying to figure out how to combine them -- or if that is even possible. (Trouble is, I'm new to Splunk but I've been a programmer for decades -- trying to wrap my head around "the Splunk way"...) Returning servers and services with 25+ errors:     index=iis sourcetype=requests sc_status=503 | rex field=cs_uri_stem "(?i>(?<ServiceURI>^\S*\.svc\/)" | stats count by host,ServiceURI | where count > 25 | eval ErrCount=count,ErrHost=host | table ErrCount,ErrHost,ServiceURI     Results look like this: ErrCount ErrHost ServiceURI 106 server123 /app1/route1/filename1.svc/ 203 server456 /app2/filename2.svc/   Querying the once-daily service data is a bit more tricky -- each server can host multiple services, and each service can expose multiple base URIs which are stored as a comma-delimited list. So retrieving the service data for a specific server looks like this:     index=iis sourcetype=services earliest=-24h host=server456 | makemv delim="," RootURIs | mvexpand RootURIs | table AppName,RootURIs | where RootURIs != ""      Results look like this: AppName RootURIs pool2 /app2 pool2 /alsoapp2 pool2 /stillapp2/with/routes pool3 /app3 pool4 /app4   Both searches produce relatively low row-counts. I've seen the first produce maybe 15 or 20 hits max on a really bad 15 minute period (these will become alerts). The app info for a really large server might produce a maximum of maybe 200 root URIs. Both execute very quickly (a few seconds, tops). So my goals are: 1. feed the server names from the first search (503 errors) into the second search, 2. match the start of the first search's ServiceURIs to specific AppNames and RootURIs from the second search, 3. output the fields listed in the table statements in both searches Is this even remotely possible or realistic? I tried to do this via subsearches, but I think the subsearch has to produce just one result. With my developer and SQL backgrounds, a join was the obvious solution but I couldn't figure out how to get other fields (like the ServiceURI) into the joined search -- I thought I could use $ServiceURI$ but it didn't seem to work. I saw an intriguing comment that the splunk-way is to "collect all the events in one pass and then sort it out in later pipes with eval/stats and friends" but even if that's a good approach, I suspect the volume of data makes this a bad idea (millions of requests logged during a 15 minute window). Help an old code-monkey discover The Splunk Way! ... View more