Try this: my search | bin span=5min _time | stats avg(Throughput) AS Throughput span=5m by id _time | eval zero_throughput_time=if(Throughput=0, _time, NULL) | eventstats latest(zero_throughput_time) AS latest_zero_throughput_time BY id | eval latest_zero_throughput_time=if(isnull(latest_zero_throughput_time), 0, latest_zero_throughput_time) | where _time>=latest_zero_throughput_time ... View more

You can determine which monitor stanza is responsible for this by examining the output of: splunk list monitor After which you can work to tune your blacklist/whitelist to try to prevent it from being indexed. ... View more

You could consider using a virtual IP address (VIP) managed by a high availability mechanism, such as pacemaker on Linux. I have used this with success on CentOS for syslog receivers. It allowed us to run two syslog servers side-by-side, with only one having the VIP active at a time. This way if one syslog server goes down, the other takes over. The syslog server was also running a heavy forwarder to send events to the indexers. ... View more

Can you paste the XML of your dashboard that is not working? ... View more

Try: | stats dc(UserId) AS count_UserId, values(UserId) AS UserId BY src_ip | search count>20 This should show you src_ip values associated with more than 20 UserId values, and which UserId values they are. ... View more

If you deem any of the posted answers valid and satisfactory, please accept one so this question no longer appears open. ... View more

I think the issue with your regex may be that it doesn't enforce that there are no additional characters after the nine digits. ... View more

You can accomplish this with streamstats. A run anywhere example of this in action is: | makeresults | eval type="delete", _time=5000, commonfield="B" | append [| makeresults | eval type="generate", _time=1005, commonfield="C"] | append [| makeresults | eval type="delete", _time=1000, commonfield="A"] | append [| makeresults | eval type="generate", _time=995, commonfield="A"] | append [| makeresults | eval type="generate", _time=995, commonfield="B"] | eval delete_time=if(type="delete", _time, NULL), generate_time=if(type="generate", _time, NULL) | streamstats current=false last(delete_time) AS last_delete_time BY commonfield | eval time_between_generate_delete=last_delete_time-generate_time | eval is_not_cleared_within_10_mins=if(type="generate" and (isnull(time_between_generate_delete) or time_between_generate_delete>600), "TRUE", NULL) ... View more

That search runs once per hour. If you want it to run every 5 minutes change your cron expression to: */5 * * * * As you have it now it runs only when minute=1 (which will only happen once per hour). ... View more

So the search works outside of the alert, but the alert isn't firing from the scheduled run? ... View more

Edited my answer to reflect grouping by from and subject. ... View more

The posted script won't return anything to splunk. Instead it writes to a file (that you probably don't have Splunk configured to monitor). I see commented out lines that suggest you may have tried this or something similar, but try having your script only echo to stdout instead of redirecting to a file. ... View more

Please edit your question to format your code. Highlight it and click the code button (labeled 101010). ... View more

One possibility is to use transaction , which you can configure with the start/end criteria as well as the field to join on. Here is a run anywhere example of this in action: | makeresults | eval _raw="Jan 5 16:48:25 MyAccount ERROR: com.ebpsource.XYZ123Consumer.Customer0: - E0 00300 Customer0 - ERROR: XYZGetAccountDefault returned 0 records for account=9999999 [email protected] dateFrom=12/17/2017 dateTo=01/05/2018 doLinking=true isUDLAccount=false." | append [| makeresults | eval _raw="05 Jan 2018 16:45:19,951 - INFO [http-bio-8080-exec-409] (Util.java:44) - 9999999: begin getAccountDefault()"] | rex "(?<returned> records for account=(?<account>[^ ]+))" | rex "(?<begin>- (?<account>[^:]+): begin)" | transaction startswith=eval(isnotnull(begin)) endswith=eval(isnotnull(returned)) account This gives you one event per session, and will contain all of the fields from the contributing events. It will also add another field named duration , which is what it sounds like. ... View more

Great! If you consider the answer valid and satisfactory, please accept it so this question no longer appears open. ... View more

Try this instead: SHOULD_LINEMERGE = false LINE_BREAKER = ([\n\r]+)Authentication request received ... View more

Can you paste samples of the logs that this search would make use of? ... View more

Splunk ES depends heavily on the Common Information Model, which grants you some options in terms of what will be included. There are predefined tags that are used to associate events with the datamodel it should belong to. If your sourcetypes don't set this tag , they won't be part of the datamodel. Many of the official TAs set the tags and other fields appropriately to be CIM compliant, so if you have these in place all of the events handled by those apps would already be tagged to be a part of the datamodel. Another option is to use the CIM-provided macros to more finely tune which events should be a part of its datamodels. The macros are named cim_<datamodel>_indexes , but they are general macros and could really contain event search (you can't include | , though). So for instance, the Web datamodel has this macro: [cim_Web_indexes] definition = () If you knew that you only wanted to include the apache sourcetype, you could set it to this: [cim_Web_indexes] definition = (sourcetype=apache) Note that although I mentioned that ES relies heavily on the CIM, nothing prevents ES from using non-datamodel searches, and if you have any correlation searches that don't use the datamodel, you'd have to determine how you want to filter those when you create them. You would also need to take the sourcetypes into consideration when crafting your drilldown searches, which are almost always against raw data, which means the CIM macros wouldn't be in effect. ... View more

I suggest looking at the Splunk Fundamentals 1 course, which is free: https://www.splunk.com/view/SP-CAAAPX9 ... View more

You can use | outputlookup to write search results to a lookup file, which would allow you to create it programmatically. My sample lookup file could have been produced by this search: | makeresults | eval agent="nfc*", limit=700 | append [| makeresults | eval agent="mm_write", limit=5000] | append [| makeresults | eval agent="breeze", limit=1500] | append [| makeresults | eval agent="megafon_bitmap", limit=1000] | append [| makeresults | eval agent="bankm_cashback", limit=5000] | append [| makeresults | eval agent="*", limit=7000] | fields - _time | outputlookup agent_limits.csv But the logic that determines what those limits should be based on your data, I don't know how to help you there. ... View more

I just ran into this issue and this also corrected the behavior. ... View more

Things get trick when trying to use a timepicker for anything other than _time . Does your event actually have two legitimate timestamps (event time and PublishTime), or should PublishTime be the time of the event instead? If the latter, consider working to get that fixed at index time. But to answer the actual question posted, you can do some gross things with searches that set tokens (epoch time for earliest/latest) based on other tokens (timepicker), and use the former in your search later. Here is a run anywhere example of this in action: <form> <label>610251</label> <fieldset submitButton="false"> <input type="time" token="timepicker"> <label></label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> </fieldset> <search id="set_earliest_latest_epoch"> <query>| makeresults | addinfo | eval earliest_tok=info_min_time, latest_search=if(info_max_time="+Infinity", "", "PublishTime<".info_max_time)</query> <earliest>$timepicker.earliest$</earliest> <latest>$timepicker.latest$</latest> <done> <condition match="'job.resultCount' == 1"> <set token="earliest_tok">$result.earliest_tok$</set> <set token="latest_search_tok">$result.latest_search$</set> </condition> </done> </search> <row> <panel> <table> <search> <query>index=_internal | eval PublishTime=relative_time(_time, "-5min") | search PublishTime>=$earliest_tok$ $latest_search_tok$ | stats count</query> <sampleRatio>1</sampleRatio> </search> <option name="count">20</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> <panel> <table> <search> <query>index=_internal | stats count</query> <earliest>$timepicker.earliest$</earliest> <latest>$timepicker.latest$</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">20</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> </row> </form> ... View more

1) did you fill in the host field with the name of the forwarder? And did you run this search from the search head? 1a) did you run splunk list forwarder-server on the forwarder? 2) did you run head /var/log/messages as the user that splunk is running as? ... View more

From the Splunk Cloud docs: Only approved apps and add-ons can be installed in a Splunk Cloud deployment. Approved apps have been examined by Splunk to ensure they comply with the security requirements of Splunk Cloud. You can't install apps from file in Splunk Cloud. There is an approval process required to install custom apps. ... View more