Scenario: We have a data source of interest that we wish to analyze. The data source is hourly host activity events. An endpoint agent installed on a user's host monitors for specific events. The endpoint agent reports theses events to the central server, aka manager/collector. Then the central server sends the data/events to Splunk for ingest. We found that a distinct count of specific action events per hour per host is very interesting to us. If the hourly count per user is greater than "the normal behavior" average then we want to be alerted. We define normal behavior as the "90 day average of distinct hourly counts per host/user". We define an outlier/alert as an hourly distinct count above 2 standard deviations from the 90day hourly average. For instance, if the 90 day hourly average is 2 events for a host, then 10 events in a single hour for that host would fire an alert. We tried many different methods and found some anomalies. One issue is the events' arrival time to Splunk. Specifically, the data does not always arrive to Splunk in a consistent interval. The endpoint agent may be delayed in processing or sending the data to the central server if the network connection is lost or the running host was suspended/shutdown shortly after the events of interest occurred.  We have accepted this issue as its very infrequent. Methodology: In order to conduct our analysis we have multiple phases. Phase 1 > prepare the data and output to KVstore lookup We run a query to prime the historic data.         index=foo earliest=-90d@h latest=-1h@h foo_event=* host=* | timechart span=1h dc(foo_event) as Foo_Count by host limit=0 | untable _time host Foo_Count |outputlookup 90d-Foo_Coun         Then we modify and save the query to append the new data, we use the -2h@h and -1h@h to mitigate lagging events.  This report runs first every hour at minute=0.         index=foo earliest=-2@h latest=-1h@h foo_event=* host=* | timechart span=1h dc(foo_event) as Foo_Count by host limit=0 | untable _time host Foo_Count |outputlookup 90d-Foo_Count append=t          Phase 2 > calculate the upperBound for each user This report runs second every hour at minute=15.  We add additional statistics for investigation purposes.         |inputlookup 90d-Foo_Count |timechart span=1h values(Foo_Count) as Foo_Count by host limit=0 | untable _time host Foo_Count | stats min(Foo_Count) as Mini max(Foo_Count) as Maxi mean(Foo_Count) as Averg stdev(Foo_Count) as sdev median(Foo_Count) as Med mode(Foo_Count) as Mod range(Foo_Count) as Rng by host | eval upperBound=(Averg+sdev*exact(2)) | outputlookup Foo_Count-upperBound         Phase 3 > trim the oldest data to maintain a 90d@h interval This report runs third every hour at minute=30.         |inputlookup 90d-Foo_Count | eval trim_time = relative_time(now(),"-90d@h") | where _time>trim_time | convert ctime(trim_time) |outputlookup 90d-Foo_Count         Phase 4 > detect outliers This alert runs fourth (last) every hour the minute=45.         index=foo earliest=-1h@h latest=@h foo_event=* host=* | stats dc(foo_event) as as Foo_Count by host limit=0 | lookup Foo_Count-upperBound host output upperBound | eval isOutlier=if('Foo_Count' > upperBound, 1, 0)         This method is successful alerting on outliers. RE: event lag, we monitor and keep track of how significant. Originally, we tried using the MLTK with a DensityFunction and partial fit, however we have approximately 65 million data points which causes issues with the Smart Outlier Detection assistant. The question is whether anyone has a different or more efficient way to do this? Thank you for your time!   ... View more

Hi, Let's say I have a Company directory lookup (e.g. Company_Directory) and I want to lookup the entire hierarchy of supervisors for a specific employee. For instance>>> Alice reports to Bob, then take Bob as new lookup criteria... Bob reports to Cathy, etc....   and append this all  in a chain of command >>> Alice, Bob, Cathy, Donna, Eric, Fred.... etc Does Splunk have a command/capability to take the results as feed back to a lookup loop? Thank you ... View more

Hi,  I'm having no luck getting a filter-n-drop setup...  I referenced  https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad Discard specific events and keep the rest   props.conf [source::/opt/fooBar/*] TRANSFORMS-null = setnull transforms.conf [setnull] REGEX = ^(DEBUG) DEST_KEY = queue FORMAT = nullQueue     I am not sure if the REGEX is correct, but "debug" is seen in ERROR events so I only want to capture and drop events where DEBUG is the first word...   Any help appreciated. Thank you! ... View more

Hi - I have a Splunk UF monitoring many directories on a rsyslog (receiver) server. One of the directories populated with logs as expected. However, the input stanza had the incorrect sourcetype and the data/logs did not index. Now after removing the sourcetype, I need to reset the UF to re-monitor the log files in that single directory "only".   I do NOT want to re-index everything the UF monitors. Please advise the best way to handle this... Thank you ... View more

Hi -  Let's say you have a scheduled query / report that runs daily (at mid-night) looking over a time range of Last 24 hours.  And you summarize the results to index=summary_index_foo. There was a "foo" data source outage for a couple days, however you were able to backfill the data to index=foo. What is the best to re-run the query without creating a lot of duplicates.   I am pretty sure if you use "collect" that will create duplicates. But will re-scheduling a one-time clone of the report over the outage days and summarizing results create duplicates if the time range overlaps into the data (before and after the outage)? In other words, the outage time frame was not to the minute, hour, or day exactly.  When you re-schedule/re-summarize the query will that create duplicates if the same data/event exists in the summary index for that time?    Or will Splunk drop duplicates when using the summary index?    I am guessing duplicates will still be created but need a sanity check. Thank you   ... View more

Hi  I am trying to speed up a query. When I run >>>   index=foo | stats values(host) as F_host   It take less than a minute to return the results. I want to take those results and create an outlookup and match host values against another lookup.  However I need to deliminate the stats results to individual values.   Something like,    ... | makemv delim=" " F_host | outputlookup ...   or maybe,    ... | eval D_host = split(F_host, " ") etc   If I run the original query >>>    index=foo | lookup bar-host.csv barHost AS host OUTPUTNEW barHost as match-host | stats values(match-host) by host   it takes forever.   In this case, bar-host.csv is the lookup filename and barHost is the fieldname. Maybe this is just plain old wrong, any advice appreciated... Thank you ... View more