I dunno.  I have somewhat of the same issue.  A search result shows while its searching and will stay if lower than a certain number of days but then disapears when the search completes over a number of days that is not consistant.  So seems related to the length of time of the search.  My search has no dedup in it.

Should be able to check index=_audit for search run times for each individual search running on dashboards. Something like this. index="_audit" action=search provenance="UI:Dashboard:*" | table _time, user, app, provenance, savedsearch_name, search_id, total_run_time, event_count, result_count, search_et, search_lt | eval dashboard_id=mvindex(split(provenance, ":"), 2, -1)

hi @muradgh i’m having the same issue on my fortigate logs using TCP but we’re using splunk cloud so modifying the props.conf file i think is not a straightforward task for us to do so i’m planning to use UDP instead..  are you able to share with me your syslog-ng.conf for fortigate logging if that’s ok with you? i also need inputs on setting up the correct filters to make the raw output readable and one line per event. did you also set the log format on fortigate firewall to use rfc 5424 when sending to syslog-ng? thank you in advance!

Thanks - that is a lot more detailed than my solution and I like the intersection - that will be useful for me to help people know what was in there - we often have hundreds of keys returned and to see which ones were retuned is really useful.    Thanks,  Steven

And I have managed to solve it.. should have fetched a coffee before posting I guess.  So just needed to add a |Search and IN after the |Split index="PreProduction" source="Transactions"  | eval KeysSplit=split(Keys,", ") | search PKSSplit IN($ObjectRefs$) I can then |table my results.  Hopefully this may be useful to someone else. 

Hi, we encountered the same issue after upgrading Splunk ES to 7.2.0. I am kindly asking to be more detailed by what do you mean by : I removed the stanza from the default folder, (which file in the default folder?) I added a stanza with disabled = 1 in local folder, (again, in which file you added the stanza?) Also, are you referring to this recommendation (Ref: hxxps://docs.splunk.com/Documentation/ES/7.2.0/RN/KnownIssues )? Add the following comment at the end of the file. Conf File Check for Bias Language [confcheck_es_bias_language_cleanup://default] debug = <boolean>

Not sure if this is exactly what you are looking for but I think it is pretty close. I got this output by stringing together a couple of streamstats with window=<int> and reset_before=<criteria> parameters | sort 0 +Machine, +time | streamstats count as row | eval TimeStamp=strftime(time, "%m/%d/%Y %H:%M:%S") | fields - _time | fields + row, Machine, TimeStamp, time | streamstats window=3 count as running_count, min(time) as min_time, max(time) as max_time by Machine | eval seconds_diff='time'-'min_time', duration_diff=tostring(seconds_diff, "duration") | streamstats window=3 reset_before="("seconds_diff>300")" count as running_count by Machine | eval Occurrence=if( 'seconds_diff'<=300 AND 'running_count'==3, "TRUE", "FALSE" ) | fields + row, Machine, TimeStamp, Occurrence   Here is the full SPL I used to generate the screenshot (results may vary because of the use of relative_time()) | makeresults | eval Machine="machine 1", time=relative_time(now(), "-2h@s") | append [ | makeresults | eval Machine="machine 1", time=relative_time(now(), "-2h+18s@s") ] | append [ | makeresults | eval Machine="machine 1", time=relative_time(now(), "-2h+34s@s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d@d+20h@h+31m@m+48s@s") ] | append [ | makeresults | eval Machine="machine 1", time=relative_time(now(), "-2h+52s@s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d+5h+5m+2s") ] | append [ | makeresults | eval Machine="machine 1", time=relative_time(now(), "-2h+302s@s") ] | append [ | makeresults | eval Machine="machine 1", time=relative_time(now(), "+2d-5h+18s@s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d+5h+18s@s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d+5h+2m+1s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d+5h+2m+34s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d+5h+4m-12s") ] | append [ | makeresults | eval Machine="machine 2", time=relative_time(now(), "+4d@d+20h@h+43m@m+5s@s") ] | sort 0 +Machine, +time | streamstats count as row | eval TimeStamp=strftime(time, "%m/%d/%Y %H:%M:%S") | fields - _time | fields + row, Machine, TimeStamp, time | streamstats window=3 count as running_count, min(time) as min_time, max(time) as max_time by Machine | eval seconds_diff='time'-'min_time', duration_diff=tostring(seconds_diff, "duration") | streamstats window=3 reset_before="("seconds_diff>300")" count as running_count by Machine | eval Occurrence=if( 'seconds_diff'<=300 AND 'running_count'==3, "TRUE", "FALSE" ) | fields + row, Machine, TimeStamp, Occurrence

It is not clear why row 5 should be true since you haven't shared the data (number of errors in each event). Having said that, are you trying to implement a sliding 5 minute window, or are you using time bins? If you are using time bins, the row 5 is in a different bin to rows 1-4.

Additionally, you can use one of several apps implementing such source tracking. For example - https://splunkbase.splunk.com/app/4621 On the other hand, you can use Forwarder Monitoring in Monitoring Console to see "lost" forwarders (but this relies on _internal logs from the forwarder, not on the actual "production" events forwarder from given UF)

Hi @maede_yavari, the best approach is having a lookup (called e.g. perimeter.csv) containing the lista of all UFs to monitor (at least one column: host). Then you could run (e.g. every 15 minutes) a search like this: | tstats count WHERE index=_internal BY host | append [ | inputlookup perimeter.csv | eval count=0 | fields host count ] | stats sum(count) AS total BY host | where total=0  If you don't wnt to have this lookup, you could use this search to run every 15 minutes: | tstats count WHERE index=_internal earliest=-30d latest=now BY host _time | eval period=if(_time<now()-900,"Previus","Last") | stats dc(period) AS period_count values(period) AS period BY host | where period_count=1 AND period="Previus" this second solution has the advantage that you don't need to maintain the lookup but gives you less control because you don't check servers that aren't sending logs from 30 days and it's more heavy. Ciao. Giuseppe