one reason why you might see this is because the file system maintains a lock on a file if it is in use by a current process. so while you may have issued your file delete command, since splunk was actively using it, the file descriptor may not release the file - splunk will continue ingesting it The option to sinkhole/batch process mentioned is a good way though of deleting the file after ingest ... View more

slight typo on the query - this tstats search is faster than the metadata one if correctly typed: | tstats max(_time) as lastReportedOn where index=A sourcetype=A source=/tmp/A.app.log by index sourcetype source | convert ctime(lastReportedOn) ... View more

one idea for detecting a spike or outlier in logons would be to summarize, by user, how often they login.. say # of times a week or day.. you could try for hour but you'll likely get a lot of noise.. start with something like 'index=<security> sourcetype=<logins> |bin span=1d _time | stats count by <user> _time' Then summary index this data over a recurring period to be able to detect outliers. - try the |collect command Now apply the standard deviation method of detecting outliers - essentially check if the sample is some standard devs from the mean. The MLTK toolkit has some fun features for this, some math behind it is: mean logins by user calculate standard deviation test for : (logins < (mean + 2 standard deviations) ) if true - this is technically an outlier. you can adjust your standard deviation to experiment Then schedule this search to run over a time period and create an alert if there are results.. or to send you a report I would check out the MLTK though, lots of great stuff in there.. Security Essentials has a user case for this as well ... View more

Thanks for this! Helped us out in our very large environment ... View more

It doesn't return all alerts however - alert.track is set to 1 by default but if someone changes it, or is set otherwise by an app, the query above does not return all alerts, alert action or not. This comment thread serves to inform users of the query above to be on the lookout for this scenario - it is not a guarantee that all configured alerts will be returned. ... View more

This isn't necessarily accurate - if for some reason alert.track has not been set, this will not return all results. You can search for this yourself by using the GUI counts vs the results of the searches above. The workaround would be to narrow down the search results in a different way - most configured alerts will have at least one action associate with it so I used something along the lines of |rest/servicesNS/-/search/saved/searches | search actions!=""|<fields go here> ... View more

worked for me - thanks MHibbin ... View more

I can confirm - this is the definitely the issue!! Thanks hayduk - I will escalate this as much as I can -ali from Splunk PS ... View more