https://community.splunk.com/t5/Splunk-Search/How-to-define-a-search-to-find-missing-data/m-p/283651#M85729 <P>Have you looked at <CODE>accelerated datamodel</CODE>? From you description, is appears, you have the right query to get you the desired results, what you looking for is a faster solution. </P> <P><A href="http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Acceleratedatamodels">http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Acceleratedatamodels</A></P> Sat, 17 Dec 2016 01:06:15 GMT sundareshr 2016-12-17T01:06:15Z https://community.splunk.com/t5/Splunk-Search/How-to-define-a-search-to-find-missing-data/m-p/283650#M85728 <P>I have an application that has predictable log entries when it starts a series of activities and when it finishes. I can create transactions, etc. - all good. What I'm struggling with, however, is how to construct a search that tells me which activities didn't complete. Basically - identifying that a set of activities was started, but didn't result in the log entries that indicate it finished. I've done some searches like </P> <PRE><CODE>index="cloudwatch" | regex "\w{8}\-\w{4}\-\w{4}\-\w{4}\-\w{12}/\d{6}/\d+/\w{8}\-\w{4}\-\w{4}\-\w{4}\-\w{12}" | rex "(?\w{8}\-\w{4}\-\w{4}\-\w{4}\-\w{12})/(?\d{6})/(?\d+)/(?\w{8}\-\w{4}\-\w{4}\-\w{4}\-\w{12})" | stats count by admin,ticket,token,chunk_id | where count&lt;7 </CODE></PRE> <P>the regex is basically identifying all the events that have the string that can be used to identify participation in the same transaction - then the rex extracts the individual parts that are meaningful. I can run this, but over a set of tens of millions of events, it's not the fastest in the world. Even setting this up as a scheduled search, I'll end up with phantom records because not all events are within the timeframe being searched - you'll get some orphans at the edges. I can also search for just the beginning / end, adding something like</P> <PRE><CODE>(("received event" manifest.json) OR "writing postdata") </CODE></PRE> <P>to the base search, that can speed up the search, but only by a little - and it'll get worse as the data set grows. Ultimately, I want to define a search that finds 'chunk_id's that didn't complete, schedule it, and get an alert. The reason being, there's usually some corrective action needed and it can be time-sensitive.</P> <P>I feel like I've struggled with this notion of "finding the data that isn't there" numerous times in the past, never quite getting something that seemed "right" - so, finally posting something up here in case anyone has some pointers.</P> <P>Thanks!</P> Fri, 16 Dec 2016 22:31:08 GMT https://community.splunk.com/t5/Splunk-Search/How-to-define-a-search-to-find-missing-data/m-p/283650#M85728 bdruth 2016-12-16T22:31:08Z https://community.splunk.com/t5/Splunk-Search/How-to-define-a-search-to-find-missing-data/m-p/283651#M85729 <P>Have you looked at <CODE>accelerated datamodel</CODE>? From you description, is appears, you have the right query to get you the desired results, what you looking for is a faster solution. </P> <P><A href="http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Acceleratedatamodels">http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Acceleratedatamodels</A></P> Sat, 17 Dec 2016 01:06:15 GMT https://community.splunk.com/t5/Splunk-Search/How-to-define-a-search-to-find-missing-data/m-p/283651#M85729 sundareshr 2016-12-17T01:06:15Z