I have a bunch of events coming in the format of the below example. They are random in the time it takes from start to end.
I want to query a term in the 'processing' event and have it return the entire session. I have been doing so by doing a subsearch for the term and getting the session id, searching for the session ID and transactioning the result. This is very CPU intensive, slow, and causes my subsearches to time out so I can only search a very limited time frame.
I have 2 thoughts on how to solve this issue:
- Correlate at index time - I am not entirely sure how to do this since the events are streamed in with no predefined start and end, and it makes me very wary of data loss.
- Correlate in a summary index - I could run the transaction command every hour to populate a summary index. However, if a session isn't complete, I assume I would lose that data in my query. If I overlap than that would lead to duplication. I am not sure if there is some way to use the overlap command to help with this?
Event 1: session_1 Start
Event 2: session_1 Processing
Event 3: session_1 Finish
Event 4: session_2 Start
Event 5: session_2 Processing
Event 6: session_2 Processing
Event 7: session_2 Finish
Any help trying to figure this out would be much appreciated.
If you can live with a bit of delay and have an upper bound for the length of a transaction you could do the following, assuming the upper bound is one hour:
That way you get the best of all worlds: No duplicates, no missed transactions, a fast summary to search through, and all that at the cost of a bit of delay - as the worst case, a transaction needs two hours to make it into the summary index.
If you can live with a bit of delay and have an upper bound for the length of a transaction you could do the following, assuming the upper bound is one hour:
That way you get the best of all worlds: No duplicates, no missed transactions, a fast summary to search through, and all that at the cost of a bit of delay - as the worst case, a transaction needs two hours to make it into the summary index.