I'm looking for an efficient way to find events that have not been indexed. Given a sequentially increasing number (requestId) how can you find missing requestIds in the events:
This search gets me close, but does not find when an entire bin of requestIDs are missing:
index=myindex (requestId > 78815600) AND (requestId < 78915600) | sort 0 +requestId num | bin requestId span=10 | stats count by requestId | where count !=10
Ideally I would be looking for a list of missing requestIds - Thoughts?
Give this a try
index=myindex (requestId > 78815600) AND (requestId < 78915600)
| stats count by requestId
| streamstats current=f window=1 values(requestId) as prev
| eval gap=requestId-prev
| where gap>1 AND isnotnull(gap)
| eval missing=mvrange(prev+1,requestId)
| table missing
| mvexpand missing
Give this a try
index=myindex (requestId > 78815600) AND (requestId < 78915600)
| stats count by requestId
| streamstats current=f window=1 values(requestId) as prev
| eval gap=requestId-prev
| where gap>1 AND isnotnull(gap)
| eval missing=mvrange(prev+1,requestId)
| table missing
| mvexpand missing