Problem when updating the KV Store -- some filtered events are lost

[Javip]

Hi all,

We have about 15 Kvstores running ok but sometimes I detect that we had a update problem because we don't have all the filtered events there, we lose some... And we have to reload all the kvstore to fix it and again we have a fresh and updated kvstore version.
But I don't know why we have this problem, really, I can read this kind of error in _internal logs:

2017-09-19T12:17:33.254Z I COMMAND [conn148894] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1505820444000|496 }, $or: [ { ns: { $regex: "^s_." }, $or: [ { op: { $in: [ "i", "u" ] }, o.user: "nobody" }, { op: "d" }, { op: "c", $or: [ { o.create: { $exists: 1 } }, { o.createDatabase: { $exists: 1 } }, { o.drop: { $exists: 1 } }, { o.dropDatabase: { $exists: 1 } } ] } ] }, { op: "c", ns: "admin.$cmd", o.renameCollection: { $exists: 1 }, o.to: { $regex: "^s." } }, { ts: Timestamp 1505820444000|496 } ] } cursorid:27155961562 ntoreturn:0 keyUpdates:0 writeConflicts:0 exception: getMore executor error: UnknownError no details available code:17406 numYields:0 nreturned:1 reslen:103 locks:{ Global: { acquireCount: { r: 4 } }, MMAPV1Journal: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, oplog: { acquireCount: { R: 2 }, acquireWaitCount: { R: 2 }, timeAcquiringMicros: { R: 3536 } } } 1ms

Have you seen that before?
We upgrade Kv stores every 5 minutes and a couple of them are very big (+770K lines)...
Perhaps Splunk/mongodb is still upgrading a KV Store when we try to update it again?

Thanks a lot for your comments
Javier.

[Javip · Accepted solution]

Yes, I was right, it was a problem with our update frecuency. I've increased all the updates to every 15 minutes or so and we don't have errors and we are not losing events anymore.

Thanks!