Knowledge Management

Problem when updating the KV Store -- some filtered events are lost

Javip
Path Finder

Hi all,

We have about 15 Kvstores running ok but sometimes I detect that we had a update problem because we don't have all the filtered events there, we lose some... And we have to reload all the kvstore to fix it and again we have a fresh and updated kvstore version.
But I don't know why we have this problem, really, I can read this kind of error in _internal logs:

2017-09-19T12:17:33.254Z I COMMAND [conn148894] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1505820444000|496 }, $or: [ { ns: { $regex: "^s_." }, $or: [ { op: { $in: [ "i", "u" ] }, o.user: "nobody" }, { op: "d" }, { op: "c", $or: [ { o.create: { $exists: 1 } }, { o.createDatabase: { $exists: 1 } }, { o.drop: { $exists: 1 } }, { o.dropDatabase: { $exists: 1 } } ] } ] }, { op: "c", ns: "admin.$cmd", o.renameCollection: { $exists: 1 }, o.to: { $regex: "^s." } }, { ts: Timestamp 1505820444000|496 } ] } cursorid:27155961562 ntoreturn:0 keyUpdates:0 writeConflicts:0 exception: getMore executor error: UnknownError no details available code:17406 numYields:0 nreturned:1 reslen:103 locks:{ Global: { acquireCount: { r: 4 } }, MMAPV1Journal: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, oplog: { acquireCount: { R: 2 }, acquireWaitCount: { R: 2 }, timeAcquiringMicros: { R: 3536 } } } 1ms

Have you seen that before?
We upgrade Kv stores every 5 minutes and a couple of them are very big (+770K lines)...
Perhaps Splunk/mongodb is still upgrading a KV Store when we try to update it again?

Thanks a lot for your comments
Javier.

0 Karma
1 Solution

Javip
Path Finder

Yes, I was right, it was a problem with our update frecuency. I've increased all the updates to every 15 minutes or so and we don't have errors and we are not losing events anymore.

Thanks!

View solution in original post

0 Karma

Javip
Path Finder

Yes, I was right, it was a problem with our update frecuency. I've increased all the updates to every 15 minutes or so and we don't have errors and we are not losing events anymore.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...