Knowledge Management

Problem when updating the KV Store -- some filtered events are lost

Javip
Path Finder

Hi all,

We have about 15 Kvstores running ok but sometimes I detect that we had a update problem because we don't have all the filtered events there, we lose some... And we have to reload all the kvstore to fix it and again we have a fresh and updated kvstore version.
But I don't know why we have this problem, really, I can read this kind of error in _internal logs:

2017-09-19T12:17:33.254Z I COMMAND [conn148894] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1505820444000|496 }, $or: [ { ns: { $regex: "^s_." }, $or: [ { op: { $in: [ "i", "u" ] }, o.user: "nobody" }, { op: "d" }, { op: "c", $or: [ { o.create: { $exists: 1 } }, { o.createDatabase: { $exists: 1 } }, { o.drop: { $exists: 1 } }, { o.dropDatabase: { $exists: 1 } } ] } ] }, { op: "c", ns: "admin.$cmd", o.renameCollection: { $exists: 1 }, o.to: { $regex: "^s." } }, { ts: Timestamp 1505820444000|496 } ] } cursorid:27155961562 ntoreturn:0 keyUpdates:0 writeConflicts:0 exception: getMore executor error: UnknownError no details available code:17406 numYields:0 nreturned:1 reslen:103 locks:{ Global: { acquireCount: { r: 4 } }, MMAPV1Journal: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, oplog: { acquireCount: { R: 2 }, acquireWaitCount: { R: 2 }, timeAcquiringMicros: { R: 3536 } } } 1ms

Have you seen that before?
We upgrade Kv stores every 5 minutes and a couple of them are very big (+770K lines)...
Perhaps Splunk/mongodb is still upgrading a KV Store when we try to update it again?

Thanks a lot for your comments
Javier.

0 Karma
1 Solution

Javip
Path Finder

Yes, I was right, it was a problem with our update frecuency. I've increased all the updates to every 15 minutes or so and we don't have errors and we are not losing events anymore.

Thanks!

View solution in original post

0 Karma

Javip
Path Finder

Yes, I was right, it was a problem with our update frecuency. I've increased all the updates to every 15 minutes or so and we don't have errors and we are not losing events anymore.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...