topic Re: kvstore mongo directory is very large in Getting Data In

kvstore mongo directory is very large

aecruzp — Tue, 29 Sep 2020 23:22:11 GMT

Hi.

I have a issue, we migrate Splunk from 6.6.11 to 7.2.3 in both cluster (SH and Indexer), on indexer we aply migration migration-kvstore, but not on the SH nodes.

The mongo (/home/splunk/splunk/var/lib/splunk/kvstore/mongo) directory have 350 GB ocuppied of the hard disk, and We are critical.
On the log file say (many lines):
2019-02-18T15:17:11.083Z I STORAGE [initandlisten] Found drop-pending namespace s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.system.drop
i2713t-1.c with drop optime { ts: Timestamp(1549620824, 2713), t: -1 }
2019-02-18T15:17:11.083Z I STORAGE [initandlisten] Found drop-pending namespace s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.system.drop

An the directory living this files (and many more):
-rw-------. 1 root root 536608768 feb 17 19:33 s_monitoDjADiK3LuYveVmB44TZEiI13_DATA_GI8XK1TVzglQkuXOSwiJtOFXl.636
-rw-------. 1 root root 536608768 feb 17 20:03 s_monitoDjADiK3LuYveVmB44TZEiI13_DATA_GI8XK1TVzglQkuXOSwiJtOFXl.637
-rw-------. 1 root root 536608768 feb 17 20:33 s_monitoDjADiK3LuYveVmB44TZEiI13_DATA_GI8XK1TVzglQkuXOSwiJtOFXl.638

its possible delete with linux command?

Re: kvstore mongo directory is very large

aecruzp — Tue, 29 Sep 2020 23:22:14 GMT

-rw-------. 1 root root 536608768 feb 17 07:12 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.29
-rw-------. 1 root root 536608768 feb 17 07:13 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.27
-rw-------. 1 root root 536608768 feb 17 07:13 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.31
-rw-------. 1 root root 536608768 feb 17 07:13 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.30

Re: kvstore mongo directory is very large

agneticdk — Mon, 25 Mar 2019 07:36:30 GMT

We also see this. Exact same size. Same splunk version (7.2.3)

Re: kvstore mongo directory is very large

agneticdk — Mon, 25 Mar 2019 07:38:33 GMT

Just saw this for 7.2.5, fixed issues in release notes:

2019-03-07 SPL-167347, SPL-165968 Frequent searches with outputlookup may trigger highly increased KV Store storage usage or in some cases crash of the mongod process

André