Getting Data In

kvstore mongo directory is very large

Path Finder

Hi.

I have a issue, we migrate Splunk from 6.6.11 to 7.2.3 in both cluster (SH and Indexer), on indexer we aply migration migration-kvstore, but not on the SH nodes.

The mongo (/home/splunk/splunk/var/lib/splunk/kvstore/mongo) directory have 350 GB ocuppied of the hard disk, and We are critical.
On the log file say (many lines):
2019-02-18T15:17:11.083Z I STORAGE [initandlisten] Found drop-pending namespace s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.system.drop
i2713t-1.c with drop optime { ts: Timestamp(1549620824, 2713), t: -1 }
2019-02-18T15:17:11.083Z I STORAGE [initandlisten] Found drop-pending namespace s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.system.drop

An the directory living this files (and many more):
-rw-------. 1 root root 536608768 feb 17 19:33 s_monitoDjADiK3LuYveVmB44TZEiI13_DATA_GI8XK1TVzglQkuXOSwiJtOFXl.636
-rw-------. 1 root root 536608768 feb 17 20:03 s_monitoDjADiK3LuYveVmB44TZEiI13_DATA_GI8XK1TVzglQkuXOSwiJtOFXl.637
-rw-------. 1 root root 536608768 feb 17 20:33 s_monitoDjADiK3LuYveVmB44TZEiI13_DATA_GI8XK1TVzglQkuXOSwiJtOFXl.638

its possible delete with linux command?

0 Karma

Path Finder

Hi

Just saw this for 7.2.5, fixed issues in release notes:

2019-03-07 SPL-167347, SPL-165968 Frequent searches with outputlookup may trigger highly increased KV Store storage usage or in some cases crash of the mongod process

André

0 Karma

Path Finder

-rw-------. 1 root root 536608768 feb 17 07:12 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.29
-rw-------. 1 root root 536608768 feb 17 07:13 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.27
-rw-------. 1 root root 536608768 feb 17 07:13 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.31
-rw-------. 1 root root 536608768 feb 17 07:13 s_monitoDjADiK3LuYveVmB44TZEiI13_OBJ_GExkG7i403ybeNVt3NN3M3U4J4.30

0 Karma

Path Finder

We also see this. Exact same size. Same splunk version (7.2.3)

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!