I've been pushing 10s of millions of rows into KV Store on Splunk 6.3.3 (on Windows). When I do large data inputs we occasionally run into issues with the data not completing. I worked with Splunk support to modify one setting to resolve the issue.
[kvstore]
max_documents_per_batch_save = 500
This has the impact of slowing down the data input, but it completed.
Splunk 6.4.x has improvements to the KV Store and Splunk support reported (but not tested by me). The output I received from support was:
Used search: index=kvcheck1 | outputlookup kvstorecoll
6.3.3 result (failure)
Job inspector
This search did not successfully execute. Any results returned from this job are not consistent and should not be used.
And our favorite entry:
search.log
06-13-2016 10:04:59.730 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to read 4 bytes from socket within 300000 milliseconds.
06-13-2016 10:04:59.748 ERROR KVStoreLookup - KV Store output failed with code -1 and message '[ "{ \"ErrorMessage\" : \"Failed to read 4 bytes from socket within 300000 milliseconds.\" }" ]'
6.4.1 result (success)
Job inspector
This search has completed and has returned 1,999,744 results by scanning 1,999,744 events in 534.54 seconds.
Duration (seconds) Component Invocations Input count Output count
325.37 command.outputlookup 1 8,000,000 8,009,664
As stated the socket timeout problem (this is Windows only issue!) should be fixed in 6.4.x releases.
So for most purposes, up to 70,000,000+ rows, the defaults work well from my experience.
... View more