Hello,
The issues that I am having are regarding being able to generate the WA_sessions lookup over a long period of time. In my environment, we currently have 29 sites that have been sending logs to Splunk for a couple years now. I want to be able to query session information from at least year-to-date, but when I try to generate the WA_sessions lookup for any time greater than a month I get the following errors in the search.log:
ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to read 4 bytes from socket within 300000 milliseconds.
ERROR KVStoreLookup - KV Store output failed with code -1 and message '[ "{ \"ErrorMessage\" : \"Failed to read 4 bytes from socket within 300000 milliseconds.\" }" ]'
There are 3 500 000+ events when querying year-to-date. I have tried defragmenting the Splunk server drives and cleaning the kvstores, but keep getting the same errors.
What I would like to know what happens over time to the WA_pages and WA_sessions kvstores, as well as the Web data model? For example, what happens to the WA_sessions kvstore each time one manually generates the lookup, and do old sessions get removed as the automated job runs? If I could split up the generation of sessions into smaller spans and then combine them, this could help me work around the issues I'm having. Do you have any other suggestions for how I could resolve the issue?
Thank you in advance,
Chris
... View more