All Apps and Add-ons

Splunk app Windows Infrastructure - kvstore disk saturations

mabonjean
Explorer

Hi,

I use on my Search Head Cluster (with 80GB of disk space for each SH) the application "Splunk App Windows Infrastructure" that's carrying several kvstores and collections configurations.

Theses KVStore consummed all the disk space. I must disable several scheduled tasks. Mainly all kvstore and lookups updates tasks.

My problem is still here. This app still consummed my disk space and the KVStore doesn't rotate old data.
I won't clean it to prevent losts of valuables datas.

How can I modify / optimize the configuration to stop the high disk consummation ?

Best regards.

nick405060
Motivator

The answer is that you likely have both the Exchange and Windows app installed, which is the issue. To me this is something that should be addressed ASAP, it seems like a very serious issue that any Splunk customer that installs both apps gets their disk space blown up (unsure which versions of Splunk this affects):

https://docs.splunk.com/Documentation/MSExchange/4.0.0/DeployMSX/Platformandhardwarerequirements#Do_...

I ran this per @dwaddle:

|  rest splunk_server=local /services/server/introspection/kvstore/collectionstats 
|  fields data 
|  mvexpand data 
|  rename data as _raw 
|  spath 
|  fields - _raw
|  fields ns size storageSize totalIndexSize

And it showed me that the terminal service trackers are the culprits:

splunk_app_microsoft_exchange.tSessions_collection and splunk_app_windows_infrastructure.tSessions_collection

Per @automine, app savedsearches likely have the same name and are using the same collection. You can disable tSessions_Lookup_Update* savedsearches in the apps, or, like the Documentation link says, just uninstall the Windows Infra app.

This is also the same issue:

https://answers.splunk.com/answers/716097/kvstore-mongo-consuming-40gb-space.html

Oh also a local clean of the kvstore should clear it out

0 Karma

mabonjean
Explorer

Hi Nick,

Thanks for your reply.

We solve (I expect permanently) my issue with cleaning all local KVStore.

The cause is somes old lookups files (old from months ago) can't be delete by Splunk.
When I clean a local KVstore, the non replicated old lokkup files are replicate after the cleaning.

I'll check you answer and make sur that all is ok with my apps.
I'll make a return after.

Thank you.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...