Deployment Architecture

Splunk DBX1 1.2.2 on Splunk 6.1.4 - still indexing old data

jasonheb
Explorer

We have splunk dbx 1.2.2 on a splunk 6.1.4 install
Its been working fine but we had an incident where an incorrect DB Input was created which started indexing 10's of GB of data
We disabled all query but his made no difference. The only thing that stopped the index and license storm was to disable the the DBX app.
When I restart the DBX app it appear to then keep indexing the 10's of GB of data - looks like something is caching this feed - either to the DB or the DBX to indexer
Either way it means I have to keep the dbx stopped until I can work out where this index storm is coming from so I can get the DBX started again safely without blowing out our license.
Any assistance greatfully received to understand this issue

0 Karma

jasonheb
Explorer

Found what looks to be the issue - had a number of kv_xxxxx.dbmonevt files in the /var/lib/spool/dbmon folder
Removed them an all is calm now

0 Karma

jasonheb
Explorer

I should add this is on a search head instance of splunk

0 Karma

jasonheb
Explorer

As a next step - I made sure DBX was stopped (disabled after a restart) then went in and removed all DB connection info from the conf files in local, also renamed the var/log/splunk/persisteantstorage/dbx entried to try and ensure those werent participating.
On restart - more indexing despite no DB connections setup

I then removed the app using splunk remove app dbx
restarted,

installed the app clean
restarted

and still indexing ... so disabled it again

Is there a cached entry for something like the javabridge server which is sending this data through?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...