Deployment Architecture

Splunk DBX1 1.2.2 on Splunk 6.1.4 - still indexing old data

jasonheb
Explorer

We have splunk dbx 1.2.2 on a splunk 6.1.4 install
Its been working fine but we had an incident where an incorrect DB Input was created which started indexing 10's of GB of data
We disabled all query but his made no difference. The only thing that stopped the index and license storm was to disable the the DBX app.
When I restart the DBX app it appear to then keep indexing the 10's of GB of data - looks like something is caching this feed - either to the DB or the DBX to indexer
Either way it means I have to keep the dbx stopped until I can work out where this index storm is coming from so I can get the DBX started again safely without blowing out our license.
Any assistance greatfully received to understand this issue

0 Karma

jasonheb
Explorer

Found what looks to be the issue - had a number of kv_xxxxx.dbmonevt files in the /var/lib/spool/dbmon folder
Removed them an all is calm now

0 Karma

jasonheb
Explorer

I should add this is on a search head instance of splunk

0 Karma

jasonheb
Explorer

As a next step - I made sure DBX was stopped (disabled after a restart) then went in and removed all DB connection info from the conf files in local, also renamed the var/log/splunk/persisteantstorage/dbx entried to try and ensure those werent participating.
On restart - more indexing despite no DB connections setup

I then removed the app using splunk remove app dbx
restarted,

installed the app clean
restarted

and still indexing ... so disabled it again

Is there a cached entry for something like the javabridge server which is sending this data through?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...