All Apps and Add-ons

How to migrate KVSTORE to SQL DB?

New Member

currently I have as issue with huge data on KVSTORE, so i'm planning to migrate KVSTORE data which is on one of the search Head to SQL DB.

so please guide me on process flow how to migrate KV store to SQL and output it to Splunk DB Connect add-on using lookup.

please let me know what all stanzas do I need to create and configure.

Help will be appreciated, thankyou

0 Karma

SplunkTrust
SplunkTrust

So, what you are going to want to do is the following.

First, set up DB Connect on your Search Head.

Make sure you have configured DB Connect with an appropriate Java version and the right drivers for your DB.

Create identities for Splunk to use to access your SQL DB, then create the database connection.

SIDEBAR: Spend a few minutes in a regular old search working out how you want the data to come out. It'll likely be some variant of | inputlookup mylookupname followed perhaps by some fields or table commands. Remember this search, or save it off, or copy it into Notepad - something.

The last piece of prep to get into place is build a table in SQL for your data. You will have to look at your source data to figure out field types, sizes and so on. Make sure the user you set up in identities has permission to write to this table.

OK, now that we have all the plumbing in place we can get started.

Open up your DB connect app, go to DB Outputs and start following the output wizard. Most of this is pretty straightforward. You'll want to use the search you created in the sidebar above as your search, make sure to click on each field so they'll all be included in the output. Pick your destination server/schema/table and then assign fields as appropriate. Your execution frequency is, I believe, not important for your exact use case but since it is required, set it to something like 80000 seconds (that way you have nearly a whole day to finish the step below... 🙂 .

So save it and watch it run. Keep your eye on SQL until the data is all across. Once it is, go to your Splunk DB Connect outputs and disable the input. Since you are just moving data out once, you don't need it scheduled.