I have an issue where we are in the middle of an ES installation, and the professional services created a new index for all of the unix/linux data to go to. This is fine since ES (Enterprse Security) requires things to be organized in a specific way. The issue is now all previous dashboards (and the old *nix) app look only at the old index=os. Here are the solutions that sound like they would work in my head:
1. Combine the indexes (can you just mv or cp the database files i verified they have different names)
2. Go through each search/app and add the new index (time consuming and manual process)
I appreciate any help in advance.
I spoke with the ES professional services people at Splunk yesterday and they suggest against what they call "bucket swapping" ie moving the databases around as it can get messy. So the best fix for this is to addd the Index OR Index to all of you searches and apps. sed is your friend for this one for sure
sed -i 's/index = index1/index = index1 OR index = index2/' /opt/splunk/etc/apps/appname/local/filename.conf
or in vim:
%s/index = index1/index = index1/g
Instead of trying to move data across indexes (it's tricky, at best), I'd instead adjust my search.
In the saved searches, and macros within the Unix app, they make mention of "index=os" or "index="os"" (yes, quoted the value there). I replaced this with a macro of my own
`get_os_index`. This way, I can say set it to "index=linux" or "index=dev_os" or "index=prod_os", etc. Note that you can also set it to something like "(index=os OR index=ps_created_index_for_os_metrics)" and find across both indexes simultaneously!
Admittedly, this approach requires a bit of work, but I've submitted my idea of using a macro (or eventtype would work, too) to allow for flexibility on the location of the OS metrics to the application owners; hopefully the next version will incorporate this idea!
You can move or copy the buckets, and then restart splunk.
Make sure each bucket has a unique ID (the last set of digits in the name of the folder). It is not enough that the folder name is different.
correct when i said i verified the names I meant the buckets. for example:
db_1377812393_1377810573_2 db_1377813498_1377813074_5 db_1377894728_1377889722_8 db_1378754106_1375478578_12 hot_v1_14
db_1377808814_1377806097_0 db_1377812632_1377812478_3 db_1377889267_1377810021_6 db_1377899587_1377894849_9 db_1378838353_1378740173_13
db_1377810600_1377808839_1 db_1377813113_1377812722_4 db_1377890193_1377889566_7 db_1378326596_1377899764_10
You have two sets of data. One in the old index, and one in the new index. If you point the app to an index, then it will not see the other index. I suppose you could try to point the apps to both index=1 OR index=2.
Also, I think you'll may find that the index is specified in more than just the saved searches. For example: The files found in unix/default/data/ui/views.