Monitoring Splunk

Tested with success, but looking for validation to ensure that this is an appropriate way to move an index to new LUN

paimonsoror
Builder

Hi Folks;

As our network indexes has grown rapidly over time, I am looking to preserve data and splunk performance, while making sure that we have the capacity to store the network data. In doing so, I have requested a second LUN for our network index. I have performed the following steps in my non-Prod environment, and it seems like everything was successful, but I do want to make sure that I didn't miss a step:

  1. Set maintenance mode on the cluster
  2. For each individual indexer
    • Stop indexer
    • edit etc/splunk-launch.conf to add a new 'SPLUNK_NETWORK_DB' variable
    • edit etc/slave-apps/all_indexes/local/indexes.conf to update the network db/thaweddb/colddb reference to use new var
    • mv var/lib/splunk/network/*db /opt/splunk_network_data
    • start indexer
  3. disable maintenance mode
  4. update master index file
  5. deploy master index.conf to cluster to make sure all indexers are in sync
Tags (2)
0 Karma
1 Solution

mbuehler_splunk
Splunk Employee
Splunk Employee

Paimonsoror,

This would work, there are a few things to consider:

First, adding a new "SPLUNK_NETWORK_DB" variable is not needed, and might someday cause issues with maintainability.

I would, following best practice just change the path in the indexes.conf, that way you don't have to edit multiple files to make a "simple" change.

Second, just a word of caution, editing the Slave-apps contents can lead you down a dangerous path, so just be careful.

But yes this will work.

View solution in original post

0 Karma

mbuehler_splunk
Splunk Employee
Splunk Employee

Paimonsoror,

This would work, there are a few things to consider:

First, adding a new "SPLUNK_NETWORK_DB" variable is not needed, and might someday cause issues with maintainability.

I would, following best practice just change the path in the indexes.conf, that way you don't have to edit multiple files to make a "simple" change.

Second, just a word of caution, editing the Slave-apps contents can lead you down a dangerous path, so just be careful.

But yes this will work.

0 Karma

paimonsoror
Builder

Thanks for the quick response. And after thinking about it, I agree that the extra Var isn't needed. Especially because that means now if i stand up a new indexer, i need to remember to add that var to the conf file.

Regarding your second point, would there be a better alternative so that I can make sure that the indexer points to the right place for the network data when i start it back up, but before i push out a new bundle?

0 Karma

mbuehler_splunk
Splunk Employee
Splunk Employee

Painmonsoror,

I don't know that in a clustered environment you have a better option, so I would do that. because Slave-apps takes the highest precedent. So I would do it how you suggest.

Good luck!

paimonsoror
Builder

I appreciate it! Our nonprod testing went well, so crossing my fingers for Prod :D. Thanks again for your help

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...