Getting Data In

How do we add a new indexer and search head to our Splunk instance?

Explorer

We are planning to expand existing Splunk setup.

Present : We have one Splunk indexer (172.16.XX.XX) , we are forwarding data to that indexer and accessing Splunk UI for that server.
Planning : one new indexer in new server (172.16.XX.XX) and one search head in new server (172.16.XX.XX).
Final setup : Two indexers and one search head.

Below are some queries,
• Here, What do we need to install one new indexer and search head ?
• How we can forward data to the new indexer and to the old indexer from all the forwarders ? How we can access Splunk UI for new search head. Any ports or any routes we need to open on new servers ?
• Here, we will forward data to the both new and old indexers , we can access splunk UI for new search head. How we can see the indexing data in Splunk UI ?

1 Solution

SplunkTrust
SplunkTrust

Hi @RAYUDU_NARA,

Please find below answers.

  1. For splunk installation same package is used for Indexer, Search Head, Heavy Forwarders, Cluster Master. You can download splunk from https://www.splunk.com/en_us/download/splunk-enterprise.html . But you need to keep in mind that it will be good to run same version of splunk on both indexers and search head. Here you can see different version compatibility between search head and indexer
  2. Here is good documentation to configure forwarder to send data to multiple indexers. I am assuming that you will not run Indexer Cluster in that case when forwarder will send data to different indexer based on autoLB, data will be distributed among both the indexers and in future if any one indexer will go down then you will lose half of the data so you need to consider this as risk. To avoid this situaltion it will be good to setup Indexer Cluster.
  3. Once you will install splunk on search head by default splunk will start on http so you can access splunk search head UI with URL http://<search head FQDN/IP>:8000
  4. Regarding port opening if you will run both indexer and search head in same VLAN then firewall not come into picture and no port opening is require. But if you will run both indexer and search head in different VLAN then below port opening is require.
  5. Search Head -> Both Indexers (Port 8089)
  6. Search Head -> Both Indexers (Port on which Indexers will receive data) (This is require to forwarder Search Head data to Indexer)
  7. To search data from both indexers on search head , you need to configure search peers (Indexers) in search head. Very good documentation here

To conclude above steps in splunk way, you can read this documentation

I hope this helps.

Thanks,
Harshil

View solution in original post

Contributor

Hi @RAYUDU_NARA,
I don't know if i understood your question correctly.
If you want to change your standalone server role from search head, you can do it via monitoring console of your standalone instance.
1. In Splunk Web, navigate to Monitoring Console > Settings > General Setup.
2. Click Edit on instance that you want to disable as a search head.
3. Click Apply Changes to complete setup.

Kindly go through the below splunk docs it might help you.
https://docs.splunk.com/Documentation/Splunk/7.0.1/DMC/Configureinstandalonemode

0 Karma

Explorer

Hi @nikita_p

Yes, this is really helpful. Thank you.

I have one more doubt, We want to forward data to the two indexers from all the forwarders. Can you please suggest one good way to configure.

We have license for existing setup. How we can share with new indexer ?

0 Karma

Splunk Employee
Splunk Employee

Hey, @RAYUDU_NARA, if @harsmarvania57 answered your question, please remember to "√Accept" the answer to award karma points and to let other Splunkers know it’s a golden answer. We’re hosting a karma point contest, so it’s particularly awesome to up vote on Answers these days. 😄

0 Karma

SplunkTrust
SplunkTrust

Hi @RAYUDU_NARA,

Please find below answers.

  1. For splunk installation same package is used for Indexer, Search Head, Heavy Forwarders, Cluster Master. You can download splunk from https://www.splunk.com/en_us/download/splunk-enterprise.html . But you need to keep in mind that it will be good to run same version of splunk on both indexers and search head. Here you can see different version compatibility between search head and indexer
  2. Here is good documentation to configure forwarder to send data to multiple indexers. I am assuming that you will not run Indexer Cluster in that case when forwarder will send data to different indexer based on autoLB, data will be distributed among both the indexers and in future if any one indexer will go down then you will lose half of the data so you need to consider this as risk. To avoid this situaltion it will be good to setup Indexer Cluster.
  3. Once you will install splunk on search head by default splunk will start on http so you can access splunk search head UI with URL http://<search head FQDN/IP>:8000
  4. Regarding port opening if you will run both indexer and search head in same VLAN then firewall not come into picture and no port opening is require. But if you will run both indexer and search head in different VLAN then below port opening is require.
  5. Search Head -> Both Indexers (Port 8089)
  6. Search Head -> Both Indexers (Port on which Indexers will receive data) (This is require to forwarder Search Head data to Indexer)
  7. To search data from both indexers on search head , you need to configure search peers (Indexers) in search head. Very good documentation here

To conclude above steps in splunk way, you can read this documentation

I hope this helps.

Thanks,
Harshil

View solution in original post

Explorer

Hi Harshil,

It is very clear. Thanks for your help.

Regards,
Rayudu

0 Karma

Explorer

Hi Harshil,

Now we have Indexer and search head on same server. We need to disable search head on old server and make it new server as a search head for both the indexer. How we can do this ?

0 Karma

SplunkTrust
SplunkTrust

Hi Rayudu,

If you want to use new search head with fresh installation and do not want to migrate any reports/alerts , dashboards, field extractions etc. from old search head then you just need to follow last points which I have given in my answer To search data from both indexers on search head , you need to configure search peers (Indexers) in search head. Very good documentation here

If you want to migrate knowledge objects (reports/alerts , dashboards, field extractions etc.) from old search head to new search head then follow last points in my answers and additionally you need to copy
1.) $SPLUNK_HOME/etc/apps/<App name>/local/ and $SPLUNK_HOME/etc/users/<Username>/<App name>/local/ directory from old search head to new search head
2.) $SPLUNK_HOME/etc/apps/<App name>/metadata/local.meta and $SPLUNK_HOME/etc/users/<Username>/<App name>/metadata/local.meta directory from old search head to new search head.

At last restart splunk on new search head.

I hope this helps.

Thanks,
Harshil

0 Karma

Explorer

Hi @harsmarvania57 - Thank you for you help. few more questions again ,

1) For ITSI app, do we have any seperate config to move to new search head.
2) What happens to scheduled searches – do they move to the new search head?
3) What happens to our non-forwarded inputs – eg, REST (lots of these now). Do they stay on their existing node?

Please suggest.

0 Karma

Explorer

Hi Harshil,

I think this is for to make a new server as a search head. How we can disable search head from the existing setup (stand-alone)

0 Karma

SplunkTrust
SplunkTrust

There are no such setting to disable search head on Indexers, however you can remove customised role mapping from authentications.conf and custom roles from authorize.conf.

Also you need to remove knowledge objects (reports/alerts , dashboards, field extractions etc.) which were created by users from that indexer so that it will not run those knowledge objects.

0 Karma

Explorer

Hi @harsmarvania57 - i accepted your answer that is helpful to me. Thank you.

For more clarity few more doubts, Please suggest.

We want to forward data to the two indexers from all the forwarders. Can you please suggest one good way to configure ? (autoLB or any node configuration)

We have license for existing setup. How we can share with new indexer ?

0 Karma

SplunkTrust
SplunkTrust

autoLB is good method to distribute data among Indexers but only drawback in your environment is when one indexer will go down you will lose half of the data.

Another method is to clone data to both the indexers but it will use double license & storage so I'll not prefer that method and stick with autoLB and consider risk as you will lose data when any indexer will be down.

Regarding license to share with another indexer, can you please let us know on which server license is currently installed ?

0 Karma

Explorer

If we configure auLB frequency method. Data will forward like 40sec to one indexer after 40sec it will forward data to the another indexer. If any one indexer is down, it will forward data to the active indexer. is it correct ?

License:

Now we have license in existing server (indexer and search head is there in that server). after upgrade we will use this server as indexer-1 and will move search head to new server, and indexer-2 also in another new server.

0 Karma

SplunkTrust
SplunkTrust

When you configure autoLB at that time if you do not want to lost any data from forwarder to indexer you need to use useACK=true on forwarder side (Ref doc here), when you use useACK=true forwarder will wait until Indexer will give acknowledgement back to forwarder that Indexer got data if forwarder will not receive acknowledgement from Indexer in that case forwarder will send same data to another indexer so you will not lose any data during transmission.

Re license: I'll suggest you to move license on Search Head and make search head as license master and both Indexer as license slave. Ref. doc https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Swapthelicensemaster and http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Configurealicenseslave

0 Karma

Explorer

Thank you @harsmarvania75 , will test it and get back to you.

0 Karma

SplunkTrust
SplunkTrust

Hi Rayudu,

It will be good to accept my answer so that this question will be closed and other people will able to refer this answer in future.

Thanks,
Harshil

0 Karma