How to combine two servers in distributed environment?

[wojtek_swiatek]

I have a legacy smallserver.example.com which works as a standalone splunk> server, indexer and search head. It has two indexes index1 and index2.

I also have now a brand new bigserver.example.com which I would like to somehow-move splunk so that:

• bigserver starts to index data in index1
• searches would reach out to smallserver and bigserver to gather all data from both index1

The rationale for the changes are that smallserver is a historical mess, with plenty of indexes I am not interested in anymore, except for index1. bigserver would take over the indexing and searches (but would still need data from index1 on smallserver).

I would instruct the data sources to point to bigserver. For some time the interesting data (on index1) will therefore reside on both servers.

My questions:

• is it enough to deploy splunk on bigserver and somehow connect both (I was looking for the "somehow" in the docs about distributed search but they cover way more complicated cases than mine)
• how can I reach to index1 on smallserver while on bigserver? I looked at how to configure search peers but again it seems overkill for my case.

Thank you!

[martin_mueller · Accepted solution]

To let bigserver search smallserver you indeed set up smallserver as a distributed search peer in bigserver. It's not as complicated as it sounds - just go to Settings -> Distributed Search -> Add new Search Peer on bigserver. Enter smallserver:8089 as URL and authenticate as a smallserver admin to link them up - that auth is only used once.

Do you have any custom sourcetypes, field extractions, lookups, etc. on smallserver that you need on bigserver?
The most critical bits are index-time configurations such as timestamping, event breaking, etc. - if you don't have those on bigserver when it starts indexing data for index1 then you may be in trouble.

[martin_mueller]

index1 existing on both servers isn't going to cause a clash, but rather both instances of index1 to look like one large index when viewed from bigserver. That's intentional, it's how Splunk lets larger customers churn through terabytes of new data per day.

The info you see on bigserver includes search peers. When you search for data you can take a look at the splunk_server field present in every event that tells you which splunk server that event was retrieved from.

[wojtek_swiatek]

Interestingly when I look at the main page on bigserver I see
Events Indexed: 34,301,598
Earliest Event: 5 years ago
I hope this is just a view on smallserverand not that the data were actually indexed by bigserver

[wojtek_swiatek]

Thank you for your reply. I added the new search peer, can do a search on index=index1 and I will check whether the index1 on bigserver does not clash with the existing one on smallserver(hopefully not). I have plenty of customizations on smallserver but I will port the required subset on bigserver before sending live data to it.