Deployment Architecture

Distributed Search concepts are confusing me - what am I missing?

Path Finder

I have set up the following:

1 - Splunk Index/Search machine running v4.3.3 running on Ubuntu Linux Server 12.04.

  It acts as Deployment server which is functional.
  Apps installed and configured:
       Sideview Utilities app
       Splunk for Windows App
       Google Maps app
       MAXMIND app
       Cisco Security Suite app
       Cisco Firewall app
       Deployment Monitor
       Splunk on Splunk

1 - Splunk Universal Forwarder running on Windows 2008 R2
Kiwi Syslog Server
Windows Event Collector

All seems to be functional. I have deployed app updates to the forwarder using the Deployment Server and I know it is working well.

I can use the various apps successfully at this point on the standalone machine.

Now.... to the issues... and the more I read in the documentation and the forums, the MORE confused I get.
I have a small environment right now but will be ramping this up into a much larger scale production environment so I want to do it right to begin with, and UNDERSTAND what I am doing 🙂

1. I want to add a dedicated search head to the mix.

I have attempted this and have run into some problems. Let me review MY understanding of the distributed search environment. Note, I am using only a single dedicated search head so I don't need to worry yet about the pooling configurations. I am also beginning with a single indexer. I know if I can get an understanding of the this, then the rest should add on relatively easily down the road.

  1. I removed all of the apps from the indexer, thinking that I would reinstall them on the search head and they would distribute to the indexer (peer) as needed.
  2. I set up my Search head as a Windows 2008 R2 box (this was to take advantage of the Windows Apps).
  3. I set up the distributed search peer on the search head to point at my indexer. It sees it, and talks to it fine.

  4. I created a Windows Share on my search head at the C:\Program Files\Splunk\etc folder, and created a user that can see it (read AND write).

  5. I added the required SMB configs to Ubuntu, and successfully mounted the Windows share as /mnt/searchhead/ on the Ubuntu box. I can go to the terminal of the indexer and navigate to the /mnt/searchhead folder and see all of the contents.

  6. I then added my apps to the search head thinking they would propagate down to the indexer. (I have only added Sideview, SOS, and Windows + the SOS TA) They do distribute, somewhat. I can see the knowledge bundles in the /var/run/searchpeer folder on the indexer.

My /opt/splunk/etc/system/local/distsearch.conf file on the search head looks like this:

[searchhead:XXX-SPLUNKHD1]   (name sterilized)
mounted_bundles = true
bundles_location = /mnt/splunkhead

My /opt/splunk/etc/system/local/distsearch.conf file on the indexer (peer) looks like this:

autoAddServers = 0
disabled = 0
removedTimedOutServers = 0
servers =      (address sterilized)

What I am confused about is the following:
1. Should I see the apps in the App listing on the indexer GUI screen? I don't see them there, and if I go the the Apps Manager I don't see any of them listed there either.
2. If I try to install the Google Maps app for example, I never see it on the indexer, and it gives me errors when I try to do a search from the search head.

At this point, I am confused about WHAT I should be seeing and WHERE I should be seeing it. The documentation and forums have provided some limited help, but as I said after looking at them I am even more confused than when I began. Does Deployment server need to be configured into the Distributed Search configuration? I started out thinking they were totally separate, but now I am wondering if I am understanding the pieces correctly.

Any help with this would be greatly appreciated!

Tags (1)
0 Karma


1. Should I see the apps in the App listing on the indexer GUI screen? I don't see them there, and if I go the the Apps Manager I don't see any of them listed there either.

No, you should/will not.

What happens in the kind of setup that you have - search head (SH) + indexer as search peer (SP) - is that when you issue a search from the SH, it will compile a bundle of all the knowledge objects it has and replicate that bundle to the search peer, which will then make use of knowledge objects in that bundle - and ONLY that bundle - for the extent of that specific search. That is, in executing the search request from the search head, the search peer will not make use of any of its own knowledge objects (field extractions, tag, eventtypes, ...). It will only use those that came in the bundle from the search head, and this bundle is valid for specific search in question only. The bundles will linger for a short while in the filesystem somewhere in $SPLUNK_HOME/var/run/searchpeer but effectively they are discarded.

So, do not confuse the search head's bundle replication with the deployment server's functionality. The deployment server distributes apps for permanent use by deployment clients, while the search head distributes knowledge objects for temporary use by search peers.

2. If I try to install the Google Maps app for example, I never see it on the indexer, and it gives me errors when I try to do a search from the search head.

Please give details on how you installed the app and what errors you're encountering. I don't recall what the Maps app contains, if it's search-time stuff only or if you need it on the indexer as well for some reason.

Path Finder

Ok. I will give that a shot and see how it works. There are a lot of little quirks in this system that you have to figure out to make everything work. It is simple once you understand the concepts, but it takes a while to get up to speed on it. Thanks for your help!

0 Karma


Yes, mounted bundles will make search peers look for bundles in the defined directory instead of having the search head replicate its bundles at the time of a search. The search peers will still only use these bundles only within the scope of the searches issued from the search head though, so it's still important not to confuse this with what the deployment server does.

Regarding the google maps issue, it's possible that apps with custom search commands need to be installed onto the search peer as well as the search head.

0 Karma

Path Finder

Ok, that helps me understand the bundle part. Using the mounted bundles should just make this part more efficient in a high load scenario, correct?

I had G Maps on the indexer before I built the SH. I have a frwrder that sends data to a new index (INX) on the indexer.

I removed G Maps from the indexer, and installed the SH. I added G Maps to the SH after setting up the mounted bundles.

When I run a search with the geoip command against the index INX, it gives me an error. Digging into the debug it shows that it cannot find my index. What do I need to do on the SH for it to be aware of IDX?

0 Karma

Path Finder

My only reason was because it stated in the documentation that it would generate less load on the head and peers if they could all read from a mounted bundle. This system will be heavily used so I am trying to be ready for that.

I do see that the status of the peer is Up and MOUNTED on the search head.

I guess my biggest question is this: Should I see the applications listed in the App Manager on the PEER machine? Am I getting hung up on something that does not show up, but isn't supposed to show up? How can I see what bundles the peer has as active?

0 Karma


Why question is why are you mounting sharebundles, doyou have large amounts of search-time data what can not be handled by replication process? If you added your search peers through the Splunk web GUI. Splunk should take care of replicating the bundles.

autoAddServers = 0
checkTimedOutServersFrequency = 300
disabled = 0
removedTimedOutServers = 1
servers = :8089,:8089

From the Search Head Navigate to Manger>> Distrbuted Search>> search peer. You sould see Replication status as Successful.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!


Or Learn More in Our Blog >>