I have 2 servers available to deploy Splunk. If I read this doc : http://docs.splunk.com/Documentation/Splunk/6.2.4/Capacity/Referencehardware I understand that I should put a Search head on one server and the Indexer on the other.
But I was thinking: Isn't it better to deploy 1 indexer on each server and install the search head role on one of the servers? This way, I can distribute the cost of a search since the search engine is on the Indexers.
Since the search head is just a web application, and I will have only 2-3 users using this web application, so I think I don't need to dedicate a whole server for that role.
Moreover, when running a search with the
stats command, can you tell me the job of the search head? Is it just displaying data?
Or doesn't it take the data from the indexer and organize it?
What do you think?
That is a great question. I don't think there is a one size fits all answer. I think you are on the right path by checking looking at the number of users that will be using the application. You may also want to consider the number of queries per user. Some queries may be more intense than others.
I would not consider the search head as "just a web application". There is quite a bit of processing going on after the data is pulled from the indexes. There are things you can do to the indexes to make search time faster before or after indexing. There can be much more processing after the indexing is completed. I would take a look at the docs on link text and link text to see index time extraction and search time extraction. You should also think about all the applications and add-ons you will be using.
Personally, I like to dedicate the search head. It makes admin / troubleshooting problems easier if one system is not performing. Many add-on apps can be added to the search head, without effecting the indexer. Additionally, the cost of adding an additional indexer (if needed later) should be minimal compared to the total cost of the Splunk configuration. If you are spending the time / money to distribute your Splunk cluster, you will probably want to dedicate your search head.
Sorry. I was trying to link to 'props.conf' and 'transforms.conf' which talks about generating fields for your events. This is the basis of creating the Alerts / Graphs / Knowledge Objects and the other interesting things that make Splunk search so powerful.
The search head is responsible for more then just displaying the results. The search head does the following, and more..;
1) Distribute search jobs and associated knowledge objects to its search peer nodes (distributed search)
2) Collects the results from the search peers, collates, dedup's, and peforms any perfunctory stats and reductions on the data sets
3) Graphing and charting of the search results
4) Alerting based on custom alerts
5) Search Time knowledge object extraction and application
6) RBAC for indexes
7) Saved / Scheduled / Real time searches
While this isnt an exhaustive list of search head functionality, it is a good high level description of what the SH does.