The latest version of Splunk (4.2.1) has search head pooling. This means that you can have multiple instances of Splunk share search and app configuration data across both systems. Even the scheduled searches are synchronized. It is important to note that the indexer is what typically performs the heavy lifting in searches. Therefore, if your intent is to have two search heads distributing to a single indexer, you will not gain any performance by having an additional search head. Instead, you should have a single search head distributing to many (additional) indexers so our map reduce technology is leveraged. UPDATE: Conceptually, any Splunk instance (except for the universal forwarder) can act as a search head. Therefore, you could have two servers that perform both indexing and searching that peer to each other. When search head pooling is configured, the core user configs are shared as well as the search jobs. So from a standpoint of managing scheduled searches, Splunk will maintain the schedule between both systems (no duplicates). ... View more

In 4.2.1, you do not need to account for bucket sizes to figure out how to partition. You can easily set the hot/warm and cold with specific parameters: [volume:volume_name] maxVolumeDataSizeMB = <integer> In general, auto_high_volume works well and you should not be required to perform manual bucket sizing. Review the indexes.conf.spec or example file for more information. To answer the setting question, "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems. ... View more

Sounds like you should either: preparse the data and separate into individual classes create a multivalue field that takes the initial class value as input for the field name of curr_conns (must be an indexed field to do this) You can probably get away with doing some fancy regexes, but you will need to use the multivalue field parameters (MV_ADD in config files, max_matches when using rex). ... View more

One could write a Novel on this, but I'll focus on reporting type searches... Use the Advanced Charting view - this typically speeds up most searches of this type From a the "search" view: remove any unnecessary field extractions by: turning off field discovery; using the fields command so that it only returns the field you desire (eg - "my error | fields host"). For reports that need to analyze millions of events AND they are run consistently, use summary indexing Create a dashboard that persists a saved search (see below) For dashboards, create a saved search and force the dashboard to use the persisted result (useHistory parameter) If you have multiple indexes, search the specific index you need to report on. If your result set contains indexed fields, leverage them in your search (by default Splunk indexes host source and sourcetype) Create a distributed search environment and leverage the map/reduce feature (add an indexer) For extremely large summary search reports on systems where you have many cpus available for searching, schedule parallel searches on subsets of the data If your search environment is distributed and you DON'T need to run it remotely, use the 'localop' command (e.g. - a local summary index search or geoip lookup) - ... View more

You should be able to do this by editing the alert_actions.conf file in $SPLUNK_HOME/etc/system/local. Specifically, create/add the email stanza to specify csv non-inline results to be included.: [email] format = csv sendresults = 1 inline = 0 For reference, see the alert_actions.conf.spec file. ... View more