Unfortunately, as of Splunk 4.3 you cannot have more than one functional stanza/input per UDP port. The devil is in the details of inputs.conf.spec where this limitation is made implicit by the following statement : udp://<remote server>:<port> * If <remote server> is specified, the specified port will only accept data from that server. It's important to understand that this refers to the specified port, not to the specified data input. This will be further clarified in our next maintenance release by the addition of the following statement to inputs.conf.spec : * Only one stanza per port number is currently supported. If you have configured more than one UDP input stanza using the same port but specifying different hosts of origin, pushing the UDPInputProcessor log channel to DEBUG level will typically yield messages like this one : 02-03-2011 11:19:07.167 DEBUG UDPInputProcessor - The UDP packet data from '10.1.5.139' was refused only accepting data from '10.1.8.120' This means that in your case, you have to use transformations to route your UDP data to different indexes, as per Lisa's 2nd suggestion. A few considerations I would add : A) If you are using a [host::] stanza in props.conf to conditionally apply your routing, I would suggest to use something like this to add resilience to RDNS failures: [host::(my.hostname.com|10.1.8.120)] B) I personally am not too crazy about host-based filtering in the stanza name of props.conf. I would rather do something like this: B.1) In props.conf, use a sourcetype-based stanza to apply a generic routing transformation to the data coming from the UDP input: [syslog] TRANSFORMS-idx_routing = generic_idx_routing B.2) In transforms.conf, use a regular expression on SOURCE_KEY = MetaData:Host to route the events to a different index, whose name will be dynamically based on the value of "host": [generic_idx_routing] SOURCE_KEY = MetaData:Host REGEX = (device1|device2|device3)\.domain\.com DEST_KEY = _MetaData:Index FORMAT = $1 In this example, events with a value of "device1.domain.com" for the "host" field will be sent to the "device1" index, while "host=device2.domain.com" and "host=device3.domain.com" will respectively go to indexes "device2" and "device3". Note : This implies that you will have created the indexes necessary for the routing before-hand, based on the hostnames of the devices sending data to UDP:514. If you configure a new device to send data to that port, make sure that you create the corresponding index on the indexers first. C) Finally, an alternative to dynamic index-routing would be to use several transformations with hard-coded REGEX and FORMAT values, like so: C.1) In props.conf, use a sourcetype-based stanza to apply several routing transformation to the data coming from the UDP input: [syslog] TRANSFORMS-idx_routing = device1_idx_routing, device2_idx_routing C.2) In transforms.conf, use a regular expression on SOURCE_KEY = MetaData:Host to route the events to a different index, whose name will be dynamically based on the value of "host": [device1_idx_routing] SOURCE_KEY = MetaData:Host REGEX = device1\.domain\.com DEST_KEY = _MetaData:Index FORMAT = device1 [device2_idx_routing] SOURCE_KEY = MetaData:Host REGEX = device2\.domain\.com DEST_KEY = _MetaData:Index FORMAT = device2 This would be particularly useful if you want to use index names that are not included in the sending device's host name or in the event's raw data. Note: In both scenarios, you might prefer to use the event raw data as the field on which to apply the REGEX rather than "host", in which case you would set SOURCE_KEY = _raw and modify your REGEX accordingly to dynamically extract the name of the target index from the event data. ... View more

As of Splunk on Splunk 2.0, SoS ships with the ps_sos.sh scripted input which once enabled, allows you to track the CPU and memory usage of the main Splunk components : Splunkweb, the cherrypy-based web front-end. splunkd, the back-end daemon which manages the indexing of data. The searches fired by splunkd to retrieve events and build reports. Note: Currently, this scripted input only exists on Linux and Unix. A similar functionality for Windows will be added in a future SoS release. Also: You do not need to install the SoS technology add-on on an instance where SoS is already installed. The ps_sos.sh data input ships both with the SoS app and with the SoS technology add-on for Unix and Linux. 1) On a search-head or stand-alone indexer: Install the SoS app. Enable the ps_sos.sh scripted input by one of the following methods: a) Go to Manager > Data Inputs > Scripts. Enable the ps_sos.sh data input. or b) Run the following command from a terminal window: $SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0 2) On search peers: Install the SoS technology add-on for Unix and Linux. Enable the ps_sos.sh scripted input by one of the following methods: a) If Splunkweb is running, go to Manager > Data Inputs > Scripts. Enable the ps_sos.sh data input. or b) Run the following command from a terminal window: $SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/TA-sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0 Going forward, you will be able to track the CPU and memory usage of Splunk in the "Splunk CPU/Memory Usage" SoS view. Per-search memory usage for the biggest memory-consuming searches can be consulterd in the "Distributed Searches Memory Usage" view. For further information on deployment best practices for SoS, please refer to this Splunk Answer. ... View more

As per this documentation topic, violations are counted per-pool. When a given enterprise pool reaches 5 violations, all slaves of that pool see their search disabled. Other pools should not be affected by this. ... View more

As of Splunk 4.3, you can now export an unlimited number of events from the UI. Do note, however, that exporting too many events in that manner (typically, several millions) could cause Splunkweb to misbehave and possibly to become temporarily unresponsive. If you really need to often export large number of events, we would still recommend the use of the outputcsv command and/or to run the search from the CLI. ... View more

I believe you are experiencing a crash that was discovered just today and filed as a bug with reference SPL-47232. Symptoms: The signature of this crash is as follows : The crashing thread is always DispatchReaper, the thread which curates the search artifacts in the dispatch directory ( $SPLUNK_HOME/var/run/splunk/dispatch ). The crashing thread is indicated at the very beginning of the crash log file that can be found in $SPLUNK_HOME/var/log/splunk : [build 115073] 2012-01-12 14:54:12 Received fatal signal 6 (Aborted). Cause: Signal sent by PID 28468 running under UID 0. Crashing thread: DispatchReaperCrashing thread: DispatchReaper <== The file logging splunkd's stderr output ( $SPLUNK_HOME/var/log/splunk/splunkd_stderr.log ) will contain the following error: terminate called after throwing an instance of 'PropertyPagesException' what(): Cannot get user to act as: No user info provider registered (user: splunk-system-user, app: user-prefs, root: /opt/splunk/etc) The crash almost always occurs on splunkd start-up, sometimes on the first start-up attempt after the 4.3 upgrade. It has also been known to occur during normal operation of the Splunk instance. Work-around: Until this crash is fixed in an upcoming release, you'll have to take the following steps to allow splunkd to start again: Delete the contents of the search dispatch directory: rm -rf $SPLUNK_HOME/var/run/splunk/dispatch/* Start up Splunk: $SPLUNK_HOME/bin/splunk start Note: If you are unfamiliar with the search dispatch directory, it is the location where Splunk stores search artifacts for past and currently running searches. That data is volatile by nature and can be regenerated by re-running the searches that generated it. ... View more

Starting with version 2.0, the Splunk on Splunk app (a.k.a SoS) leverages distributed search to allow introspective analysis and troubleshooting on your deployment from a centralized location. Let's cover the details of deploying the SoS app, F.A.Q-style: - Where should SoS be installed? The SoS app should be installed on your search-head. - What if I have more than one search-head? SoS can only analyze and report on Splunk instances that it can reach by means of distributed search. Unless you have a federating search-head for which the other search-heads are declared as search-peers, you should install SoS on each search-head that you wish to gain introspection on. - What about search-head pooling? Any extra steps I need to take to install SoS on a search-head pool? You will almost certainly need to manually restart Splunk on the pool members other than the one where SoS was installed from the user interface in order for all pool members to become aware of the app. - Are there any requirements for SoS to work? The SoS app depends on the UI module library of the Sideview Utils app. Make sure you install SVU (minimum version required : 1.17) prior to SoS. - Does anything need to be installed on the search peers? Optionally, you can install the SoS technology add-on on your Unix and Linux search peers. This add-on provides scripted inputs that track the resource usage (CPU, memory, file descriptors) of Splunk. - Do I also need to install the SoS technology add-on on the search-head? No, that would be redundant. The SoS app ships with the same data inputs than the technology add-on. - What do I need to do for the SoS data inputs to track Splunk resource usage? By default, the data inputs for SoS and SoS-TA are disabled. They must be enabled in the UI (look for ps_sos.sh and lsof_sos.sh in Manager > Data Inputs > Scripts) or from the command line (refer to the README file for more information on this) before they can report resource usage information. - What if my search-head is configured to forward data back to the search peers? Any action required for the SoS app to work in that scenario? If the search-head where the SoS app is installed forwards its own events to the indexers it queries (typically, to spray summarized events back to the indexers) you have to make sure that events for the _internal index are also forwarded. This is not the case by default due to the forwardedindex filters in outputs.conf, which prevents _internal events from being forwarded. To correct this, add the following configuration to $SPLUNK_HOME/etc/apps/sos/local/outputs.conf : [tcpout] forwardedindex.3.whitelist = _internal - What about my heavy forwarder? Can SoS query and report on it? Yes. Please see this Splunk Answer for details on how to achieve this. - How can I make sure that SoS reports accurate license usage information in the Metrics view? 1. Make sure to install SoS on the search-head that acts as a license master, or at least that the license master's _internal index can be accessed by distributed search. 2. [Splunk 4.2.x only] Make sure to enable the scheduled search named sos_summary_volume_daily in Manager > Searches and Reports. This search populates the sos_summary_daily summary index, aggregating your license usage on a daily basis. 3. [Splunk 4.2.x only] Backfill the summary index "sos_summary_daily" by running the following command: $SPLUNK_HOME/bin/splunk cmd python $SPLUNK_HOME/bin/fill_summary_index.py -app sos -name sos_summary_volume_daily -et -28d -lt -1d -j 10 -owner admin Note: You do not need to perform steps #2 and #3 if you are running Splunk 4.0.x/4.1.x or 4.3.x. ... View more