Let's start out by separating your concerns and working out a logical sequence of first principles to build on. Make Splunk able to receive a trap and index it. Make a server able to send ANY trap successfully to Splunk (and see it get indexed) Make a server able to send traps based on a threshold For #1, you can choose to use the native snmptrapd on the host running Splunk, or you can choose to use the modular input. Given the options, I would personally choose to run snmptrapd for the following reasons. It separates the concerns of listening on the socket from indexing the data. Limits the amount of code that must run as root to support the "standard" UDP port of 162. That is, Splunk does not have to run as root, and there is no iptables port redirection magic required in order to map packets arriving at 162 to a higher port. It makes debugging easier, because I have a file that traps should land in -- regardless of whether Splunk is properly configured to process that file or not. It minimizes the possibility of missing SNMP traps due to Splunk restarts. Similar to syslog, the snmptrapd daemon will need to be restarted far less often than Splunk and it will restart orders of magnitude faster. MIBS are suddenly orders of magnitude easier to deal with, because we're not fooling with the obtuse (or is it abstruse?) MIB format that pySNMP uses. So given the decision to use the native snmptrapd then our list of steps gets refined a little. Configure snmptrapd to receive traps and write them to a logfile Make a server able to send ANY trap successfully to snmptrapd Configure Splunk to be able to monitor snmptrapd's log file and see it get indexed Make a server able to send traps based on a threshold STEP THE FIRST I'll use my CentOS 6.2 box as a guinea pig here. We start by making sure Net-SNMP is installed. sudo yum install -y net-snmp net-snmp-utils Now let's configure it. In /etc/snmp/snmptrapd.conf : # Example configuration file for snmptrapd # # No traps are handled by default, you must edit this file! # # authCommunity log,execute,net public # traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script cold authCommunity log public And in /etc/sysconfig/snmptrapd : # snmptrapd command line options # OPTIONS="-Lsd -p /var/run/snmptrapd.pid" OPTIONS="-A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid" We can now do a service snmptrapd start and see the daemon start, and see it log a little intro line to the snmptrapd.log file: [root@localhost log]# tail -f snmptrapd.log Created directory: /var/lib/net-snmp/mib_indexes NET-SNMP version 5.5 STEP THE SECOND Now we do a simple test to see if another host (my Mac) can send an arbitrary trap and have it show up in the log. snmptrap -v 1 -c public 192.168.96.190 .1.3.6.1.6.3 "" 0 0 coldStart.0 And hey, look, I see it in the log on the CentOS box: 2015-08-09 09:50:12 0.0.0.0(via UDP: [192.168.96.1]:64763->[192.168.96.190]) TRAP, SNMP v1, community public SNMPv2-SMI::snmpModules Cold Start Trap (0) Uptime: 0:00:00.00 If your trap didn't make it, then you should be looking at things like firewalls and so forth in order to debug the issue. STEP THE THIRD So, this should be as simple as adding a Data input in the Splunk UI, or editing inputs.conf directly. I'll go with inputs.conf because that's how I roll. [monitor:///var/log/snmptrapd.log] sourcetype=snmptrapd And I restart Splunk. These events are now visible in Splunk. STEP THE FOURTH This is where it actually gets interesting. But, it is also the "least Splunkadelic" part of the question. For most SNMP agents, the idea of "sending based on a threshold" is not a common concept. So, how you go about this part may vary from agent to agent. Since I have NET-SNMP, and it is thoroughly documented in the world, I'm going to lean heavily on a couple of references. http://serverfault.com/questions/248332/set-up-snmp-trap-for-disk-usage http://www.net-snmp.org/wiki/index.php/TUT:DisMan_Monitoring In NET-SNMP land, the general idea of a "threshold trap" is done using the DisMan Event MIB. Following the steps in the net-snmp wiki link, I added this to /etc/snmp/snmpd.conf : createUser lemonitor SHA lepassword AES rouser lemonitor iquerySecName lemonitor trap2sink 127.0.0.1 public monitor -r 5 machineTooBusy hrProcessorLoad > 2 And then ran a service snmpd restart . Because I set the threshold so low, I almost immediately got a trap: 2015-08-09 10:10:28 localhost [UDP: [127.0.0.1]:37402->[127.0.0.1]]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (6520) 0:01:05.20 SNMPv2-MIB::snmpTrapOID.0 = OID: DISMAN-EVENT-MIB::mteTriggerFired DISMAN-EVENT-MIB::mteHotTrigger.0 = STRING: machineTooBusy DISMAN-EVENT-MIB::mteHotTargetName.0 = STRING: DISMAN-EVENT-MIB::mteHotContextName.0 = STRING: DISMAN-EVENT-MIB::mteHotOID.0 = OID: HOST-RESOURCES-MIB::hrProcessorLoad.196608 DISMAN-EVENT-MIB::mteHotValue.0 = INTEGER: 51 And there you have it. ... View more

I don't follow your use case entirely, but you can use a subsearch to emit earliest and latest. See http://answers.splunk.com/answers/65255/returning-time-from-subsearch-to-main-search.html HOWEVER, looking at your use case in the reply below you can probably accomplish the same with relative time trickery, something like: earliest=@d+10h latest=@d+17h <rest of search> The "additive/subtractive" modifiers on the relative time operators are a great way of getting to a particular point in time. You can add to them in nearly arbitrarily complex ways too. earliest=-1d@d+10h+32m latest=@d-15h+30m Or other such tomfoolery. Perhaps this is more like what you're trying to do? Also also, if you are doing day-over-day comparisons or other such things, you should know about the timewrap app. https://splunkbase.splunk.com/app/1645/#/overview ... View more

I don't think this is a supported configuration. To my knowledge, Splunk support does not support the use of a hardware load balancer between forwarders and indexers. What you would gain from one is also perhaps dubious. When you configure Splunk to use a single IP (or a DNS name that resolves to a single IP), then the forwarders connect their TCP session and never disconnect. This would mean that each forwarder would, for the life of its process, always send data to a single indexer. (Unless, of course, the TCP connection was interrupted for some reason). When Splunk has multiple IPs, or a DNS name that resolves to multiple IPs, then its round-robin will connect, send data, and disconnect. This way, you get a more even spread of data across indexers. I just don't think a physical load balancer gives you anything good here. Actually, it probably makes things worse. ... View more

By default, no, there is no crypto between forwarders and indexers or forwarders and intermediate forwarders. If you choose to do encryption, the certificate situation is up to you. However, I would STRONGLY suggest use of either a private CA or 3rd party certs. Don't use the default self-signed stuff. George Starcher and I did a .conf talk on this subject at .conf 2014, should be able to find it in the talk archives. In the case of UF -> HF -> Indexer, the UF is an "SSL Client", the HF Is both an SSL "server" (toward the UF) and an SSL "client" (toward the indexer), and the indexer is an SSL "server" ... View more

Big question. TSM has a lot of data available, and what do you want to do with it exactly? "What problem are you trying to solve, exactly?" is always a good question. One good simple starting point is the dsmaccnt.log file. It's a basic CSV of performance and accounting data around every TSM server connection. Lots of good data there, and it's simple to ingest and parse. Beyond that, there's several good data points inside the TSM database itself, such as the activity log, the summary data, the nodes table, and several other things. In "modern" TSM (V6.1 and above) all of this is stored in DB2, so you might be able to get at it using the DB Connect app pretty easily. If you aren't deeply familiar with both TSM and Splunk this might be the right kind of job to turn over to Professional Services to help you figure out how to ingest this data. ... View more

For starters, I don't think Splunk can arbitrarily expand environment variables in every possible place in a .conf file. Some places will expand $SPLUNK_HOME , but I don't think this is universal. You should learn about either host_segment or host_regex options in inputs.conf to automatically grab your hostname= for you. This way, you do not have to hardcode a stanza for each host. I don't think that adding the 'originating forwarder' to the source field is necessarily the best idea either. Some of the solutions that @mikaelbje provides links to can help. One solution I like is to add a completely new indexed field of syslog_receiver or similar. So we would wind up with: (inputs.conf) [monitor:///opt/inboundlogs/*/*_syslog.log] syslog_receiver = syslog_server_1_hostname host_segment=3 (props.conf) [source::/opt/inboundlogs/*/*_syslog.log] TRANSFORMS-addreceiver = addreceiver (transforms.conf) [addreceiver] SOURCE_KEY = syslog_receiver REGEX = (.*) FORMAT = syslog_receiver::$1 WRITE_META = true (fields.conf) [syslog_receiver] INDEXED=true It doesn't answer the question you asked, but it solves the problem you want to solve while reducing your number of hardcodes to one - which is the name of the syslog_receiver . And with proper configuration-file-overlaying, this is not an issue at all. For example you could have a $SPLUNK_HOME/etc/system/local/inputs.conf with [default] syslog_receiver = forwarder_host_name And then the .conf layering just handles it for you. ... View more

The "junk" is an expansion of reverse lookups that happens because of various CIM-compliant TAs you have installed. I'll explain the concept of reverse lookups, but first let's go back to automatic lookups. Say you have a sourcetype bob defined with an automatic lookup. In props.conf: [bob] LOOKUP-actions = boblookup someinputfield OUTPUT action And in boblookup.csv you have: someinputfield,action potato,deny tomato,accept blueberry,accept Now your normal expectation is that when you do a search on sourcetype=bob that events will be matched against the lookup field and have a new field named action when someinputfield has a value of "tomato, potato, or blueberry". But, this same thing can be applied in reverse too. If you search on action=accept , then Splunk can look through all of its config files and reason-out something like this: Sourcetype bob has a lookup that outputs a field named action based on this CSV file. I see here in the CSV file that action=accept is returned whenever sometinputfield=blueberry or someinputfield=tomato . So there is an equivalency here: ( sourcetype = bob AND ( someinputfield = blueberry OR someinputfield = tomato ) ) This is the fundamental step of a reverse lookup - the goal is to attempt to make automatic lookup fields searchable. This is a necessary evil for CIM-compliant apps like Enterprise Security because of how often they use automatic lookups to normalize field names and values. There's a whole longer discussion here about the performance impacts around this. While it made your example situation slower, there are many other counter examples where this approach (up to a point) speeds things up. ... View more

As discussed in irc: Make your own local/authorize.conf with: [default] schedule_rtsearch = ... View more

With subsearches, by default fields matter. Your subsearch is actually outputting something more like ( ip = 1.2.3.4 ) So the assumption is that source "B" has a field named ip, with a value of 1.2.3.4. You can rename the output field from the subsearch to match the source "B" fields, or you can tell the subsearch to output the field "naked" by renaming the output field ip to query . See http://docs.splunk.com/Documentation/Splunk/6.2.3/Search/Changetheformatofsubsearchresults ... View more

In my opinion, this is just a bad recommendation in the documentation. The default-provided certificates & configuration give you no practical authentication strength, because the certificates are not verified. The default-generated certificates and associated configuration settings should be considered "appropriate for a test / proof-of-concept / pilot, but completely inadequate for production" Using Deployment Server as an example of inter-Splunk communication - do you REALLY want your forwarders to trust ANY server claiming to be the deployment server? The default configuration puts you in a position where any number of attacks (arp poisioning, DNS games, etc) could cause a forwarder to trust a rogue deployment server. The person who controls the content of the apps on a deployment server effectively has permissions equal to the user running Splunk on each deployment client. Are you sure this is a good place for weak SSL configs? Generate your own certs, off of your private CA, and enable (at the barest minimums) certificate verification and common name checking. In my opinion, asking Splunk to make the default certs "seem stronger" with a larger key length and newer hash algorithm is a step in the wrong direction. The default certs should stand out like sore thumbs in a production install, so that they are replaced with something appropriate. ... View more