Thanks, Duane! Great presentation, by the way! The only problem we've noticed in testing is that the UF does not stop properly (at least on Windows) when the sslCommonNameToCheck does not match the cert presented by the DS. It hangs and chews some CPU for a while before it finally stops. In that sense it doesn't have a "solid" feel to it; I wonder if we should send a feature request for Splunk to build explicit support for this use case? ... View more

Yes, null queue is an index time transform so it requires a restart. ... View more

In addition to helping to ensure that files are completely written to before being read and deleted by the batch input, using this approach also lets you inspect the logs in cases like this. Getting headers and no data is just weird; I think being able to see the files would be helpful here. Oh, I also get a lot of files with names starting with tmon. that only have headers and I wrote the batch input to ignore those; are you sure you aren't indexing those as cisco_wsa_squid? ... View more

#!/bin/bash set -e set -u SRC_PATH="/opt/splunk/bigdata/inputs/wsa" TGT_PATH="/opt/splunk/bigdata/inputs/wsabatch" DIRS=($(find "$SRC_PATH" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 basename)) # Note that all target directories must exist... for DIR in "${DIRS[@]}" do # Move all files whose inode was modified at least one minute ago... find "$SRC_PATH/$DIR" -type f -cmin +1 -print0 | xargs -0 -r -I file mv -t "$TGT_PATH/$DIR" -- file done ... View more

Here's what I use on the forwarder (is interval a valid option for a batch input?): [batch:///opt/splunk/bigdata/inputs/wsabatch/*/aclog*] disabled = false move_policy = sinkhole host_segment = 6 crcSalt = sourcetype = cisco_wsa_squid index = wsa I also decided to have the WSA send its files over SCP to a folder that isn't directly monitored by Splunk, then I move the files to a folder that's watched by a batch stanza if they haven't been written to in over a minute. I create two identical directory structures then run a script with cron every minute. I'll post that separately. ... View more

Please move that config to inputs.conf in the affected stanza and restart the indexer. ... View more

You're right that you do need the WSA app on the heavy forwarder since it has some props entries used in parsing. Probably not required on the indexer but it won't hurt. Since you got this working on your indexer, my guess is that you have some config on the heavy forwarder that's causing the sourcetype, destination index, or both, to be modified. Maybe a props entry based on source, or an ambiguous inputs stanza? I haven't noticed any tricks to routing WSA logs through a heavy forwarder but due to the bursty nature of the log uploads I use a dedicated forwarder and wouldn't have run into app conflicts. ... View more

Using metadata is probably the fastest way to get the start/end timestamps for each source. You can then get the difference in seconds pretty easily. You'll need to have some kind of search that returns the record count for each file and that might involve using rex to pull that field out. That field extraction should happen in the subsearch part of the example below: | metadata type=sources index=yourindex | eval duration=lastTime-firstTime | join source [search index=yourindex recordcount=* | top 1 recordcount by source] | table source, duration, recordcount If you're going to have a lot of events, optimizing that subsearch will be crucial when you start getting into those weekly/monthly groupings. There will be some stuff to add to turn this into a chart, but I'd probably start with the field extraction and getting the duration and record count into a table and go from there. Good luck! ... View more

I think the config looks fine. Something isn't right about that log message, though; it looks like it's logging a failure to send the alert email rather than the alert itself. Do you have the WSA configured to send email to you when this fails? Entries from both the WSA and the SSH server would be great. A few guesses as to what else might be wrong: - Directory permissions for /opt/splunk/var/log (should be writable by usrscp and at least readable by Splunk) - Permissions for ~/.ssh for the usrscp user (should be 700). - Permissions for ~/.ssh/authorized_keys (should be 644) ... View more

Hmm...I'm surprised that didn't fix it. To be clear, you're still getting the exact same error after setting the log subscription to SSH2? Do you have any log entries from the SSH host? ... View more

The WSA has an option for SSH1 or SSH2 on the log subscription. I imagine most SSH servers have disabled version 1 so that could be your problem; I'd try selecting SSH2 on the WSA and giving it another try. Good luck! ... View more

Couple of links about using custom certs: http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousesignedcertificates ... View more

If you want to be really sure, you could use Wireshark to capture traffic between the forwarder and indexer on your SSL port. Capture for a couple of minutes to make sure you see several connections (assuming 30s connect interval). Follow one of the TCP streams and the only text that should be readable includes things like SplunkCommonCA that pertain to the default certs. This, combined with the info from metrics.log on the indexer, should give you pretty good confidence that SSL is working properly. If you're that concerned about SSL, make sure you're using a custom CA and verifying certs. ... View more

Here's a good entry from metrics.log on an indexer. Note connectionType=cookedSSL and ssl=true: 11-21-2013 18:21:04.211 -0600 INFO Metrics - group=tcpin_connections, 10.10.10.10:57308:9997, connectionType=cookedSSL, sourcePort=57308, sourceHost=10.10.10.10, sourceIp=10.10.10.10, destPort=9997, _tcp_Bps=157.95, _tcp_KBps=0.15, _tcp_avg_thruput=0.15, kb=2.31, _tcp_Kprocessed=2.31, _tcp_eps=0.27, build=163460, version=5.0.3, os=Windows, arch=x64, hostname=forwarder, guid=ABCDEF12-1234-ABCD-1234-ABCDEF123456, fwdType=uf, ssl=true, lastIndexer=None, ack=false ... View more

Unless the "timed out" message is appearing repeatedly in the logs, if the events are being indexed you might be OK. However, again, I was only able to match your indexer log output when I intentionally added duplicate inputs. My live indexers output only one set of TcpInputConfig events when starting: 10-29-2013 06:09:51.078 -0500 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL) 10-29-2013 06:09:51.078 -0500 INFO TcpInputConfig - IPv4 port 9997 is not compressed Have you run "splunk btool inputs list" just to double-check that you don't have a duplicate? ... View more

The log on the indexer suggests that you might have two input.conf stanzas using port 9997. Can you verify that you don't have a regular splunktcp:9997 stanza in there? If you want to support both non-SSL and SSL forwarders, you'll need to choose a different port for either of the inputs. Here's working configuration from a forwarder and indexer: Indexer [splunktcp-ssl:9997] [SSL] password = password rootCA = $SPLUNK_HOME/etc/auth/cacert.pem serverCert = $SPLUNK_HOME/etc/auth/server.pem Forwarder [tcpout] defaultGroup = indexer01 [tcpout:indexer01] server = indexer01:9997 sslCertPath = $SPLUNK_HOME/etc/auth/server.pem sslPassword = password sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem I'm not using compression in my example since the documentation for inputs.conf says that it only applies to non-SSL inputs. I'd remove that from the indexer and forwarder config and restart both instances to make sure that's not part of the problem, but my main concern is the possibility that there are two listeners using the same port. With the above configuration, you should start seeing events for your forwarder host in the _internal index almost right away. Good luck! ... View more

We're using the same approach; we use ACE in production but have also tested with F5. You can take advantage of the fact that syslog is a fire-and-forget protocol and therefore don't need to use SNAT; that will keep your source IPs intact. I would create a Standard UDP virtual server with a SNAT Pool of None. You might need to tune the default Action On Service Down on the pool to Reject; on the ACE we were having problems with active connections sticking to servers where the health monitor had failed. Speaking of health monitors, we run both TCP and UDP on the syslog servers and use a TCP monitor; UDP monitoring didn't work for us. Here's an F5 config that I just tested, hope this helps (note that the syslog sender, receivers, and F5 are all on different subnets): ltm virtual udpsyslog_514_vs { destination 10.10.10.10:514 ip-protocol udp mask 255.255.255.255 pool udpsyslog_514_pool profiles { udp { } } vlans-disabled } ltm pool udpsyslog_514_pool { members { 10.10.20.10:514 { address 10.10.20.10 session monitor-enabled state up } 10.10.20.20:514 { address 10.10.20.20 session monitor-enabled state up } } monitor tcp service-down-action reset } ... View more

There was a problem with 5.0.2 that affected real time alerts and was fixed in 5.0.3. It's in the 5.0.3 release notes as "Real Time Alerts not working consistently in 5.0.2. (SPL-62129)". Might be worth taking a brief outage to upgrade to 5.0.5. Good luck! ... View more

Interesting! I didn't get that error but I tested on 5.0.2. Thanks for posting what worked for you! ... View more