I got this answer from my SE - support didn't get back to me. I searched our cases and based on responses that support has given to another customer, this is a known issue (SPL-122185) and is targeted to be fixed in 6.4.2. . Here is their suggested workaround for another customer - For the time being please attempt following workaround: 1 - create a log-local.cfg (cloned from log.cfg) inside $SPLUNK_HOME/etc, see also here: http://docs.splunk.com/Documentation/Splunk/6.4.0/Troubleshooting/Enabledebuglogging#log-local.cfg 2 - edit log-local.cfg and add following line inside the [splunkd] stanza (at the bottom will be ok): category.AuditTrailManager=CRIT 3 - restart splunk. 4 - check splunkd.log again and check if you still the error messages you noticed earlier. This won't affect any main splunk functionalities. It just will affect the events in the _audit index to not have the sequence ID, while all other info will be there. Please let me know how it goes, many thanks. Sorry for the inconvenience. ... View more

I came upon this thread a bit late, but with the advent of v10 / v11 software, and multimodule Big-IP systems (WAM + ASM + LTM on one system, for example), I've found a different tack on the transforms.conf. You can see some of it in the $SPLUNK_HOME/etc/apps/SplunkforF5/default/ config files. Each of the modules that runs through AlertD will prepend the log message with a number in this format: REGEX = (\d{4}[0-9A-Fa-f]{4}:\d+:). For example, the apm_log regex is transforms.conf:REGEX = :\s(?:0149[0-9A-Fa-f]{4}:\d+:|0125[0-9A-Fa-f]{4}:\d+:\s[0-9A-Fa-f]{8}:) Make sure your props.conf file calls those dynamic transforms out, and then you don't have to manually define every LTM on your network in transforms.conf. TRANSFORMS-f5 = firepass_sourcetyper, asm_sourcetyper, apm_sourcetyper, irule_sourcetyper, PSM_sourcetyper_smtp, PSM_sourcetyper_http, PSM_sourcetyper_ftp, bigip_sourcetyper ... View more

Method 3 works perfect for me. ... View more

Thanks, I actually hadn't looked at it from the asm point of view. I was focused on the ltm piece, but I see what you are referring to. I'll give it a try, although I'd still like to have splunk process both asm and ltm logs. ... View more

HSL logging via irules is excellent for application traffic, but not for administration traffic, audit logs, and irule event logging. Also, HSL is only available in Big-IP v 10.0 and newer. For syslogging administrative activity, you want this (also 10.0 syntax, but it is there in v. 9 as well, via 'bigpipe syslog' commands). # tmsh list /sys syslog sys syslog { remote-servers { syslog { host 10.10.10.2 } } } Note that if you use a syslog server via your OOB management network interface, you will be able to log messages during software upgrades, or during failsafe events, where the application network is not available. However, your System logs will then come from the management interface address, and your HSL logs will come from a different address. Also note that in any event, both members of an HA pair will syslog as their own device IP addresses, and will not use the floating address to send logs. HSL logging has a method for specifying the address that the message should come from, so HA pair logs as the same address. Cheers~ Casey F5 IT Network Engineer ... View more