So, we have some access_combined (mostly standard) formatted Apache and lighttpd logs coming in via a remote syslog-ng forwarder. As I said, the messages are completely standard aside from the fact that there's a timestamp and IP put on from the syslog-ng server (note, this CANNOT be changed, stuck with this formatting). Example: Mar 5 22:27:42 xxxxxxxx lighttpd[13476]: 192.168.48.30 xxxxxxxxx - [05/Mar/2012:17:27:42 -0500] "GET /f9360/lads/archive/MOD04_L2/kfjsdslkdjflksdjf.hdf HTTP/1.1" 200 1315454 "-" "-" In the interim, I've created another sourcetype called "fwd_access_combined", based on the "access_combined" out of default/props.conf and default/transforms.conf. This has been placed in the local/props.conf and local/transforms.conf (as you'd expect). And it works fine. In local/props.conf: [fwd_access_combined] MAX_TIMESTAMP_LOOKAHEAD = 150 REPORT-access = fwd-access-extractions SHOULD_LINEMERGE = false TIME_PREFIX = \[ and in local/transforms.conf (note, I included all the supplementary transforms that are used by the base "access_combined" in defaults.conf as well. ie - nspaces, alphas, alnums, etc): [fwd-access-extractions] # matches access-common or access-combined apache logging formats # Extracts: clientip, clientport, ident, user, req_time, method, uri, root, file, uri_domain, uri_query, version, status, bytes, referer_url, referer_domain, referer_proto, useragent, cookie, other (remaining chars) # Note: referer is misspelled in purpose because that is the "official" spelling for "HTTP referer" REGEX = ^\w{3}\s++\d{1,2}\s++\d{2}\:\d{2}\:\d{2}\s++\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s++[[nspaces:process]]\:\s++[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]] As I said, this all works fine. If I just search for "sourcetype=fwd_access_combined" everything is extracted properly. However, what I want to do is to be able to use the single sourcetype "access_combined" and have it extract these as well. At first I tried using "rename = access_combined" under the fwd_access_combined sourcetype. Unfortuantely, as the docs state (I later found), this then only applies the extractions/transforms of the type that you're renaming too (thus nothing gets extracted properly). Then I tried doing away with the "fwd_access_combined" type entirely, and copied the "access_combined" out of default/props.conf into local/props.conf and added the "fwd-access-combined" as another entry for "REPORT". In other words, I put the following under local/props.conf: [access_combined] pulldown_type = true maxDist = 28 MAX_TIMESTAMP_LOOKAHEAD = 150 REPORT-access = access-extractions,fwd-access-combined SHOULD_LINEMERGE = False TIME_PREFIX = \[ Even with that though, the custom transform doesn't seem to be getting applied at all. Is there an easier way to accomplish this that I'm missing somehow? Any help with this would be greatly appreciated. (Note, Splunk was restarted inbetween all the changes to make sure they were properly picked up each time, that isn't the issue.) ... View more

I'm trying to get a result table of all he hosts in our OSSEC environment that have changed status over the past 24 hours (vs the previous 24 hours). To that end, I've got the following, which works perfectly as far as returning the info I'm looking for: | set diff [search sourcetype=ossec_agent_control earliest=-2d@d latest=-1d@d | dedup reporting_host | table reporting_host,status] [search sourcetype=ossec_agent_control earliest=-1d@d | dedup reporting_host | table reporting_host,status] My problem is that this returns a table with two rows entries for each reporting host: the first line being the status from yesterday, and the second line being the status from today. So, say that a host went from "Disconnected" to "Connected". the table would be: reporting_host status HOST1 DISCONNECTED HOST2 CONNECTED What I'd like is to also have another column at the end specifying "Yesterday" for the first row, and "Today" for the second. Anyone know of a way to do that? Thanks. ... View more

Haven't been able to find any other questions anywhere asking about this, so I was wondering if anyone has tried it before? Basically, we have a firewall log that we're ingesting, which, for 95% of the events indexes on the field of "start_time=" (this is forced at index time in props.conf). This works great, except for the other 5% that aren't actually traffic messages, but rather adminstrivia type stuff generated by the firewall. These messages do however have a time-stamp at the end of the message included in parenthesis. Since this 5% of the messages is generating "Unable to extract time-stamp" error messages on the indexers whenever they're encountered, I'd like to be able to have this sourcetype ("netscreen") be able to extract the date from the "start_time" field in almost all cases, but, when that doesn't exist, key in on that time stamp at the end of the message. Is something like this possible? We already use very specific transforms for the netscreen sourcetype (called in props.conf to do the extractions)... would it be possible to include the multiple time-stamp recognition in there somehow? Or are we just out of luck (and will have to accept the error messages on the indexers; letting Splunk index on the time the event was received)? Thanks in advance, and sorry for any incomprehensibility of this... it's late. ... View more