Getting Data In

Can you add another transform to an existing, built-in, sourcetype (access_combined)?

tmeader
Contributor

So, we have some access_combined (mostly standard) formatted Apache and lighttpd logs coming in via a remote syslog-ng forwarder. As I said, the messages are completely standard aside from the fact that there's a timestamp and IP put on from the syslog-ng server (note, this CANNOT be changed, stuck with this formatting). Example:

Mar  5 22:27:42 xxxxxxxx lighttpd[13476]: 192.168.48.30 xxxxxxxxx - [05/Mar/2012:17:27:42 -0500] "GET /f9360/lads/archive/MOD04_L2/kfjsdslkdjflksdjf.hdf HTTP/1.1" 200 1315454 "-" "-"

In the interim, I've created another sourcetype called "fwd_access_combined", based on the "access_combined" out of default/props.conf and default/transforms.conf. This has been placed in the local/props.conf and local/transforms.conf (as you'd expect). And it works fine. In local/props.conf:

[fwd_access_combined]
MAX_TIMESTAMP_LOOKAHEAD = 150
REPORT-access = fwd-access-extractions
SHOULD_LINEMERGE = false
TIME_PREFIX = \[

and in local/transforms.conf (note, I included all the supplementary transforms that are used by the base "access_combined" in defaults.conf as well. ie - nspaces, alphas, alnums, etc):

[fwd-access-extractions]
# matches access-common or access-combined apache logging formats
# Extracts: clientip, clientport, ident, user, req_time, method, uri, root, file, uri_domain, uri_query, version, status, bytes, referer_url, referer_domain, referer_proto, useragent, cookie, other (remaining chars)
# Note: referer is misspelled in purpose because that is the "official" spelling for "HTTP referer"
REGEX = ^\w{3}\s++\d{1,2}\s++\d{2}\:\d{2}\:\d{2}\s++\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s++[[nspaces:process]]\:\s++[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]

As I said, this all works fine. If I just search for "sourcetype=fwd_access_combined" everything is extracted properly. However, what I want to do is to be able to use the single sourcetype "access_combined" and have it extract these as well. At first I tried using "rename = access_combined" under the fwd_access_combined sourcetype. Unfortuantely, as the docs state (I later found), this then only applies the extractions/transforms of the type that you're renaming too (thus nothing gets extracted properly). Then I tried doing away with the "fwd_access_combined" type entirely, and copied the "access_combined" out of default/props.conf into local/props.conf and added the "fwd-access-combined" as another entry for "REPORT". In other words, I put the following under local/props.conf:

[access_combined]
pulldown_type = true
maxDist = 28
MAX_TIMESTAMP_LOOKAHEAD = 150
REPORT-access = access-extractions,fwd-access-combined
SHOULD_LINEMERGE = False
TIME_PREFIX = \[

Even with that though, the custom transform doesn't seem to be getting applied at all.

Is there an easier way to accomplish this that I'm missing somehow? Any help with this would be greatly appreciated. (Note, Splunk was restarted inbetween all the changes to make sure they were properly picked up each time, that isn't the issue.)

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

It should work fine. However, is it a typo in your REPORT-access line, where you call the transform fwd-access-combined rather than fwd-access-extractions?

You can also try putting this in a separate REPORT- line, alphabetically before access. I'd recommend overriding just this property in local/props.conf that way:

[access_combined]
REPORT-aaccess = fwd-access-extractions
0 Karma

tmeader
Contributor

Yes, sorry, that was a typo in my post. It does not work with fwd-access-extractions. Likewise, the method you stated doesn't work either. Can you verify that that sourcetype "overloading" works properly for you? Splunk 4.3.1 running now.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...