All Apps and Add-ons

How do you customize inputs for Splunk App for Web Analytics?

DaClyde
Contributor

Due to extensive lack of foresight, I am working in an environment with a Splunk instance that is ingesting Tomcat logs (supporting a Liferay instance) that are not in the standard index/sourcetype (i.e., not access_combined) with non-standard field extractions. Basically the field extractions line up more with IIS than with Apache access logs.

Has anyone successfully managed to implement the Splunk App for Web Analytics in a similar scenario? After digging through the .conf files, I would think it would just require adjusting all of the sourcetypes and field references to use my environment's settings, but in some cases, I am not entirely able to tell which are standard fields, and which are fields being created by the app.

So has any one had any success trying this?

0 Karma
1 Solution

jbjerke_splunk
Splunk Employee
Splunk Employee

Of course!

You can adjust the props.conf freely. There are a couple of bundled sourcetype configurations and if they don't line up I suggest you use the access_combined as a base and move the columns around until you are happy.

Access_combined is controlled from a standard transforms called access-extractions that is bundled with Splunk. You can find this in system/etc/local/transforms.conf

Here is an example of a custom one I developed recently:

[access-extractions-custom]

##0:0:0:0:0:0:0:1 - anonymous 13/Jan/2019:23:58:23 +0100 "GET /mywebrequest.html HTTP/1.1" 200 2 "http://localhost/referringurl.html" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"

REGEX = ^(?<clientip>[^\s]+)\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++(?<req_time>[^\"]+)\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]

You refer to this stanza in props.conf for your custom sourcetype.

You also need to add the custom sourcetype to the eventtype.conf file in the stanza called "web-traffic"

Let me know how you get along.

j

View solution in original post

jbjerke_splunk
Splunk Employee
Splunk Employee

Of course!

You can adjust the props.conf freely. There are a couple of bundled sourcetype configurations and if they don't line up I suggest you use the access_combined as a base and move the columns around until you are happy.

Access_combined is controlled from a standard transforms called access-extractions that is bundled with Splunk. You can find this in system/etc/local/transforms.conf

Here is an example of a custom one I developed recently:

[access-extractions-custom]

##0:0:0:0:0:0:0:1 - anonymous 13/Jan/2019:23:58:23 +0100 "GET /mywebrequest.html HTTP/1.1" 200 2 "http://localhost/referringurl.html" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"

REGEX = ^(?<clientip>[^\s]+)\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++(?<req_time>[^\"]+)\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]

You refer to this stanza in props.conf for your custom sourcetype.

You also need to add the custom sourcetype to the eventtype.conf file in the stanza called "web-traffic"

Let me know how you get along.

j

View solution in original post

DaClyde
Contributor

I finally got everything working!

However, it has introduced a new problem with my old sourcetype.

Is there a way to create a sourcetype alias like a field alias, so the same data can be accessed with two different sourcetype names? We had already built some custom web stats pages using the old sourcetype and associated field extractions, and now those are all broken due to the new sourcetype name.

0 Karma

DaClyde
Contributor

I think maybe I got it working. I added a sourcetype rename and created field aliases to line up all of the existing fields with what Web Analytics wanted (thanks for the tip about transforms.conf, that put me in the right place).

I'm running the "Generate user sessions" for the first time and data is showing up, so now it is just wait and see. There is a whole year's worth of log data to crunch.

0 Karma

DaClyde
Contributor

Got both lookups generated and build the data model. I had to edit the eventtype to remove the IIS logs (we use IIS but not for our main website), but now I'm getting KV Store issues trying to rebuild it.

However, the good news is that the sourcetype rename and field aliases enabled Web Analytics to recognize my log data, so the basics are good.

0 Karma

DaClyde
Contributor

Well, the "Generate user sessions" worked, but the "Generate pages" came back empty. I think I might have missed a field alias somewhere.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!