Help breaking down a log entry

[sov_gwright]

Hi, I'm trying to parse log entries from Oracle Weblogic and no matter how I extract the fields I can't quite get things right. Here is a log entry example :

10.135.188.74 2020-02-26 08:44:59 GET /psc/PORTAL/EMPLOYEE/ERP/c/MANAGE_PURCHASE_ORDERS.PURCHASE_ORDER.GBL 200 30091 "https://hostname.com/psp/PORTAL/EMPLOYEE/ERP/c/MANAGE_PURCHASE_ORDERS.SRM_WORKCENTER.GBL" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"

The extractions didn't work "out of the box" using access_combined (there was no "file" field) so I had to do new extractions. The problem is things like the referer aren't working properly. Here's how I broke down the log entry with regex :

clientip - 10.135.188.74
http_method - GET
file and http_request) - /psc/PORTAL/EMPLOYEE/ERP/c/MANAGE_PURCHASE_ORDERS.PURCHASE_ORDER.GBL
status - 200
http_referer - https://hostname.com/psp/PORTAL/EMPLOYEE/ERP/c/MANAGE_PURCHASE_ORDERS.SRM_WORKCENTER.GBL
user_agent - Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko

Firstly I don't expect file and http_request are supposed to be the same thing, but I couldn't make the regex work any other way without running into one of those regex too complex errors. Secondly even though the referer is coming from itself (this application) it gets the external_referer eventtype which then blows up the dashboards with thousands of referers.

I also have a bunch of other problems like the Audience page showing lots of "Error in map: did not find value for require attribute" but let's take one issue at a time 🙂

Thanks for your help on this

[jbjerke_splunk · Accepted solution]

Hi all

There is now a new version of the app that supports Oracle Weblogic
https://splunkbase.splunk.com/app/2699/#/details

The sourcetype name is configured to be "oracle:weblogic"

Johan

[sov_gwright]

This works great, I did have to add a couple of things - on the Setup/Websites page the query doesn't include the Weblogic source type so no sites show up. I also had to manually add that sourcetype to the eventtypes.conf I'd modified in local because my data is in a different index.

[to4kawa]

| makeresults 
| eval _raw="10.135.188.74 2020-02-26 08:44:59 GET /psc/PORTAL/EMPLOYEE/ERP/c/MANAGE_PURCHASE_ORDERS.PURCHASE_ORDER.GBL 200 30091 \"https://hostname.com/psp/PORTAL/EMPLOYEE/ERP/c/MANAGE_PURCHASE_ORDERS.SRM_WORKCENTER.GBL\" \"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko\""
| rex "(?<clientip>\S+)\s(?<timestamp>(?<date>\S+)\s(?<time>\S+))\s(?<http_method>\S+)\s(?<http_request>\S+)\s(?<status>\d+)\s(?<size>\d+)\s(?<http_referer>\S+)\s(?<user_agent>.*)"
| rex field=http_request "(?<path>.*\/)(?<file>[^¥/]+$)"
| foreach * [ eval <<FIELD>> = trim('<<FIELD>>',"\"")]
| eval _time=strptime(timestamp,"%F %T")

for http_method = CONNECT : | rex field=url "(?<domain>.*?)\:(?<port>\d+$)"
maybe, you can make transforms.conf .

[Richfez]

So you actually asked like 3 questions. But that's OK. 🙂

First, an inline regex that extracts the above:

...| rex "(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+(?<dt_year>\w{4})-(?<dt_mon>\w{2})-(?<dt_day>\w{2})\s+(?<dt_time>\d{2}:\d{2}:\d{2})\s+(?<http_method>\w+)\s+(?<file>[^ ]*)\s+(?<status>\d+)\s+(?<bytes>\d+)\s+(?<http_referer>[^ ]*)\s+(?<user_agent>.*)"

If you want one for http_request, if they're sufficiently different I think it's best to just make two extractions. Each will then match whatever log line that it matches, and the other won't. But I think the above will work in both cases, in which case you can alias "file" in the above to http_request or whatever. Or make the canonical extracted one be http_request and alias it to file if you want.

Second, you mention thousands of referers (gosh, I wish they would have just written that right in the spec!) blowing up the dashboard. I think you have a couple of choices there - either a) it's right, deal with it. 🙂 Or b) its right but annoying, so just remove that search from the dashboard, or c) it's wrong, so figure out how to fix it, or post a new Question here about that.

Lastly, the other issues. I agree, let's take it one step at a time. Build a new question for those so we can keep answers straight. 🙂

[to4kawa]

in-the-oracle-weblogic-app-for-splunk-how-come-i-c

[sov_gwright]

I'm not using the Weblogic Splunk app, I'm using Splunk App for Web Analytics to read Access logs generated by Weblogic. After re-writing my field extractions for the 10th time I think I finally have this working at least somewhat.

[jbjerke_splunk]

Excellent, I was going to see if i can add this to the app as a supported sourcetype. Are you happy to share what you created?