All Apps and Add-ons

Splunk Add-on for Blue Coat ProxySG: cs_uri field is missing

mikaelbje
Motivator

The field cs_uri is missing which creates problem for the Splunk App for Enterprise Security. I suggest adding the following to the code:

[bluecoat:proxysg:access:syslog]
EVAL-cs_uri = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

And the same for the file input.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Mikael, can you tell us a bit more about the Blue Coat configuration you're working with? I'm seeing cs_uri extracting fine on our test systems. I've added a backlog story for this more robust option, but I'm curious how prevalent that configuration is.

0 Karma

mikaelbje
Motivator

Thanks. I'll check this in a week when I'm back on-site, but I don't believe we've changed anything from the defaults. We're receiving the proxy logs through syslog. The auto_kv_for_bluecoat_v6_5_x transform which is shipped in the app does not contain the cs_uri field extraction, but the auto_kv_for_bluecoat_v5_3_3 transform does, which suggests that one must eval this field for Bluecoat 6.5.x.

The FIELDALIAS-url = cs_uri as url must perhaps also be added after the EVAL. Not sure if order matters.

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

Mikael, thanks for pointing out this problem. We are working with bcreportermain_v1 format, which does not include cs_uri field in SGOS 6.5.

Since evals happen after fieldalias commands, just populate url directly in the eval.

EVAL-url = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

mikaelbje
Motivator

Thanks, that seems to have done the trick. We also had to make the following additions after migrating away from the old Bluecoat app:

local/props.conf

[bluecoat:proxysg:access:syslog]
TRANSFORMS-main=nullPound
TIME_FORMAT = %Y-%m-%d %T
SHOULD_LINEMERGE = false
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = UTC

EVAL-url = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

local/transforms.conf

#[auto_kv_for_bluecoat_v6_5_x]
#REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
#FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 x_exception_id::$13 x_exception_id::$14 sc_filter_result::$15 sc_filter_result::$16 cs_categories::$17 cs_categories::$18 cs_Referer::$19 cs_Referer::$20 sc_status::$21 sc_status::$22 s_action::$23 s_action::$24 cs_method::$25 cs_method::$26 rs_Content_Type::$27 rs_Content_Type::$28 cs_uri_scheme::$29 cs_uri_scheme::$30 cs_host::$31 cs_host::$32 cs_uri_port::$33 cs_uri_port::$34 cs_uri_path::$35 cs_uri_path::$36 cs_uri_query::$37 cs_uri_query::$38 cs_uri_extension::$39 cs_uri_extension::$40 cs_User_Agent::$41 cs_User_Agent::$42 s_ip::$43 s_ip::$44 sc_bytes::$45 sc_bytes::$46 cs_bytes::$47 cs_bytes::$48 x_virus_id::$49 x_virus_id::$50 x_bluecoat_application_name::$51 x_bluecoat_application_name::$52 x_bluecoat_application_operation::$53 x_bluecoat_application_operation::$54

[nullPound]
REGEX = ^\#
DEST_KEY=queue
FORMAT=nullQueue

[auto_kv_for_bluecoat_v6_5_x]
DELIMS=" "
FIELDS="date","time","time_taken","c_ip","cs_username","cs_auth_group","x_exception_id","sc_filter_result","cs_categories","cs_Referer","sc_status","s_action","cs_method","rs_Content_Type","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_uri_extension","cs_User_Agent","s_ip","sc_bytes","cs_bytes","x_virus_id"

We are getting the logs from syslog, not files.

I'm also wondering if the cs_uri_port and cs_uri_scheme fields should be used in the url EVAL, or does that mess it up for ESS?

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Mikael, I think we've got this one fixed in the latest release. http://docs.splunk.com/Documentation/AddOns/latest/BlueCoatProxySG/Releasenotes Please let us know if you still see issues.

Thanks!

0 Karma

mikaelbje
Motivator

Thanks. Version 3.4.1 seems to be working good! I still had to keep the following in local/transforms.conf, probably because we're logging in an unstandard format:

 [auto_kv_for_bluecoat_v6_5_x]
 DELIMS=" "
 FIELDS="date","time","time_taken","c_ip","cs_username","cs_auth_group","x_exception_id","sc_filter_result","cs_categories","cs_Referer","sc_status","s_action","cs_method","rs_Content_Type","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_uri_extension","cs_User_Agent","s_ip","sc_bytes","cs_bytes","x_virus_id"
0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Yeah, I would expect that sort of thing to be required for the foreseeable future when working with W3C logs because they are so wonderfully great.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!