All Apps and Add-ons

Splunk Add-on for Blue Coat ProxySG: cs_uri field is missing

Motivator

The field cs_uri is missing which creates problem for the Splunk App for Enterprise Security. I suggest adding the following to the code:

[bluecoat:proxysg:access:syslog]
EVAL-cs_uri = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

And the same for the file input.

0 Karma

Splunk Employee
Splunk Employee

Hi Mikael, can you tell us a bit more about the Blue Coat configuration you're working with? I'm seeing cs_uri extracting fine on our test systems. I've added a backlog story for this more robust option, but I'm curious how prevalent that configuration is.

0 Karma

Motivator

Thanks. I'll check this in a week when I'm back on-site, but I don't believe we've changed anything from the defaults. We're receiving the proxy logs through syslog. The autokvforbluecoatv65x transform which is shipped in the app does not contain the cs_uri field extraction, but the autokvforbluecoatv533 transform does, which suggests that one must eval this field for Bluecoat 6.5.x.

The FIELDALIAS-url = cs_uri as url must perhaps also be added after the EVAL. Not sure if order matters.

0 Karma

Splunk Employee
Splunk Employee

Mikael, thanks for pointing out this problem. We are working with bcreportermainv1 format, which does not include csuri field in SGOS 6.5.

Since evals happen after fieldalias commands, just populate url directly in the eval.

EVAL-url = coalesce(csuri, cshost + csuripath + if(csuriquery == "-", "", csuriquery))

Motivator

Thanks, that seems to have done the trick. We also had to make the following additions after migrating away from the old Bluecoat app:

local/props.conf

[bluecoat:proxysg:access:syslog]
TRANSFORMS-main=nullPound
TIME_FORMAT = %Y-%m-%d %T
SHOULD_LINEMERGE = false
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = UTC

EVAL-url = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

local/transforms.conf

#[auto_kv_for_bluecoat_v6_5_x]
#REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
#FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 x_exception_id::$13 x_exception_id::$14 sc_filter_result::$15 sc_filter_result::$16 cs_categories::$17 cs_categories::$18 cs_Referer::$19 cs_Referer::$20 sc_status::$21 sc_status::$22 s_action::$23 s_action::$24 cs_method::$25 cs_method::$26 rs_Content_Type::$27 rs_Content_Type::$28 cs_uri_scheme::$29 cs_uri_scheme::$30 cs_host::$31 cs_host::$32 cs_uri_port::$33 cs_uri_port::$34 cs_uri_path::$35 cs_uri_path::$36 cs_uri_query::$37 cs_uri_query::$38 cs_uri_extension::$39 cs_uri_extension::$40 cs_User_Agent::$41 cs_User_Agent::$42 s_ip::$43 s_ip::$44 sc_bytes::$45 sc_bytes::$46 cs_bytes::$47 cs_bytes::$48 x_virus_id::$49 x_virus_id::$50 x_bluecoat_application_name::$51 x_bluecoat_application_name::$52 x_bluecoat_application_operation::$53 x_bluecoat_application_operation::$54

[nullPound]
REGEX = ^\#
DEST_KEY=queue
FORMAT=nullQueue

[auto_kv_for_bluecoat_v6_5_x]
DELIMS=" "
FIELDS="date","time","time_taken","c_ip","cs_username","cs_auth_group","x_exception_id","sc_filter_result","cs_categories","cs_Referer","sc_status","s_action","cs_method","rs_Content_Type","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_uri_extension","cs_User_Agent","s_ip","sc_bytes","cs_bytes","x_virus_id"

We are getting the logs from syslog, not files.

I'm also wondering if the csuriport and csurischeme fields should be used in the url EVAL, or does that mess it up for ESS?

0 Karma

Splunk Employee
Splunk Employee

Hi Mikael, I think we've got this one fixed in the latest release. http://docs.splunk.com/Documentation/AddOns/latest/BlueCoatProxySG/Releasenotes Please let us know if you still see issues.

Thanks!

0 Karma

Motivator

Thanks. Version 3.4.1 seems to be working good! I still had to keep the following in local/transforms.conf, probably because we're logging in an unstandard format:

 [auto_kv_for_bluecoat_v6_5_x]
 DELIMS=" "
 FIELDS="date","time","time_taken","c_ip","cs_username","cs_auth_group","x_exception_id","sc_filter_result","cs_categories","cs_Referer","sc_status","s_action","cs_method","rs_Content_Type","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_uri_extension","cs_User_Agent","s_ip","sc_bytes","cs_bytes","x_virus_id"
0 Karma

Splunk Employee
Splunk Employee

Yeah, I would expect that sort of thing to be required for the foreseeable future when working with W3C logs because they are so wonderfully great.