Why is the Splunk Bluecoat app not parsing the required data?

[Kaushikkatta03]

Hi team ,

We have splunk blueocoat installed , the logs are fetching but we are not getting logs like ,

url
port
protocol
destination ip
user agent string
sourceip

we have installed this Add-on .
Splunk Add-on for Blue Coat ProxySG.

Below is the regex we have in /opt/splunk/etc/apps/ta-bluecoat/default/transforms.conf

[bluecoat_proxy_action_lookup]
filename = bluecoat_proxy_actions.csv
case_sensitive_match = false

Automatic kv

[auto_kv_for_bluecoat_v5_3_3]
REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d+)"|(\d+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
FORMAT = date::$1 date::$2 time::$3 time::$4 c_ip::$5 c_ip::$6 sc_bytes::$7 sc_bytes::$8 time_taken::$9 time_taken::$10 s_action::$11 s_action::$12 sc_status::$13 sc_status::$14 rs_status::$15 rs_status::$16 rs_bytes::$17 rs_bytes::$18 cs_bytes::$19 cs_bytes::$20 cs_auth_type::$21 cs_auth_type::$22 cs_username::$23 cs_username::$24 sc_filter_result::$25 sc_filter_result::$26 cs_method::$27 cs_method::$28 cs_host::$29 cs_host::$30 cs_version::$31 cs_version::$32 sr_bytes::$33 sr_bytes::$34 cs_uri::$35 cs_uri::$36 cs_Referer::$37 cs_Referer::$38 rs_Content_Type::$39 rs_Content_Type::$40 cs_User_Agent::$41 cs_User_Agent::$42 cs_Cookie::$43 cs_Cookie::$44

[auto_kv_for_bluecoat_v6_5_x]
REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 x_exception_id::$13 x_exception_id::$14 sc_filter_result::$15 sc_filter_result::$16 cs_categories::$17 cs_categories::$18 cs_Referer::$19 cs_Referer::$20 sc_status::$21 sc_status::$22 s_action::$23 s_action::$24 cs_method::$25 cs_method::$26 rs_Content_Type::$27 rs_Content_Type::$28 cs_uri_scheme::$29 cs_uri_scheme::$30 cs_host::$31 cs_host::$32 cs_uri_port::$33 cs_uri_port::$34 cs_uri_path::$35 cs_uri_path::$36 cs_uri_query::$37 cs_uri_query::$38 cs_uri_extension::$39 cs_uri_extension::$40 cs_User_Agent::$41 cs_User_Agent::$42 s_ip::$43 s_ip::$44 sc_bytes::$45 sc_bytes::$46 cs_bytes::$47 cs_bytes::$48 x_virus_id::$49 x_virus_id::$50 x_bluecoat_application_name::$51 x_bluecoat_application_name::$52 x_bluecoat_application_operation::$53 x_bluecoat_application_operation::$54

[bluecoat_categories]
SOURCE_KEY = cs_categories
REGEX = (?[^;]+)
MV_ADD = true

[bluecoat_header]
REGEX = ^(#)
FORMAT = bluecoat_header::$1

[TrashHeaders]
REGEX = ^#
DEST_KEY = queue

I had compared with the app in splunk base https://splunkbase.splunk.com/app/2758/#/details, but still, we aren't getting logs. Is there any customization that needs to be done? if so help us where I need to make the changes.

[isabel_ycourbe]

All those aliases, transforms and so on are sourcetype based, can you check if you apply the proper sourcetype to your data ?

[harsmarvania57]

Hi @Kaushikkatta03,

Can you please provide some sample events (Please mask any sensitive data) ?

Additionally can you please let us know how you are receiving this data (using syslog) from Bluecoat Proxy and on which splunk instance (Indexer or Heavy Forwarder) ? And on which splunk instance have you installed this add-on ? And what is sourcetype of Bluecoat proxy data ?

[Kaushikkatta03]

Hi Harsmarvania ,

2/8/18
9:10:58.000 AM

Feb 8 09:10:58 xxxx.xxxx.com 2018-02-08 14:10:58 4021 xx.123.456.xxxx - - - OBSERVED "Financial Services" - 200 TCP_TUNNELED CONNECT - tcp xx.xxx.com 443 / - - "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/xx.x.xxxx.xxx Safari/537.36" xxx.xxx.xx.xxxx xxxx xxxx- - - xxx.xxx.xxx.xx

Yes we receiving through syslog on heavy forwarder instance ,
It was installed on heavy forwarder.
It's bluecoat.network.proxy.

[harsmarvania57]

While testing below sample data in my lab environment, splunk is extracting fileds properly.

2018-02-08 14:10:58 4021 12.123.230.254 - - - OBSERVED "Financial Services" - 200 TCP_TUNNELED CONNECT - tcp google.com 443 / - - "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/12.3.4567.890 Safari/537.36" xxx.xxx.xx.xxxx xxxx xxxx- - - xxx.xxx.xxx.xx

So question is, are you getting Feb 8 09:10:58 xxxx.xxxx.com in your raw data ?

Another question is have you installed same add-on on search head ? If not then please install it and try again.

[Kaushikkatta03]

hi Harsmarvania ,

This is the recent out put we have got
Feb 12 08:22:44 PRX.Domain.com 2018-02-12 13:22:44 30893 10.151.228.205 - - - OBSERVED "Online Meetings;Chat (IM)/SMS" - 200 TCP_TUNNELED CONNECT - tcp xxxxx.xxx.lync.com 443 / - - - XXX.XX.XX.X xxxx xxxxx - - - xx.xx.xx.xx

We have installed on search Heads , the above data which i have provided is from search head

Thanks

[harsmarvania57]

Is it possible to remove Feb 12 08:22:44 PRX.Domain.com portion of raw data ? It looks like added by syslog server because if you index only 2018-02-12 13:22:44 30893 10.151.228.205 - - - OBSERVED "Online Meetings;Chat (IM)/SMS" - 200 TCP_TUNNELED CONNECT - tcp xxxxx.xxx.lync.com 443 / - - - XXX.XX.XX.X xxxx xxxxx - - - xx.xx.xx.xx then Bluecoat add-on will extract data properly.

[jeet3007]

I am assuming that the first Timestamp is being appended by Syslog? If yes, you can disable that using a syslog template and then the logs should parse correctly. Hope this works! 🙂

[Kaushikkatta03]

this is from props.conf

[bluecoat:network:proxy]
pulldown_type = true
category = Network & Security
description = Data from Blue Coat ProxySG in W3C ELFF format thru syslog
KV_MODE = none
SHOULD_LINEMERGE = false
MAX_DAYS_AGO = 10951

TRANSFORMS-TrashHeaders = TrashHeaders

REPORT-auto_kv_for_bluecoat_v5 = auto_kv_for_bluecoat_v5_3_3
REPORT-auto_kv_for_bluecoat_v6 = auto_kv_for_bluecoat_v6_5_x

REPORT-categories = bluecoat_categories
REPORT-bluecoat_header = bluecoat_header

FIELDALIAS-cookie = cs_Cookie as cookie
FIELDALIAS-duration = time_taken as duration
FIELDALIAS-src = c_ip as src
FIELDALIAS-src_port = c_port as src_port
FIELDALIAS-user = cs_username as user
FIELDALIAS-http_referrer = cs_Referer as http_referrer
FIELDALIAS-status = sc_status as status
FIELDALIAS-action = s_action as vendor_action
FIELDALIAS-http_method = cs_method as http_method
FIELDALIAS-content_type = rs_Content_Type as http_content_type
FIELDALIAS-dest_host = cs_host as dest_host
FIELDALIAS-dest_port = s_port as dest_port
FIELDALIAS-user_agent = cs_User_Agent as http_user_agent
FIELDALIAS-dest_ip = cs_ip as dest_ip
FIELDALIAS-dvc = s_ip as dvc
FIELDALIAS-bytes_in = sc_bytes as bytes_in
FIELDALIAS-bytes_out = cs_bytes as bytes_out
FIELDALIAS-uri_path = cs_uri_path as uri_path
FIELDALIAS-uri_query = cs_uri_query as uri_query
FIELDALIAS-protocol = cs_protocol as protocol
FIELDALIAS-packets_in = c_pkts_received as packets_in
FIELDALIAS-session_id = s_session_id as session_id

EVAL-dest = coalesce(dest_ip, dest_host)
EVAL-bytes = bytes_in + bytes_out
EVAL-url = coalesce(cs_uri, if(isnull(cs_uri_scheme) OR (cs_uri_scheme=="-"), "", cs_uri_scheme+"://") + cs_host + cs_uri_path + if(isnull(cs_uri_query) OR (cs_uri_query == "-"), "", cs_uri_query))
EVAL-product = "ProxySG"
EVAL-vendor = "Blue Coat"
EVAL-vendor_product = "Blue Coat ProxySG"

LOOKUP-vendor_traffic_action = bluecoat_proxy_action_lookup vendor_action OUTPUT action, transport[splunk@xxxxxx default]$