All Apps and Add-ons

Why is the Splunk Bluecoat app not parsing the required data?

Explorer

Hi team ,

We have splunk blueocoat installed , the logs are fetching but we are not getting logs like ,

url
port
protocol
destination ip
user agent string
sourceip

we have installed this Add-on .
Splunk Add-on for Blue Coat ProxySG.

Below is the regex we have in /opt/splunk/etc/apps/ta-bluecoat/default/transforms.conf

[bluecoat_proxy_action_lookup]
filename = bluecoat_proxy_actions.csv
case_sensitive_match = false

Automatic kv

[auto_kv_for_bluecoat_v5_3_3]
REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d+)"|(\d+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
FORMAT = date::$1 date::$2 time::$3 time::$4 c_ip::$5 c_ip::$6 sc_bytes::$7 sc_bytes::$8 time_taken::$9 time_taken::$10 s_action::$11 s_action::$12 sc_status::$13 sc_status::$14 rs_status::$15 rs_status::$16 rs_bytes::$17 rs_bytes::$18 cs_bytes::$19 cs_bytes::$20 cs_auth_type::$21 cs_auth_type::$22 cs_username::$23 cs_username::$24 sc_filter_result::$25 sc_filter_result::$26 cs_method::$27 cs_method::$28 cs_host::$29 cs_host::$30 cs_version::$31 cs_version::$32 sr_bytes::$33 sr_bytes::$34 cs_uri::$35 cs_uri::$36 cs_Referer::$37 cs_Referer::$38 rs_Content_Type::$39 rs_Content_Type::$40 cs_User_Agent::$41 cs_User_Agent::$42 cs_Cookie::$43 cs_Cookie::$44

[auto_kv_for_bluecoat_v6_5_x]
REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 x_exception_id::$13 x_exception_id::$14 sc_filter_result::$15 sc_filter_result::$16 cs_categories::$17 cs_categories::$18 cs_Referer::$19 cs_Referer::$20 sc_status::$21 sc_status::$22 s_action::$23 s_action::$24 cs_method::$25 cs_method::$26 rs_Content_Type::$27 rs_Content_Type::$28 cs_uri_scheme::$29 cs_uri_scheme::$30 cs_host::$31 cs_host::$32 cs_uri_port::$33 cs_uri_port::$34 cs_uri_path::$35 cs_uri_path::$36 cs_uri_query::$37 cs_uri_query::$38 cs_uri_extension::$39 cs_uri_extension::$40 cs_User_Agent::$41 cs_User_Agent::$42 s_ip::$43 s_ip::$44 sc_bytes::$45 sc_bytes::$46 cs_bytes::$47 cs_bytes::$48 x_virus_id::$49 x_virus_id::$50 x_bluecoat_application_name::$51 x_bluecoat_application_name::$52 x_bluecoat_application_operation::$53 x_bluecoat_application_operation::$54

[bluecoat_categories]
SOURCE_KEY = cs_categories
REGEX = (?[^;]+)
MV_ADD = true

[bluecoat_header]
REGEX = ^(#)
FORMAT = bluecoat_header::$1

[TrashHeaders]
REGEX = ^#
DEST_KEY = queue

I had compared with the app in splunk base https://splunkbase.splunk.com/app/2758/#/details, but still, we aren't getting logs. Is there any customization that needs to be done? if so help us where I need to make the changes.

0 Karma

Path Finder

All those aliases, transforms and so on are sourcetype based, can you check if you apply the proper sourcetype to your data ?

0 Karma

SplunkTrust
SplunkTrust

Hi @Kaushikkatta03,

Can you please provide some sample events (Please mask any sensitive data) ?

Additionally can you please let us know how you are receiving this data (using syslog) from Bluecoat Proxy and on which splunk instance (Indexer or Heavy Forwarder) ? And on which splunk instance have you installed this add-on ? And what is sourcetype of Bluecoat proxy data ?

0 Karma

Explorer

Hi Harsmarvania ,

2/8/18
9:10:58.000 AM

Feb 8 09:10:58 xxxx.xxxx.com 2018-02-08 14:10:58 4021 xx.123.456.xxxx - - - OBSERVED "Financial Services" - 200 TCP_TUNNELED CONNECT - tcp xx.xxx.com 443 / - - "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/xx.x.xxxx.xxx Safari/537.36" xxx.xxx.xx.xxxx xxxx xxxx- - - xxx.xxx.xxx.xx

Yes we receiving through syslog on heavy forwarder instance ,
It was installed on heavy forwarder.
It's bluecoat.network.proxy.

0 Karma

SplunkTrust
SplunkTrust

While testing below sample data in my lab environment, splunk is extracting fileds properly.

2018-02-08 14:10:58 4021 12.123.230.254 - - - OBSERVED "Financial Services" - 200 TCP_TUNNELED CONNECT - tcp google.com 443 / - - "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/12.3.4567.890 Safari/537.36" xxx.xxx.xx.xxxx xxxx xxxx- - - xxx.xxx.xxx.xx

So question is, are you getting Feb 8 09:10:58 xxxx.xxxx.com in your raw data ?

Another question is have you installed same add-on on search head ? If not then please install it and try again.

0 Karma

Explorer

hi Harsmarvania ,

This is the recent out put we have got
Feb 12 08:22:44 PRX.Domain.com 2018-02-12 13:22:44 30893 10.151.228.205 - - - OBSERVED "Online Meetings;Chat (IM)/SMS" - 200 TCP_TUNNELED CONNECT - tcp xxxxx.xxx.lync.com 443 / - - - XXX.XX.XX.X xxxx xxxxx - - - xx.xx.xx.xx

We have installed on search Heads , the above data which i have provided is from search head

Thanks

0 Karma

SplunkTrust
SplunkTrust

Is it possible to remove Feb 12 08:22:44 PRX.Domain.com portion of raw data ? It looks like added by syslog server because if you index only 2018-02-12 13:22:44 30893 10.151.228.205 - - - OBSERVED "Online Meetings;Chat (IM)/SMS" - 200 TCP_TUNNELED CONNECT - tcp xxxxx.xxx.lync.com 443 / - - - XXX.XX.XX.X xxxx xxxxx - - - xx.xx.xx.xx then Bluecoat add-on will extract data properly.

0 Karma

New Member

I am assuming that the first Timestamp is being appended by Syslog? If yes, you can disable that using a syslog template and then the logs should parse correctly. Hope this works! 🙂

0 Karma

Explorer

this is from props.conf

[bluecoat:network:proxy]
pulldowntype = true
category = Network & Security
description = Data from Blue Coat ProxySG in W3C ELFF format thru syslog
KV
MODE = none
SHOULDLINEMERGE = false
MAX
DAYS_AGO = 10951

TRANSFORMS-TrashHeaders = TrashHeaders

REPORT-autokvforbluecoatv5 = autokvforbluecoatv533
REPORT-autokvforbluecoatv6 = autokvforbluecoatv65x

REPORT-categories = bluecoatcategories
REPORT-bluecoat
header = bluecoat_header

FIELDALIAS-cookie = csCookie as cookie
FIELDALIAS-duration = time
taken as duration
FIELDALIAS-src = cip as src
FIELDALIAS-src
port = cport as srcport
FIELDALIAS-user = csusername as user
FIELDALIAS-http
referrer = csReferer as httpreferrer
FIELDALIAS-status = scstatus as status
FIELDALIAS-action = s
action as vendoraction
FIELDALIAS-http
method = csmethod as httpmethod
FIELDALIAS-contenttype = rsContentType as httpcontenttype
FIELDALIAS-dest
host = cshost as desthost
FIELDALIAS-destport = sport as destport
FIELDALIAS-user
agent = csUserAgent as httpuseragent
FIELDALIAS-destip = csip as destip
FIELDALIAS-dvc = s
ip as dvc
FIELDALIAS-bytesin = scbytes as bytesin
FIELDALIAS-bytes
out = csbytes as bytesout
FIELDALIAS-uripath = csuripath as uripath
FIELDALIAS-uriquery = csuriquery as uriquery
FIELDALIAS-protocol = csprotocol as protocol
FIELDALIAS-packets
in = cpktsreceived as packetsin
FIELDALIAS-session
id = ssessionid as session_id

EVAL-dest = coalesce(destip, desthost)
EVAL-bytes = bytesin + bytesout
EVAL-url = coalesce(csuri, if(isnull(csurischeme) OR (csurischeme=="-"), "", csurischeme+"://") + cshost + csuripath + if(isnull(csuriquery) OR (csuriquery == "-"), "", csuriquery))
EVAL-product = "ProxySG"
EVAL-vendor = "Blue Coat"
EVAL-vendor_product = "Blue Coat ProxySG"

LOOKUP-vendortrafficaction = bluecoatproxyactionlookup vendoraction OUTPUT action, transport[splunk@xxxxxx default]$

0 Karma