Why bluecoat server incoming logs cannot be automatically broken down to fields according to what specified in doc for splunk app for bluecoat?

[kkyeung]

Hi Alls,

I am a user of the splunk app for bluecoat in our prod environment.
However the incoming logs from bluecoat server cannot be automatically broken down to fields according to what specified in doc for splunk app for bluecoat.

Since you are the builder of this app, it would be very nice of you if you could take sometimes to help us to solve the problem.

According to the splunk app for bluecoat doc, the raw log could be broken down into the following fields listed in Box1

Box1

---------------------------------------------------------------------------------------------------------------
date, time, time_taken, c_ip, cs_username, cs_auth_group, 
x_exception_id, filter_result, category,http_referrer, 
sc_status, action, cs_method, http_content_type, cs_uri_scheme, 
cs_host, cs_uri_port, cs_uri_path, cs_uri_query, 
cs_uri_extension, http_user_agent, s_ip, sc_bytes, cs_bytes, 
x_virus_id, x_bluecoat_application_name, x_bluecoat_application_operation
---------------------------------------------------------------------------------------------------------------

However the actual outcomes only show the following fields listed in Box2:

Box2

-----------------------------------------------------------------------------------------------------------------------------------------------
date_hour, date_mday, date_minute, date_month, date_second, date_wday, date_year, date_zone, 
eventype, index, linecount, punct, splunk_server, timeendpos, timestartpos
-----------------------------------------------------------------------------------------------------------------------------------------------

Since we are using the following bluecoat version :

bluecoat and proxysg info: Blue Coat ProxySG model 600-10 with software version SGOS 5.5.10.3

and we have all fields listed in Box1 except “x_bluecoat_application_name” & “x_bluecoat_application_operation”

So we modify the splunk server config according to the following suggestions from the doc

“If you are running an earlier version of SGOS, you can create an entry in the bcoat_proxysg section of your local props.conf file to use the brceportermain_v1_old format. It's the same format but without x_bluecoat_application_name and x_bluecoat_application_operation.”

However, even after the modification. The problem persists without any improvement.

Please help!

Thanks!

Patrick

[kkyeung]

Please find 2 example records below from search function with sourcetype = "bcoat_proxysg" in csv format ( 3 lines only). The first line is the header field names. "url" is a custom field created from learning. The "_raw" contains everything in one field.

raw,_time,app,date_hour,date_mday,date_minute,date_month,date_second,date_wday,date_year,date_ymd,date_zone,eventtype,host,index,linecount,product,punct,source,sourcetype,splunk_server,timeendpos,timestartpos,url,username,vendor
"2014-10-03 09:18:21 639 10.19.49.51 404 TCP_NC_MISS 2850 1069 GET http answers.splunk.com 80 /browserconfig.xml - kkyeung GOVTLAB\Proxy1%20Users DEFAULT_PARENT proxy1.scig.gov.hk text/html;charset=UTF-8 - ""Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; rv:11.0) like Gecko"" OBSERVED ""Technology/Internet"" - 10.19.242.8",2014-10-03T09:18:21.000+0800,,9,3,18,october,21,friday,2014,3/10/2014,local,bcoat_proxysg,bc02.govtlab.hksarg,main,1,,"--::...____../.-\%.../;=--""/.(.",tcp:34002,bcoat_proxysg,LOGSRV,19,0,answers.splunk.com,kkyeung,
"2014-10-03 09:02:15 114425 10.19.49.51 200 TCP_TUNNELED 4255 2020 CONNECT tcp www.google.com 443 / - kkyeung GOVTLAB\Proxy1%20Users DEFAULT_PARENT proxy1.scig.gov.hk - - ""Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko"" OBSERVED ""Search Engines/Portals"" - 10.19.242.8",2014-10-03T09:02:15.000+0800,,9,3,2,october,15,friday,2014,3/10/2014,local,bcoat_proxysg,bc02.govtlab.hksarg,main,1,,"--_::...______../-\%...--""/.(_.;;_",tcp:34002,bcoat_proxysg,LOGSRV,19,0,www.google.com,kkyeung,

[kkyeung]

I cannot attach files and hence the contents of the 2 files are posted below:
file path: .....

\Splunk\etc\apps\SplunkforBlueCoat\appserver\addons\TA-BlueCoat\default

file name: props.conf

[bcoat_log]
MAX_TIMESTAMP_LOOKAHEAD = 19
TRANSFORMS-sourcetype = nullPound, bcoat_proxysg
SHOULD_LINEMERGE = false
TZ = UTC
pulldown_type = true

[bcoat_proxysg]
REPORT-main=bcreportermain_v1
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %T
FIELDALIAS-srcip = c_ip AS src_ip
FIELDALIAS-cs_host = cs_host AS dest_host
FIELDALIAS-protocol = cs_uri_scheme as protocol
FIELDALIAS-dest_port = cs_uri_port as dest_port
FIELDALIAS-src_user = cs_username as src_user
FIELDALIAS-s_ip = s_ip as dvc_ip
FIELDALIAS-http_response = sc_status as http_response
FIELDALIAS-bytes_out = cs_bytes as bytes_out
FIELDALIAS-bytes_in = sc_bytes as bytes_in
KV_MODE = none

File name: transforms.conf

[nullPound]
REGEX = ^\#
DEST_KEY=queue
FORMAT=nullQueue

[bcoat_proxysg]
DEST_KEY = MetaData:Sourcetype
REGEX = .*
FORMAT = sourcetype::bcoat_proxysg

[bcreportermain_v1]
REGEX = (?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+\"(?[^\"]+)\"\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+\"(?[^\"]+)\"\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+\"?(?[^\"]+)\"?\s+\"(?[^\"]+)\"\s+\"(?[^\"]+)\"

[bcreportermain_v1_old]
REGEX = (?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+\"(?[^\"]+)\"\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+\"(?[^\"]+)\"\s+(?[^\s]+)\s+(?[^\s]+)\s+(?[^\s]+)\s+\"?(?[^\"]+)\"?

[MarioM]

can you confirm your events are sourcetyped bcoat_proxysg and additionnally sample of your events as it seems something is not matching

[MarioM]

can you post your props.conf and transforms.conf?