How to properly display time, add reports, modify field extractions?
These are the customizations, which I did. I am answering my own question. Hopefully, this will help someone to configure Splunk for BlueCoat plug-in and start a larger discussion on tips and tricks for this application.
Configured time to be diplayed properly.
BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:
-- Added clause:
TZ=US/Eastern
to [default] section in /etc/system/local/props.conf
-- Then marked that each proxy logs in GMT format by modifying /etc/apps/SplunkforBlueCoat/local/props.conf:
[bcoat_proxysg]
TZ=UTC
Fixed incorrect log file parsing for source IP address. The original FIELDS statement in /etc/apps/SplunkforBlueCoat/default/transforms.conf:
FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"
"dvc_ip and "src_ip" fields need to switch places.
I copied transforms.conf file into ../local directory and modified FIELDS clause as following:
FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"
This fixes "Top Clients by IP" report
Defined names for each proxy. In order for "Requests per Proxy" graph to look right, I configured each proxy as a separate input with a different source name. In /etc/apps/SplunkforBlueCoat/local/inputs.conf:
[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com
[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com
[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com
It is imperative for me to be able to correlate between firewall and BlueCoat proxy logs. Documentation for the plug-in requires that BlueCoat is configured to log in bcreportermain_v1 format. The problem with bcreportermain_v1 default format is that it does not log destination IP addresses of web traffic. It places the IP address of BlueCoat proxies, which is not really helpful.
The default bcreportermain_v1 format is:
date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid
Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:
s-ip -- IP address of the appliance on which the client established its connection
What I want to replace it with:
cs-ip -- IP address of the destination of the client's connection
BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:
date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id
and configured BlueCoat proxies to send log events to Splunk in this format.
However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.
Added "Base Report - Destination IP".
Created savedsearches.conf file in /etc/apps/SplunkforBlueCoat/local/ with the following content:
[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count
... View more