How to properly display time, add reports, modify field extractions?
These are the customizations, which I did. I am answering my own question. Hopefully, this will help someone to configure Splunk for BlueCoat plug-in and start a larger discussion on tips and tricks for this application.
BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:
-- Added clause:
TZ=US/Eastern
to [default] section in
-- Then marked that each proxy logs in GMT format by modifying
[bcoat_proxysg]
TZ=UTC
FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"
"dvc_ip and "src_ip" fields need to switch places.
I copied transforms.conf file into ../local directory and modified FIELDS clause as following:
FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"
This fixes "Top Clients by IP" report
[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com
[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com
[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com
The default bcreportermain_v1 format is:
date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid
Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:
s-ip -- IP address of the appliance on which the client established its connection
What I want to replace it with:
cs-ip -- IP address of the destination of the client's connection
BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:
date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id
and configured BlueCoat proxies to send log events to Splunk in this format.
However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.
Created savedsearches.conf file in
[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count
One more time. The answer. Since web site rendered my original answer improperly:
BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:
-- Added clause:
TZ=US/Eastern
to [default] section in
-- Then marked that each proxy logs in GMT format by modifying
[bcoat_proxysg]
TZ=UTC
FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"
"dvc_ip and "src_ip" fields need to switch places.
I copied transforms.conf file into ../local directory and modified FIELDS clause as following:
FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"
This fixes "Top Clients by IP" report
[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com
[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com
[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com
The default bcreportermain_v1 format is:
date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid
Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:
s-ip -- IP address of the appliance on which the client established its connection
What I want to replace it with:
cs-ip -- IP address of the destination of the client's connection
BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:
date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id
and configured BlueCoat proxies to send log events to Splunk in this format.
However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.
Created savedsearches.conf file in
[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count
One more time. The answer. Since web site rendered my original answer improperly:
BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:
-- Added clause:
TZ=US/Eastern
to [default] section in
-- Then marked that each proxy logs in GMT format by modifying
[bcoat_proxysg]
TZ=UTC
FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"
"dvc_ip and "src_ip" fields need to switch places.
I copied transforms.conf file into ../local directory and modified FIELDS clause as following:
FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"
This fixes "Top Clients by IP" report
[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com
[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com
[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com
The default bcreportermain_v1 format is:
date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid
Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:
s-ip -- IP address of the appliance on which the client established its connection
What I want to replace it with:
cs-ip -- IP address of the destination of the client's connection
BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:
date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id
and configured BlueCoat proxies to send log events to Splunk in this format.
However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.
Created savedsearches.conf file in
[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count
I had same issue with dvc_ip and src_ip that need to be switched.
But to log the destination IP addresses of web traffic, shouldn't be with the field r-ip instead sc-ip ?
Hi ageld, as a recommendation for easier reading/use for your users you may want to use the CIM (Common information model) fields for your header. I've used various flavors of field extractions for Blue Coat proxy logs and it's really nice to have a sane and understandable means to search the logs.
Another plus side, coming from a Security perspective is correlation of activity in the proxy to other sourcetypes.
On the configuration of time, setting the Time Zone is helpful, we ran into the pitfall of having to explicitly set the pattern for matching the timestamp too.
Happy hunting!
My answer is above.