Archive

Could someone provide some tips and tricks to configure Splunk for BlueCoat application?

Path Finder

How to properly display time, add reports, modify field extractions?

These are the customizations, which I did. I am answering my own question. Hopefully, this will help someone to configure Splunk for BlueCoat plug-in and start a larger discussion on tips and tricks for this application.

  1. Configured time to be diplayed properly.

BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:

-- Added clause:

TZ=US/Eastern

to [default] section in /etc/system/local/props.conf

-- Then marked that each proxy logs in GMT format by modifying /etc/apps/SplunkforBlueCoat/local/props.conf:

[bcoat_proxysg]
TZ=UTC

  1. Fixed incorrect log file parsing for source IP address. The original FIELDS statement in /etc/apps/SplunkforBlueCoat/default/transforms.conf:

FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"

"dvc_ip and "src_ip" fields need to switch places.

I copied transforms.conf file into ../local directory and modified FIELDS clause as following:

FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"

This fixes "Top Clients by IP" report

  1. Defined names for each proxy. In order for "Requests per Proxy" graph to look right, I configured each proxy as a separate input with a different source name. In /etc/apps/SplunkforBlueCoat/local/inputs.conf:

[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com

[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com

[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com

  1. It is imperative for me to be able to correlate between firewall and BlueCoat proxy logs. Documentation for the plug-in requires that BlueCoat is configured to log in bcreportermain_v1 format. The problem with bcreportermain_v1 default format is that it does not log destination IP addresses of web traffic. It places the IP address of BlueCoat proxies, which is not really helpful.

The default bcreportermain_v1 format is:

date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid

Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:

s-ip -- IP address of the appliance on which the client established its connection

What I want to replace it with:

cs-ip -- IP address of the destination of the client's connection

BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:

date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id

and configured BlueCoat proxies to send log events to Splunk in this format.

However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.

  1. Added "Base Report - Destination IP".

Created savedsearches.conf file in /etc/apps/SplunkforBlueCoat/local/ with the following content:

[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count

Tags (1)
1 Solution

Path Finder

One more time. The answer. Since web site rendered my original answer improperly:

  1. Configured time to be diplayed properly.

BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:

-- Added clause:

TZ=US/Eastern

to [default] section in /etc/system/local/props.conf

-- Then marked that each proxy logs in GMT format by modifying /etc/apps/SplunkforBlueCoat/local/props.conf:

[bcoat_proxysg]
TZ=UTC

  1. Fixed incorrect log file parsing for source IP address. The original FIELDS statement in /etc/apps/SplunkforBlueCoat/default/transforms.conf:

FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"

"dvc_ip and "src_ip" fields need to switch places.

I copied transforms.conf file into ../local directory and modified FIELDS clause as following:

FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"

This fixes "Top Clients by IP" report

  1. Defined names for each proxy. In order for "Requests per Proxy" graph to look right, I configured each proxy as a separate input with a different source name. In /etc/apps/SplunkforBlueCoat/local/inputs.conf:

[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com

[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com

[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com

  1. It is imperative for me to be able to correlate between firewall and BlueCoat proxy logs. Documentation for the plug-in requires that BlueCoat is configured to log in bcreportermain_v1 format. The problem with bcreportermain_v1 default format is that it does not log destination IP addresses of web traffic. It places the IP address of BlueCoat proxies, which is not really helpful.

The default bcreportermain_v1 format is:

date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid

Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:

s-ip -- IP address of the appliance on which the client established its connection

What I want to replace it with:

cs-ip -- IP address of the destination of the client's connection

BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:

date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id

and configured BlueCoat proxies to send log events to Splunk in this format.

However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.

  1. Added "Base Report - Destination IP".

Created savedsearches.conf file in /etc/apps/SplunkforBlueCoat/local/ with the following content:

[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count

View solution in original post

Path Finder

One more time. The answer. Since web site rendered my original answer improperly:

  1. Configured time to be diplayed properly.

BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:

-- Added clause:

TZ=US/Eastern

to [default] section in /etc/system/local/props.conf

-- Then marked that each proxy logs in GMT format by modifying /etc/apps/SplunkforBlueCoat/local/props.conf:

[bcoat_proxysg]
TZ=UTC

  1. Fixed incorrect log file parsing for source IP address. The original FIELDS statement in /etc/apps/SplunkforBlueCoat/default/transforms.conf:

FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"

"dvc_ip and "src_ip" fields need to switch places.

I copied transforms.conf file into ../local directory and modified FIELDS clause as following:

FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"

This fixes "Top Clients by IP" report

  1. Defined names for each proxy. In order for "Requests per Proxy" graph to look right, I configured each proxy as a separate input with a different source name. In /etc/apps/SplunkforBlueCoat/local/inputs.conf:

[tcp://1.1.1.1:34000]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy1.mycompany.com

[tcp://1.1.2.1:34001]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy2.mycompany.com

[tcp://1.1.3.1:34002]
sourcetype = bcoat_proxysg
connection_host = none
source = proxy3.mycompany.com

  1. It is imperative for me to be able to correlate between firewall and BlueCoat proxy logs. Documentation for the plug-in requires that BlueCoat is configured to log in bcreportermain_v1 format. The problem with bcreportermain_v1 default format is that it does not log destination IP addresses of web traffic. It places the IP address of BlueCoat proxies, which is not really helpful.

The default bcreportermain_v1 format is:

date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter-
result cs-categories cs(Referer) sc-status s-action cs-method
rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery
cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid

Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:

s-ip -- IP address of the appliance on which the client established its connection

What I want to replace it with:

cs-ip -- IP address of the destination of the client's connection

BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:

date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id

and configured BlueCoat proxies to send log events to Splunk in this format.

However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.

  1. Added "Base Report - Destination IP".

Created savedsearches.conf file in /etc/apps/SplunkforBlueCoat/local/ with the following content:

[Base Reports - Destination IP]
action.email.reportServerEnabled = 0
cron_schedule = 0 * * * *
dispatch.earliest_time = -1h
dispatch.latest_time = +0s
displayview = flashtimeline
enableSched = 1
request.ui_dispatch_view = flashtimeline
search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count

View solution in original post

New Member

I had same issue with dvc_ip and src_ip that need to be switched.

But to log the destination IP addresses of web traffic, shouldn't be with the field r-ip instead sc-ip ?

0 Karma

New Member

Hi ageld, as a recommendation for easier reading/use for your users you may want to use the CIM (Common information model) fields for your header. I've used various flavors of field extractions for Blue Coat proxy logs and it's really nice to have a sane and understandable means to search the logs.

Another plus side, coming from a Security perspective is correlation of activity in the proxy to other sourcetypes.

On the configuration of time, setting the Time Zone is helpful, we ran into the pitfall of having to explicitly set the pattern for matching the timestamp too.

Happy hunting!

0 Karma

Path Finder

My answer is above.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!