All Apps and Add-ons

Is is possible to disable or remove Blue Coat comments before they are indexed?

TonyLeeVT
Builder

The current Blue Coat app uses an event type called: bcproxysg_search

The content of the event type is the following:

sourcetype = bluecoat:proxysg:* NOT "#Fields:" NOT "#Version:" NOT "#Software:" NOT "#Date:"

This event type hides comments that are sent from the Blue Coat and do not appear to be used in the app.

Comment Examples:

#Date: ...
#Fields: ...
#Version: ...
#Software: ...

Unfortunately, the comments still count against the Splunk license and potentially misrepresent statistics generated from commands such as tstats. Is it possible to either prevent Blue Coat from sending these fields or have the TA prevent this data from being indexed? Thanks.

0 Karma
1 Solution

TonyLeeVT
Builder

Even though Splunk can adapt, that may not be the most graceful solution. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format (which includes the headers mentioned above). Instead of doing that, you can instruct Blue Coat to send the data in a different format such as key value pair which appears to get rid of those fields.

Have the Blue Coat admins define a custom log format... for example:

 Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

This should get rid of the headers and make ingesting data faster. See the following page for the rest of the solution:
https://answers.splunk.com/answers/376872/blue-coat-field-extractor-namecustom-client-events.html

View solution in original post

0 Karma

TonyLeeVT
Builder

Even though Splunk can adapt, that may not be the most graceful solution. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format (which includes the headers mentioned above). Instead of doing that, you can instruct Blue Coat to send the data in a different format such as key value pair which appears to get rid of those fields.

Have the Blue Coat admins define a custom log format... for example:

 Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

This should get rid of the headers and make ingesting data faster. See the following page for the rest of the solution:
https://answers.splunk.com/answers/376872/blue-coat-field-extractor-namecustom-client-events.html

0 Karma

maciep
Champion

I don't know anything about Blue Coat, so i'm not sure if they can stop being sent. But you should be able to drop them before indexing. Just update/create props/transforms conf in the local folder for your app. This should be done on whatever box does parsing in your infrastructure - typically an indexer or a heavy forwarder.

Something along these lines (regex not tested)

props.conf

[your_bluecoat_sourcetype]
TRANSFORMS-drop_comments = drop_bluecoat_comments

transforms.conf

[drop_bluecoat_comments]
REGEX = (#Fields|#Versions|#Software|#Date)
DEST_KEY = queue
FORMAT = nullQueue

There are a lot of examples on how to do this out here as well.

0 Karma

TonyLeeVT
Builder

Thank you for the solution. This is great information for any appliance in which you are stuck receiving junk data. Fortunately, it appears Blue Coat can be told to send data in a different format. See answer below.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...