Is is possible to disable or remove Blue Coat comments before they are indexed?

[TonyLeeVT]

The current Blue Coat app uses an event type called: bcproxysg_search

The content of the event type is the following:

sourcetype = bluecoat:proxysg:* NOT "#Fields:" NOT "#Version:" NOT "#Software:" NOT "#Date:"

This event type hides comments that are sent from the Blue Coat and do not appear to be used in the app.

Comment Examples:

#Date: ...
#Fields: ...
#Version: ...
#Software: ...

Unfortunately, the comments still count against the Splunk license and potentially misrepresent statistics generated from commands such as tstats. Is it possible to either prevent Blue Coat from sending these fields or have the TA prevent this data from being indexed? Thanks.

[TonyLeeVT · Accepted solution]

Even though Splunk can adapt, that may not be the most graceful solution. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format (which includes the headers mentioned above). Instead of doing that, you can instruct Blue Coat to send the data in a different format such as key value pair which appears to get rid of those fields.

Have the Blue Coat admins define a custom log format... for example:

 Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

This should get rid of the headers and make ingesting data faster. See the following page for the rest of the solution:
https://answers.splunk.com/answers/376872/blue-coat-field-extractor-namecustom-client-events.html

[maciep]

I don't know anything about Blue Coat, so i'm not sure if they can stop being sent. But you should be able to drop them before indexing. Just update/create props/transforms conf in the local folder for your app. This should be done on whatever box does parsing in your infrastructure - typically an indexer or a heavy forwarder.

Something along these lines (regex not tested)

props.conf

[your_bluecoat_sourcetype]
TRANSFORMS-drop_comments = drop_bluecoat_comments

transforms.conf

[drop_bluecoat_comments]
REGEX = (#Fields|#Versions|#Software|#Date)
DEST_KEY = queue
FORMAT = nullQueue

There are a lot of examples on how to do this out here as well.

[TonyLeeVT]

Thank you for the solution. This is great information for any appliance in which you are stuck receiving junk data. Fortunately, it appears Blue Coat can be told to send data in a different format. See answer below.