topic Re: Filtering events from Docker in Getting Data In

Filtering events from Docker

JenWun — Wed, 05 Jun 2019 10:36:54 GMT

Hello,

we are successfully logging events from Docker hosts via Splunk adapter (HEC) into splunk.
Problems arise when we try to filter some unwanted events.
I tried to send those events to the NULL-Queue

props.conf

[host::<somehost>]
TRANSFORMS-nullQueueFilterSpecificEvents = filter_setNull_4_specific_events

transforms.conf

[filter_setNull_4_specific_events]
REGEX = <someregex>
DEST_KEY = queue
FORMAT = nullQueue

It doesn't work. Still getting all events unfiltered. One possible reason could be, that depending on the HEC endpoint the events may go through a "structured parsing queue" instead of the normal route and that transformations aren't possible.
Is this a possbiel explanation?
How can I detemine what endpoint is used?

Perhaps the log config in the docker-compose.yml can give some clarification:

logging:
  driver: splunk
  options:
    tag: "ct={{.Name}}"
    splunk-token: <sometoken>
    splunk-url: https://<splunkhost-fqdn>:<port>;
    splunk-format: raw
    splunk-insecureskipverify: "true"

The events are not json but unstructured.
Can someone give me a hint or did anyone already successfully filter docker events with transformations in props.conf/transforms.conf.
Thanks in advance!

Regards,
Jens

Re: Filtering events from Docker

outcoldman — Wed, 05 Jun 2019 17:45:26 GMT

Docker Splunk Logging driver sends events to the /event (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent) endpoint. And only /raw (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw) HEC endpoint supports indexing pipeline (see http://dev.splunk.com/view/event-collector/SP-CAAAE6P).
Event endpoint expects that you will control the delivery on your end. Unfortunately Splunk Logging Driver does not have support for /raw endpoint.

With the collectord (Monitoring Docker application https://splunkbase.splunk.com/app/3723/) we support filtering out the events base on the specific regexp pattern, see example https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#example-2-dropping-messages

Re: Filtering events from Docker

JenWun — Thu, 06 Jun 2019 05:13:36 GMT

Just as I thought. Thank you outcoldman!