Need Help Routing Subset of Docker Container Logs to Different Index

TheColorBlack — Fri, 12 Mar 2021 23:54:54 GMT

Hello Splunk Community, I could certainly use you help getting myself out of a rather large jam I'm in. I need guidance on how to properly re-route a subset of events generated by one of our applications running within a Docker Container before it is indexed by Splunk.

Our development team updated the logging mechanism of this application to no longer write this subset of events to a flat log file. The subset of events is now being written out to STDOUT/STDERR within the container itself. This means the specific events I need to extract are now being lumped into the same index as every other container processes events running on the same Docker Swarm.

This is causing a problem for me as I am new to administering Splunk and struggling to understand if it's going to be possible to extract this specific subset of events I need while filtering out the noise of every other container process writing events to the same index.

To get data in from Docker, we are using the Splunk Logging Driver for Docker and have replaced "/etc/docker/daemon.json" config file with the following configuration file on all of our Docker Clusters.

{ "metrics-addr": "0.0.0.0:9323", "experimental": true, "log-driver": "splunk", "log-opts": { "splunk-format": "json", "splunk-verify-connection": "false", "splunk-token": "TOKEN", "splunk-url": "URL-TO-SPLUNK", "splunk-insecureskipverify": "true", "tag": "{{.DaemonName}}/{{.Hostname}}/{{.Name}}/{{.ID}}" } }

The HEC Token we have configured for Docker events is as follows

[http://docker] disabled = 0 index = main sourcetype = hec:swarm token = $HEC-Token useACK = 0

Since we use the same HEC Token across all of our Docker Swarms, we have some Props.conf and Transforms.conf stanzas that were configured by an old co-worker to modify the index these events flow to based on source-type.

#props.conf [hec:swarm] SEDCMD-0_hec_tags_rename = s/tag/tags/g TRANSFORMS-0_hec_indexer = docker_change_index TRANSFORMS-1_docker_sourcetyper = docker_sourcetyper TRANSFORMS-9_clean_fields = clean_docker_sourcetypes KV_MODE = json ANNOTATE_PUNCT = false [(::){0}json:platform:*] ANNOTATE_PUNCT = false KV_MODE = json REPORT-vp_app_extract = vp_app_extract REPORT-docker_msghdr_extract = docker_msghdr_extract REPORT-docker_auth_events_extract = docker_auth_events_extract REPORT-docker_kv_extract = inline_kv_extract EVAL-action = case(event == "Authentication failed", "failure", event == "Authentication success", "success", true(), null())

#Transforms.conf # Extract Message Header [docker_msghdr_extract] REGEX = \,\d{3} [^A-Z]+(?<level>\S+)(?:[^\[]+\[){2}\d+m(?<process>[^\\]+) # Extarct Authentication Events [docker_auth_events_extract] REGEX = (?<event>Authentication [^\:]+) # Extract KV pairs [docker_kv_extract] REGEX = (\S+) = [']*([^,"']*) FORMAT = $1::$2 # Extract platform application [vp_app_extract] #REGEX = tags\":\"docker(?:[^_]+)_[^_]+_(?<app>[^\.]+) REGEX = tags\":\"docker\/(?:[^\/]+)\/(?<stack>[a-zA-Z-]+)_(?<app>[^\.]+) ####### SOURCETYPERS ######################### [docker_sourcetyper] DEST_KEY = MetaData:Sourcetype REGEX = tags\":[^\/]+\/(?:[^-]+-){2}([^-]+) FORMAT = sourcetype::json:platform:$1 ####### CHANGE INDEX ############################ [docker_change_index] DEST_KEY = _MetaData:Index REGEX = \"tags\":\"docker/([^-]+)-([^-]+)-([^-]+) FORMAT = $1$2_$3 ####### CLEAN ############################ # Remove '-' from sourcetype [clean_docker_sourcetypes] INGEST_EVAL = sourcetype=replace(sourcetype, "(-|_)", "")

The event subset I'm trying to re-route to a different index contains information in the following format. This is a raw example pulled from Splunk after the event was indexed, and has been sanitized as much as possible.

{"line":"$dockerContainerID,$YYYY-$MM-$DD:$HH:$MM:$SS,$AlphaNumericString,$AlphaString,$AlphaString,$AlphaString,$numericString,$AlphaString,$numericString,,,$DollarAmount,$AlphaString,$AlphaString,$AlphaString,$FloatingPointString,$FloatingPointString,$NumericString,$NumericSring","source":"stderr","tags":"docker/$DockerNodeHostName/$DockerStack_$StackApplication.1.0fj4pexdb3m16giqp1atrfco5/47fb3b6218d5"}

Based on our current configuration could anyone lend a guiding hand on my best path forward to extract this subset of events out and redirect them to their own index, if what I'm attempting is at all possible. I fear that it is not because every container process is writing to the same index, has the same, host, source, and source-type. Based on my understanding of how Props.conf stanzas are defined I'm not sure that I can define any rules that won't effect every event.

[<spec>] * This stanza enables properties for a given <spec>. * A props.conf file can contain multiple stanzas for any number of different <spec>. * Follow this stanza name with any number of the following setting/value pairs, as appropriate for what you want to do. * If you do not set a setting for a given <spec>, the default is used. <spec> can be: 1. <sourcetype>, the source type of an event. 2. host::<host>, where <host> is the host, or host-matching pattern, for an event. 3. source::<source>, where <source> is the source, or source-matching pattern, for an event. 4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule. 5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type classification rule. These are only considered as a last resort before generating a new source type based on the source seen.

Re: Need Help Routing Subset of Docker Container Logs to Different Index

Vardhan — Sun, 14 Mar 2021 04:45:56 GMT

Hi,

Try below props and transforms.

props.conf

[sourcetype]

TRANSFORMS-docker = eventsRoute

Transforms.conf

REGEX= $DockerNodeHostName (If this doesn't work try to give a unique keyword that can differentiate from other events)

DEST_KEY = _MetaData:Index

FORMAT = New index name

If this reply helps you please upvote it.

topic Re: Need Help Routing Subset of Docker Container Logs to Different Index in Getting Data In

Need Help Routing Subset of Docker Container Logs to Different Index

Re: Need Help Routing Subset of Docker Container Logs to Different Index