Getting Data In

Need Help Routing Subset of Docker Container Logs to Different Index

TheColorBlack
Path Finder

Hello Splunk Community, I could certainly use you help getting myself out of a rather large jam I'm in. I need guidance on how to properly re-route a subset of events generated by one of our applications running within a Docker Container before it is indexed by Splunk.

Our development team updated the logging mechanism of this application to no longer write this subset of events to a flat log file. The subset  of events is now being written out to STDOUT/STDERR within the container itself. This means the specific events I need to extract are now being lumped into the same index as every other container processes events running on the same Docker Swarm.

This is causing a problem for me as I am new to administering Splunk and struggling to understand if it's going to be possible to extract this specific subset of events I need while filtering out the noise of every other container process writing events to the same index. 

To get data in from Docker, we are using the Splunk Logging Driver for Docker and have replaced "/etc/docker/daemon.json" config file with the following configuration file on all of our Docker Clusters.

 

 

 

{
  "metrics-addr": "0.0.0.0:9323",
  "experimental": true,
  "log-driver": "splunk",
  "log-opts": {
    "splunk-format": "json",
    "splunk-verify-connection": "false", 
    "splunk-token": "TOKEN",
    "splunk-url": "URL-TO-SPLUNK",
    "splunk-insecureskipverify": "true",
    "tag": "{{.DaemonName}}/{{.Hostname}}/{{.Name}}/{{.ID}}"
  }
}

 

 


The HEC Token we have configured for Docker events is as follows

 

 

[http://docker]
disabled = 0
index = main
sourcetype = hec:swarm
token = $HEC-Token
useACK = 0

 

 

Since we use the same HEC Token across all of our Docker Swarms, we have some Props.conf and Transforms.conf stanzas that were configured by an old co-worker to modify the index these events flow to based on source-type.

 

#props.conf

[hec:swarm]
SEDCMD-0_hec_tags_rename = s/tag/tags/g
TRANSFORMS-0_hec_indexer = docker_change_index
TRANSFORMS-1_docker_sourcetyper = docker_sourcetyper
TRANSFORMS-9_clean_fields = clean_docker_sourcetypes
KV_MODE = json
ANNOTATE_PUNCT = false

[(::){0}json:platform:*]
ANNOTATE_PUNCT = false
KV_MODE = json
REPORT-vp_app_extract = vp_app_extract
REPORT-docker_msghdr_extract = docker_msghdr_extract
REPORT-docker_auth_events_extract = docker_auth_events_extract
REPORT-docker_kv_extract = inline_kv_extract
EVAL-action = case(event == "Authentication failed", "failure", event == "Authentication success", "success", true(), null())

 

 

 

#Transforms.conf 

# Extract Message Header
[docker_msghdr_extract]
REGEX = \,\d{3} [^A-Z]+(?<level>\S+)(?:[^\[]+\[){2}\d+m(?<process>[^\\]+)

# Extarct Authentication Events
[docker_auth_events_extract]
REGEX = (?<event>Authentication [^\:]+)

# Extract KV pairs
[docker_kv_extract]
REGEX = (\S+) = [']*([^,"']*)
FORMAT = $1::$2

# Extract platform application
[vp_app_extract]
#REGEX = tags\":\"docker(?:[^_]+)_[^_]+_(?<app>[^\.]+)
REGEX = tags\":\"docker\/(?:[^\/]+)\/(?<stack>[a-zA-Z-]+)_(?<app>[^\.]+)

####### SOURCETYPERS #########################
[docker_sourcetyper]
DEST_KEY = MetaData:Sourcetype
REGEX = tags\":[^\/]+\/(?:[^-]+-){2}([^-]+)
FORMAT = sourcetype::json:platform:$1

####### CHANGE INDEX ############################
[docker_change_index]
DEST_KEY = _MetaData:Index
REGEX = \"tags\":\"docker/([^-]+)-([^-]+)-([^-]+)
FORMAT = $1$2_$3


####### CLEAN ############################
# Remove '-' from sourcetype
[clean_docker_sourcetypes]
INGEST_EVAL = sourcetype=replace(sourcetype, "(-|_)", "")

 

 

The event subset I'm trying to re-route to a different index contains information in the following format. This is a raw example pulled from Splunk after the event was indexed, and has been sanitized as much as possible.

 

 

 

 

 

{"line":"$dockerContainerID,$YYYY-$MM-$DD:$HH:$MM:$SS,$AlphaNumericString,$AlphaString,$AlphaString,$AlphaString,$numericString,$AlphaString,$numericString,,,$DollarAmount,$AlphaString,$AlphaString,$AlphaString,$FloatingPointString,$FloatingPointString,$NumericString,$NumericSring","source":"stderr","tags":"docker/$DockerNodeHostName/$DockerStack_$StackApplication.1.0fj4pexdb3m16giqp1atrfco5/47fb3b6218d5"}

 

 

Based on our current configuration could anyone lend a guiding hand on my best path forward to extract this subset of events out and redirect them to their own index, if what I'm attempting is at all possible. I fear that it is not because every container process is writing to the same index, has the same, host, source, and source-type. Based on my understanding of how Props.conf stanzas are defined I'm not sure that I can define any rules that won't effect every event.

 

[<spec>]
* This stanza enables properties for a given <spec>.
* A props.conf file can contain multiple stanzas for any number of
  different <spec>.
* Follow this stanza name with any number of the following setting/value
  pairs, as appropriate for what you want to do.
* If you do not set a setting for a given <spec>, the default is used.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host, or host-matching pattern, for an
                 event.
3. source::<source>, where <source> is the source, or source-matching
                     pattern, for an event.
4. rule::<rulename>, where <rulename> is a unique name of a source type
                     classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed
                            source type classification rule.
                            These are only considered as a last resort
                            before generating a new source type based on the
                            source seen.

 

Labels (3)
0 Karma

Vardhan
Contributor

Hi,

Try  below props and transforms.

props.conf

[sourcetype]

TRANSFORMS-docker = eventsRoute

Transforms.conf

REGEX= $DockerNodeHostName (If this doesn't work try to give a unique keyword that can differentiate from other events)

DEST_KEY = _MetaData:Index

FORMAT = New index name

If this reply helps you please upvote it.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...