topic Tagging Network Logs for Kubernetes Containers in Splunk Search

Tagging Network Logs for Kubernetes Containers

moogmusic — Thu, 19 Nov 2020 13:44:05 GMT

We have VPC flow and firewall logs coming into Splunk from our Kubernetes deployments in GCP. I want to be able to map our containers onto this information so I can track individual container network activity. The problem is that the IP addresses are frequently recycled between different containers. I've created a search which maps out which containers had what IP addresses at which times:

Container Name / Start Time / End Time / IP address

I can use this information to search for the flow/firewall log events for an individual container:

index=networklogs earliest=startTime latest=endTime "IP address"

What I want to do is be able to map the container names onto the networking data so that I can track networking events via the unique container names rather than IP addresses which are continually recycled between different containers as they are created and destroyed. For example, to add the container names into the events in the Network_Traffic.All_Traffic data model. The mapping also needs to be persistent so we can look back over historical data.

One idea is to try and add the container names as a key value pair lookup at ingest but any other ideas on the best way to go about this would be great. Thanks

Re: Tagging Network Logs for Kubernetes Containers

mbjerkeland_spl — Tue, 01 Dec 2020 17:47:39 GMT

Hi,

As the enrichment you want to do is based on very ephemeral data, you may want to do this at Ingest Time instead of at Search Time.

Splunk 8.1 introduced a feature called Ingest-Time Lookups. The idea is to do a lookup based on a given field in the event, get lookup results returned that you then store in an indexed field in the event at index time. Your use case sounds like a proper fit for Ingest-Time Lookups.

A way to achieve what you want is the following:

Scheduled search that stores the results containing "Container Name / Start Time / End Time / IP address" in a CSV file ip_container_mapping.csv.
Set up a INGEST_EVAL in props.conf for your Flow/firewall log sourcetypes using Ingest-Time Lookups (this should be done on the Indexers/Heavy Forwarder)

You may want to synchronize the ip_container_mapping.csv file from the Search Head that generates it to the indexers/or HF to keep your CSV up to date. If you are pulling the flow logs in with a Heavy Forwarder the easiest way to do this would be to let the HF query the Splunk indexers and save the CSV on the HF, then set up the INGEST_EVAL on the HF.

Example for point 2 which will give you an indexed field asset_name being looked up based on the src_ip in the event, returning a column in the CSV file called container_name :

<your_sourcetype_name> INGEST_EVAL = asset_name=json_extract(lookup("ip_container_mapping.csv",json_object("src_ip", src_ip), json_array("container_name")), "container_name")

Re: Tagging Network Logs for Kubernetes Containers

moogmusic — Fri, 11 Dec 2020 10:49:23 GMT

@mbjerkeland_spl - thanks for the suggestion, I'll look into this.