Description of Container Images & Deployment YAML
Splunk UF with logs collection TA: ta-k8s-logs
Splunk UF deployed as a daemonset type to collect docker JSON logs.
Default configuration is to monitor /var/log/containers/*.log. It also contains a disabled input for kubernetes advanced audit, /var/log/kube-apiserver-audit.log. If you have configured your apiserver for advanced auditting, you can enable the input [monitor:///var/log/kube-apiserver-audit.log] in ta-k8s-logs/default/inputs.conf.
New inputs can be added via config map in the yaml or by editing the app and rebuilding the docker container.
This image does not currently provide collection from journald.
Splunk UF with metadata collection TA: ta-k8s-meta
Splunk UF configured as a Deployment type to collect k8s metadata from API Server from within the k8s cluster.
A very simple collection of bash scripts that connects to the API Server (REST API), to provide a glimpse of what k8s API integration can provide for log and metric enrichment.
The polling of the API defaults to 30 seconds but is configurable in ta-k8s-meta/default/inputs.conf and should be set to a suitable value for your environment.
Future interations to explore using 'watch' instead of 'get'
Dependencies: bash & jq are required in the current scripts, but can easily be modified.
Commands/output from any k8s client or kubectl itself could easily be integrated here instead of using curl/jq.
Deployment YAML: k8s-splunk-full-demo.yaml
Deploys a single instance of Splunk Enterprise in the Kubernetes cluster with the k8s demo app pre-configured.
The k8s app currently configures an index called k8s that is hardcoded into the dashboards for simplicity.
Future iteration may update the dashboard (overview.xml), with macros to allow configuration of different indexes.
The app currently contains props/transform configurations:
sourcetype=kubernetes - index docker JSON driver logs in JSON format or by "unwrapping" the JSON from your stderr/stdout logs and using a linebreaker to provide multiline logging support.
sourcetype=k8s:api:* - sourcetypes for the API metadata
The exact configuration for your environment will vary greatly on your logging practices, see k8s/default/props.conf. The defaults to using the "unwrapped" JSON logs.
Run the k8s app and/or ta-k8s-meta on a Splunk instance outside your k8s cluster
If you would like to send your logs/metadata to a Splunk instance running outside the k8s cluster, simply deploy ta-k8s-logs to your cluster and install the k8s app on your external Splunk instance.
You may also use ta-k8s-meta outside the cluster, by retrieving your service account token and kubernetes api url using kubectl or by consulting your kubernetes admin documentation, then updating the ta-k8s-meta shell scripts accordingly.
Contains the Dockerfiles to build the 3 new images used e.g.,
Create a Secret for your Service Account
You will need to create a docker secret to pull the images for your container trusted registry.
kubectl create secret docker-registry yourdockerhubsecret --docker-server=https://index.docker.io/v1/ --docker-username= --docker-password= --docker-email=
Prepare your k8s cluster
You will require a functioning k8s cluster running 1.8+.
We have tested against
Heptio's AWS Quickstart (1.8.4): https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/
Minikube on OSX Sierra (1.8.0): https://github.com/kubernetes/minikube/releases
Configuration and Deployment
Here are the step-by-step instructions to deploy Splunk Enterprise and the 2 Splunk UFs:
Update these parameters in the k8s-splunk-full-demo.yaml yaml files
: secret to allow image pulls from a Docker Hub repo
Splunk Enterprise host - splunkenterprise: the Splunk host name used to by the UF to forward logs and metadata. The two Universal Forwarders (UFs) (1 deployed as a DaemonSet and the other as a Deployment type) require a value for the SPLUNKFORWARDSERVER. If you use the k8s-splunk-full-demo.yaml, the assumption is that you will be sending the data to the instance of Splunk Enterprise created as a Deployment type in the yaml.
Build the 3 required images:
ta-k8s-logs: docker build -t splunk/universalforwarder:7.0.0-monitor-k8s-logs -f ./docker-images/ta-k8s-logs-image/Dockerfile .
ta-k8s-meta: docker build -t splunk/universalforwarder:7.0.0-monitor-k8s-meta -f ./docker-images/ta-k8s-meta-image/Dockerfile .
k8s: docker build -t splunk/splunk:7.0.0-monitor-k8s -f ./docker-images/enterprise-k8s/Dockerfile .
Publish the 3 images to the trusted registery of your choice, e.g., docker push splunk/universalforwarder:7.0.0-monitor-k8s-meta, docker push splunk/universalforwarder:7.0.0-monitor-k8s-logs, docker push splunk/splunk:7.0.0-monitor-k8s.
Update the 3 images names you created in the k8s-splunk-full-demo.yaml. Search for image: and replace the existing images for your own image names (assuming you changed the names).
Deploy Splunk Enterprise and the two Splunk UFs: kubectl create -f k8s-splunk-full-demo.yaml
Create port forwarding to access Splunk Web UI
Run the following command: kubectl get pods
Copy the name for the Splunk Enterprise pod and run the following command: kubectl port-forward 8000:8000
Go to the following web URL using your browser: http://localhost:8000
Note, if you want to deploy the sample app to your own Splunk Enterprise instance / cluster, simply run the following commands to create the SPL file:
gtar -cvf ./app-k8s.tar ./k8s/
mv ./app-k8s.tar.gz ./app-k8s.spl
Inputs.conf in the splunk app?
If it’s not there, maybe it’s in props.conf and transforms.conf