We'd are trying out the Splunk Add-on for Kubernetes on Splunk Enterprise as we like to keep things as stock as possible (it usually allows for easier installations and upgrades), but it seems we've found a bug in the add-on for the dynamically generated field pod.
The metadata sent to Splunk does not include the field pod. It contains the entries name and generateName.
The name contains the full pod name as shown by kubectl get pods.
The generateName contains the pod name without the containers sha but keeps the dash at the end eg:
{ [-]
metadata: { [-]
creationTimestamp: 2018-11-01T13:58:42Z
generateName: apache-6c4d7cf76-
labels: { [+]
}
name: apache-6c4d7cf76-zndwq
namespace: apache
ownerReferences: [ [+]
]
requestResourceVersion: 3099501
resourceVersion: 2990020
selfLink: /api/v1/namespaces/apache/pods/apache-6c4d7cf76-zndwq
uid: 3d8fc93a-ddde-21f8-9be2-09c1c5b3a6ba
}
spec: { [+]
}
status: { [+]
}
}
It seems there is a bug in how the dynamically generated field pod as it is extracted/generated from the name field but isn't searchable by the full name it in the metadata (nor is the field name) without altering the value of the pod to use wildcards.
Practical example for a pod named apache-6c4d7cf76 with 2 containers named apache-6c4d7cf76-zndwqand apache-6c4d7cf76-a3dfc as outputted by kubect get pods --namespace xyz and shown by the metadata example above then
- If we want to search for the specific container in the pod eg
pod="apache-6c4d7cf76-zndwq" (with or without quotes) then we get zero results.
- If we manually change the search and replace the second dash with a wildcard then it works eg
pod="apache-6c4d7cf76*zndwq"`
- If we search for the container using
pod="apache-6c4d7cf76*"` then it is found but the other containers in the pod are also found and listed negating the search for that particular pod.
Is this a know bug and is there a workaround for this search problem or have we done something wrong in the installation that maks the field show and "work" in the GUI but makes it unsearchable without using wildcards to replace dashes?
