All Apps and Add-ons

Splunk Connect in OpenShift - Configuration with Wildcards for Legacy Logs

andrewcapuano
New Member

Hi Splunk Team,

 

I have deployed Splunk App for Infrastructure and it is working well (essentially it deploys SCK to my server). I have been trying to update my ConfigMap called "sck-rendered-splunk-kubernetes-logging" to setup my source.container.conf file as follows:

 

 

<source>
  @id containers.log
  @type tail
  @label @splunk
  tag tail.containers.*
  path /var/log/containers/*.log, /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/emptydir*/.*/*.log
  exclude_path ["/var/log/containers/fluentd*"]
  pos_file /var/log/splunk-fluentd-containers.log.pos
  tag kubernetes.*
  path_key source
  read_from_head true
  <parse>
    @type regex
    expression /^(?<log>.*)$/ 
    time_key time
    time_type string
    time_format %Y-%m-%dT%H:%M:%SZ
  </parse>
</source>    

 

 

Please take a look at the entry from the Path line above (copied here):   

/var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/emptydir*/.*/*.log

 

We are not correctly receiving legacy app logs in Splunk, which was the purpose of the above entry to the Path config.

 

To give some more info on our file structure, we have empty-dir mounted to legacy logging applications which is mounted to /opt/apps/weblogs on the pod and /var/lib/kubelet/pods on the Node (server) itself.

 

Here's an example of the absolute path to an application 's legacy logs that are hosted on a node:

/var/lib/kubelet/pods/e36c6c23-325b-4e1e-b11e-26ff5abffd17/volumes/kubernetes.io~empty-dir/emptydir-my-test-app-rd-mytestapp/mytestapp

 

Here's a look at the filesystem mounted inside of the pod:

sh-4.2$ cd /opt/apps/weblogs/
sh-4.2$ ls
andrew.log mytestapp
sh-4.2$ ls mytestapp
mytestapp.log  other_directory

 

With our current path (pasted once more below this), we are able to tail the file "andrew.log" in Splunk, but we are not able to tail log files from mytestapp/ and downwards through the filesystem . Can you give some advice on how to update our path in the conf file, or any other place (e.g. the regex) to help us effectively ingest logs from both andrew.log (ie /opt/apps/weblogs) + any .log file that lives in mytestapp or recursively below that (other_directory and beyond)?  

Labels (1)
Tags (3)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...