All Apps and Add-ons

How to selectively forward the log files to specific indexes in Splunk?

Is it possible to selectively forward the log files to specific indexes in Splunk.

I want to forward a docker container running 3 services logs to Splunk indexer, the problem is that if I use Docker logging driver, all the data written to STDOUT goes to the same index and data segregation is not possible. Instead of that I've setup forwarder and able to send logs but all are going to the same index, I want to configure splunk forwarder to send specific logs to a specific index.

0 Karma

Communicator

Let me start from the beginning. Running multiple processes in the same container is an anti-pattern. Try to avoid it as much as possible. Kubernetes, for example, have a great solution for your case, where they can deploy two containers in the same Pod and just setup communication between the containers on the same loopback network interface (127.0.0.1), so for the processes, it will look like they are running in the same container. See https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-v... for details.

If you still want to have all three processes in the same container, you have two options to get the logs in different indices:

routing on indexer

If you can identify logs on side of indexer, you can forward logs as you usually do and using transforms.conf on the indexer side route them to specific index, see http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

[nginx_route] 
DEST_KEY = _MetaData:Index 
REGEX = nginx .*
FORMAT = index_nginx

avoiding container logs

Another option. You can create a volume for logs, that you will share between your container and forwarder (Splunk Universal Forwarder) or our collector (https://www.outcoldsolutions.com). And in the configuration define to which index you want to forward these logs. In you container you will need to change how you write logs, instead of stdin, write them to the files.

Communicator

It is not clear, are you running one container with 3 processes or 3 different containers?

0 Karma

One Container with three processes.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!