Getting Data In
Highlighted

Add meta data to events from log correlation

Explorer

Hi!

We're pushing data into splunk over syslog port 1514. Different subsystems report different types of data. One subsystem orchestrates a docker environment and anther subsystem runs the docker containers.

The subsystem that orchestrates the docker environment logs UUIDs and the names of the applications frequently as part of a monitoring procedure but the subsystem that runs the docker containers logs only the UUID of the container. Complicated description?

Anyway! What I want to accomplish is, I have a search that will extract the container name from the orchestrator logs

index="orchestrator" source="/var/log/cluster-orchestrator/current" | rex max_match=10 "(?<json_field>{[^}]+})" | mvexpand json_field | spath input=json_field | rename Instances{}.ContainerName AS ContainerName, Instances{}.UUID AS UUID | dedup ContainerName | where UUID="8f760115-1e1e-44f2-9f20-0d643553d028"| table ContainerName , UUID

This search will return the name of the container together with the UUID in a table for visibility in this case.

The logs from the containers are sent over syslog to splunk, the log messages are visible in splunk with the sourcetype "syslog"

2017-03-28T14:28:52Z 10.0.10.15 container.8f760115-1e1e-44f2-9f20-0d643553d028.stdout[10980]: 2017-03-28 14:28:52.593+0000 [thread-6] INFO LOG MESSAGE EXAMPLE
The events have the source /opt/syslog-ng/logs/2017.03.28/8f760115-1e1e-44f2-9f20-0d643553d028

I want to have access control with roles to define which container logs that are visible and searchable without using the UUID, I want to use the ContainerName that is returned in the first search to control access.

The key for access control is the UUID, but it varies over time when the container is duplicated or upgraded, but the name that is returned from the first search is static.

How can I correlate the name/value/field(s) from a search to a source of other events and build access control ontop of that? I cannot change the way the platform logs and include the name, that would be the obvious choice otherwise.

0 Karma
Highlighted

Re: Add meta data to events from log correlation

Explorer

guess that it's not best practice to respond to your own questions, but I managed to get something working.

the first search that returns the mapping is stored in a lookup table.

<search for mapping> | outputlookup name_uuid_mapping.csv

I have saved this search as a scheduled search that runs every fifth minute to keep it fresh.

I hade added an automatic lookup for the field that represents the "uuid" when the sourcetype is syslog

So when I perform a search in the index where the syslog data is stored, a field with the is automatically.

I'm going to try this out with kvstorecoll instead of using a csv file.

View solution in original post

0 Karma
Highlighted

Re: Add meta data to events from log correlation

SplunkTrust
SplunkTrust

Nothing wrong with answering your own question! You are right it's generally not best practice, but sometimes that's just how it works.

It's OK when that happens. 🙂

Feel free to accept it now. Or what I'd do might be to work on the kvstore way and perhaps in a day or two post your own second answer in here, then pick which one you liked best to mark as Accepted. 🙂

0 Karma