Getting Data In

Logging multiple sources from Docker with HEC: Are multiple sourcetypes possible?

Influencer

We have a group using HEC to deliver logs from Docker, but there are many different types of logs in the stream. It appears that any logs they want included in the HEC stream are written to STDOUT, but that means they all show as one sourcetype. Am I missing something? I've got app logs, error logs with stack traces, and web access logs. Each has their own log style as well as time format.

Am I stuck with overriding sourcetypes with a transform, or is there a better way?

Thanks!

Legend

If you can't use multiple tokens, then you can still parse the data stream into multiple sourcetypes. You will need to use transforms on the indexers (or heavy forwarders) where the HEC runs.

0 Karma

Legend

If each type of data is sent to the stream with a different token, then the HEC can easily separate out the different types of data.
This is a good reference: Set up and use HTTP Event Collector
http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

The section "Create an Event Collector token" shows how to create multiple tokens.

0 Karma

Influencer

This would be grand if you can start up docker with multiple tokens. AFAIK, you cannot. Did I miss one of the Splunk driver options?

0 Karma

SplunkTrust
SplunkTrust

Do they have control over how the data is getting to the HEC? If so, then they can set index, sourcetype, etc. in the HTTP request to the HEC. That would likely be the best way.

If they don't have control, then it would likely be best to use transforms in the indexers.

0 Karma

Influencer

They control the docker instance, if that's what you mean. But I don't see a way to classify different sources coming from 1 docker container as different sourcetypes.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!