Getting Data In

Logging multiple sources from Docker with HEC: Are multiple sourcetypes possible?

twinspop
Influencer

We have a group using HEC to deliver logs from Docker, but there are many different types of logs in the stream. It appears that any logs they want included in the HEC stream are written to STDOUT, but that means they all show as one sourcetype. Am I missing something? I've got app logs, error logs with stack traces, and web access logs. Each has their own log style as well as time format.

Am I stuck with overriding sourcetypes with a transform, or is there a better way?

Thanks!

lguinn2
Legend

If you can't use multiple tokens, then you can still parse the data stream into multiple sourcetypes. You will need to use transforms on the indexers (or heavy forwarders) where the HEC runs.

0 Karma

lguinn2
Legend

If each type of data is sent to the stream with a different token, then the HEC can easily separate out the different types of data.
This is a good reference: Set up and use HTTP Event Collector
http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

The section "Create an Event Collector token" shows how to create multiple tokens.

0 Karma

twinspop
Influencer

This would be grand if you can start up docker with multiple tokens. AFAIK, you cannot. Did I miss one of the Splunk driver options?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Do they have control over how the data is getting to the HEC? If so, then they can set index, sourcetype, etc. in the HTTP request to the HEC. That would likely be the best way.

If they don't have control, then it would likely be best to use transforms in the indexers.

0 Karma

twinspop
Influencer

They control the docker instance, if that's what you mean. But I don't see a way to classify different sources coming from 1 docker container as different sourcetypes.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...