How to get logs into Splunk from Private Cloud Containers?

[ezajac]

We are planning a Private Cloud implementation of Java applications using a Cloud Vendor. We use Splunk Forwarders on our existing static servers. I have started to research different ways to stream the logs to Splunk once we starting using the Paas Solution. The suggestions I have read are to stream the logs to syslog and use a Data Sink to push the logs to the Splunk Indexers. Then we would need to set up a new UDP listener on each of our Indexers. This seems like a really complicated way to do this because we would have to set up a UDP port for each sourcetype for each index we want to index.

The other thought was to use a custom log4j appender to stream the logs from the container to the Splunk Indexers. This seems like a better way, but it doesn't look like you set up the logs to to separate the logs by index and sourcetype.

Is there an easier way to stream the logs into Splunk like the Splunk Forwarder provides?

[esix_splunk]

Look at the new HTTP Event Collector feature in Splunk v6.3. It designed very much for usecases as you describe it. Most likely your app should support some form of api out for logging in json format. The HTTP Event Collect (HEC) can act as an input listening for these based on api keys.

http://docs.splunk.com/Documentation/Splunk/6.3.1/Data/UsetheHTTPEventCollector

Other options would be to use a HF on premise and collect the Logs/syslog from your cloud service. You can then route that HF's output where you want to. No need to feed the inputs on each of your indexers.