Splunk Docker Driver is great for a start, but it has some limitations and its issues. As an example, it is not resistant to failures, in case of network failures, it will retry several times, but will give up at some point. But if you will install Heavy Weight Forwarder as a side container on every host and use it as a target for your logs - you will reduce the possibility of network issues between the logging driver and Splunk. And Splunk itself knows how to buffer data before forwarding. Using syslog in case if you can use Splunk Logging Driver does not make a lot of sense, because in case of Splunk logging driver you get more Splunk-friendly format, also can add pre-indexed fields from container labels. But there can be situations, where you want to use syslog - if you want to reassemble multiline messages into one. Saying that - still recommend to look on our solution https://www.outcoldsolutions.com, as we deal with all of these problems, and give you application monitoring on top of that. ... View more

Not the exact answer to your question, but want to be sure you have not missed it. Our company offers a solution for Monitoring OpenShift (Application Metrics, Application, and host logs forwarding) https://splunkbase.splunk.com/app/3836/. It is very straightforward and simple installation, takes about 5-10 minutes, https://www.outcoldsolutions.com/docs/monitoring-openshift/ After installing it, you will get application metrics, container logs and host logs forwarded to Splunk. We are also working on the new features, that we are planning to release before the RedHat Summit 2018, including support for Prometheus metrics, system components monitoring, audit logs and more. Feel free to contact us https://www.outcoldsolutions.com/contact/ or me directly denis@outcoldsolutions.com We are RedHat technology partners, and our solution is certified with RedHat (including certified images on their registry). And to answer your questions about our solutions: - Our collector will collector all system logs and container logs, fluentd is not required in the installation. - We are using JSON driver, so in case of failure (network issue between collector and Splunk) - our collector will keep retrying to resent, and all the logs will be buffered in JSON files as they usually do. ... View more

You do not mount /var/run/docker.sock to the container from the host, also few more folders, you can find them in the docs https://hub.docker.com/r/splunk/universalforwarder/ --volume /var/lib/docker/containers:/host/containers:ro \ --volume /var/log:/docker/log:ro \ --volume /var/run/docker.sock:/var/run/docker.sock:ro \ I see that this page also has a docker-compose example, but seems like this example is for general forwarder, not the forwarder with the scripts to collect docker metrics. If you don't mind paid solutions, I would also recommend taking a look at our solution for Monitoring Docker https://www.outcoldsolutions.com. We have a free 30 days trial, instructions on how to install it https://www.outcoldsolutions.com/docs/monitoring-docker/ ... View more

Hi @paimonsoror, my company is working on such solution. The main reason we are going to build an interface between metrics exposed to Prometheus and Splunk - to include better monitoring of Kubernetes and OpenShift clusters in Splunk. Curious if you are using Prometheus for different reasons? Our plan is to have this release in a month. You can find more information about us https://www.outcoldsolutions.com, and my email is denis@outcoldsolutions.com ... View more

If you are willing to try alternative ways to send the logs to Splunk, you can take a look at our solution for monitoring docker ( https://www.outcoldsolutions.com/ ). Our solution provides advanced configuration for joining log lines into one, base on the pattern, how would you expect your line to look like. If you will take a look at our example how to run our collector https://www.outcoldsolutions.com/docs/monitoring-docker/ , the only two lines you will need to add are --env "COLLECTOR__SPLUNK_JSON1=pipe.join::json__patternRegex=^{" \ --env "COLLECTOR__SPLUNK_JSON2=pipe.join::json__matchRegex.docker_container_image=^ubuntu:14\.04$" \ which matches to our configuration ( https://www.outcoldsolutions.com/docs/monitoring-docker/configuration/ ), if you want to configure it with the file, and not using the environment variables [pipe.join::json] #Set match pattern for the fields matchRegex.docker_container_image = ^ubuntu:14\.04$ # All events start from '[<digits>' patternRegex = ^{ That will tell our collector that all messages you expect to start from { for containers running from image matches regex ^ubuntu:14\.04$ , the full command to run docker run -d \ --name collectorfordocker \ --volume /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro \ --volume /proc:/rootfs/proc:ro \ --volume /var/log:/rootfs/var/log:ro \ --volume /var/lib/docker/containers/:/var/lib/docker/containers/:ro \ --volume /var/run/docker.sock:/var/run/docker.sock:ro \ --volume collector_data:/data/ \ --cpus=1 \ --cpu-shares=102 \ --memory=256M \ --restart=always \ --env "COLLECTOR__SPLUNK_URL=output.splunk__url=https://input.splunk.outcold.net/services/collector/event/1.0" \ --env "COLLECTOR__SPLUNK_TOKEN=output.splunk__token=670DD88D-AFB5-4DCE-B0C5-F7AD0A7A2FB8" \ --env "COLLECTOR__EULA=general__acceptEULA=true" \ --env "COLLECTOR__SPLUNK_JSON1=pipe.join::json__patternRegex=^{" \ --env "COLLECTOR__SPLUNK_JSON2=pipe.join::json__matchRegex.docker_container_image=^ubuntu:14\.04$" \ --privileged \ outcoldsolutions/collectorfordocker:3.0.86.180207 After that if I will try to do docker run ubuntu:14.04 bash -c "sleep 5 && echo '{ \"a\": \"b\" } '" That will result json log lines as {"log":"{\n","stream":"stdout","time":"2018-03-07T05:49:45.33335549Z"} {"log":" \"a\": \"b\"\n","stream":"stdout","time":"2018-03-07T05:49:45.333434251Z"} {"log":" }\n","stream":"stdout","time":"2018-03-07T05:49:45.333448344Z"} {"log":"\n","stream":"stdout","time":"2018-03-07T05:49:45.333459838Z"} And with our solution they will be delivered as one event ... View more

@wsanderstii I believe you should be able to configure Distributed Deployment in the way, that you will use Web UI on Deployment Server to configure HTTP Tokens and publish them to Indexing peer. But it can be tricky. It is not recommended and not supported to use Indexer Cluster Peers as Deployment Clients of Deployment Server (you have seen this note in the documentation you have found). But! You can use master cluster as a deployment client for deployment server. See http://docs.splunk.com/Documentation/Splunk/7.0.2/Updating/Aboutdeploymentserver Indexer clusters Do not use deployment server or forwarder management to manage configuration files across peer nodes (indexers) in an indexer cluster. Instead, use the configuration bundle method. You can, however, use the deployment server to distribute updates to the master node, which then uses the configuration bundle method to distribute them to the peer nodes. See "Update common peer configurations" in the Managing Indexers and Clusters of Indexers manual. Look on the https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Deploymentclientconf with repositoryLocation and https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Serverclassconf with targetRepositoryLocation , so I assume you can give it a try and use master-apps as a repository location. This is my assumption based on the documentation, and to be honest I am not sure if that actually works as documented, but you can give it a try. ... View more

@mbarbaro it is very hard and complicated question, which will require some research for sure. But let me try to answer, maybe to help you to get started. Just FYI - I am ex-Splunk employee and have a lot of experience with ElasticSearch lately (for applications type of search, like Full-Text search capabilities), I am also co-founder of the https://www.outcoldsolutions.com, which is specialized on Monitoring Docker/Kubernetes/OpenShift clusters in Splunk. Pricing For ElasticSearch if you want security, alerting, monitoring, reporting, ML and also support! - you want to get X-Pack https://www.elastic.co/products/x-pack . You can google the price (it is not published, but few people whispered it). I believe they charge for per-node. It is very Enterprise level price, so this is why I will not say that ElasticSearch is free, but Splunk is expensive. You will need to do the math based on your personal workflows to understand how much you will end up paying. Purpose Splunk was built for machine data. That was its primary intention at the beginning. It is not a Full-Text Search engine. ElasticSearch was built in mind as a Full-Text Search engine to power your applications. Now they are trying to get to the world of machine data with ElasticSearch + Kibana + Beats/LogStash. Splunk: SPL is the clearer winner here. ElasticSearch: Lucene is not enough, JSON based queries are complicated, Timelion is good for metrics, but not very good for logs. Data Mapping ElasticSearch was built in mind, that you know what you are indexing (because the primary purpose was application full-text search capabilities), so you need to know what you are indexing and how you want to map data from the beginning. If you mapped something wrong or without splitting the data correctly - you will have to reindex it, or using their JSON based queries will be very complicated (and not sure if you can make them as powerful as you can get from SPL in Splunk). Splunk allows you to start just from not knowing the mapping of your machine data at all. You want to setup few small things like source, host, and sourcetype. After that, you can extract the fields in Search time. That is slower than to have them indexed, but you can learn and adjust with time, by applying new indexing patterns while you are growing. Indexes Just a little information about implementation. ElasticSearch stores indexes in segments. Again, because ElasticSearch was built with the intention to drive Full-Text Search - these segments are continually merging, to allow to free disk space for deleted documents. But in the world of machine data - you do not delete documents, so this is just a waste of resources (a lot of IO and CPU). You also will need on your own to define when to split indexes in separate to stop this merging nonsense for old data (see https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html) In case of Splunk. It stores data in segments. There are raw files (compressed data) and indexes (see http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/HowSplunkstoresindexes). There mainly no deletes (you can delete the data, but Splunk does not do anything special to keep dropping deleted data from segments). These indexes were created specifically for storing time-series data without having capabilities to delete the documents or replace them. Security Splunk has ACL and a lot of security capabilities (just basic SSL) out of the box. In case of ELK - you need to pay for X-Pack (you can also find some third party plugins which might get you the same features). I would assume that you will hear more from Splunk about things like Federal Common Criteria Certification (https://www.splunk.com/view/SP-CAAAFQ3). Deployment Every Splunk instance is a ready-to-use-product - Indexing layer, Forwarding, and Web Access. When you need to scale your cluster you will have: Indexing Cluster (nodes to index and store data) + Indexing Cluster Master (one node to rule the Indexing Cluster), Search Head Cluster (where you will have Web access and applications), Universal Forwarders (to forward data to indexers), also some supporting nodes, like Deployment Cluster to manage forwarders, Licensing Master to keep dedicated License Master. Infrastructure can look complicated, but after you learn it - it gets pretty straightforward. In case of ELK - ElasticSearch is a data layer (you can have data nodes and separate master nodes for large deployments), Kibana is a Web Access (Monitoring, Browsing of the data, custom reporting), Logstash and Beats are tools to send data to ElasticSearch. What I don't like about ElasticSearch is how they manage indexes, they need to store in memory always parts of the Indexes, this is why you will hear a lot about OutOfMemoryExceptions in ElasticSearch. Larger your indexes (more data you have) - more RAM you will need. Scaling ElasticSearch to very large clusters (>100TB) can be very complicated. Upgrades Upgrades are pretty straightforward in Splunk clusters. You need to replace nodes one by one in a specific order (you can find manuals). ElasticSearch only from 6.0 finally started to support upgrades without downtime of whole cluster (it is few months old, shipped in end of 2017). Another problem is a lot of breaking changes in Indexes for ElasticSearch. So you will not be able to upgrade to 5.0 from 2.0, because of some ways how you indexed your data, they will require reindexing everything. Not sure if they are going to improve that in 6.0. As far as I know not really because they are going to introduce hard deprecation on types soon (maybe already did), that you cannot store multiple types in the same index. Reporting capabilities Splunk have their App Store (http://splunkbase.splunk.com) with some certified solutions (like ours), they also have a good story for developers to build their applications inside Splunk. ELK relies more on community support. You can find a LOT of very cool stuff working with ELK, but in a lot of times, you will find it abandoned and unsupported. Also, I believe the reporting capabilities are much reacher at that moment in Splunk - tokenization, several searches on the same dashboard, drill down between dashboards, SimpleXML. Long Term Support https://www.splunk.com/en_us/legal/splunk-software-support-policy.html https://www.elastic.co/support/eol It is a real pain with ElasticSearch, that they do not support V1 and V2 anymore, but there are a lot of customers who still use it. Elasticsearch 2.0 was released on 28 OCTOBER 2015, eol of 2.4 is 2018-02-28 (just a little over 2 years!) Splunk 6.0.x was released in Oct. 1st, 2013, there are still not EOL announced yet! Example of Splunk 5.0.x, released on Oct. 30th, 2012, and supported till Nov. 30th 2017 (5 years!). Just keeping up to date with Elasticsearch is a pain. Summary From these points, you might make an assumption that I am selling Splunk. At least this is how I feel after re-reading it. That was not an intention, ElasticSearch is a great product, which I believe was built with the different purpose at the beginning. You can set up very good Logging/Monitoring infrastructure with ElasticSearch, but that will require some knowledgeable people to be around to manage that, plus paying for X-Pack and Support. In case of Splunk - it is more of working out of the box product, and you need to pay for it. If I will need to choose between these two - I would weight how much resources I have, will I have a dedicated team to support it, who is going to support it, etc. For my company, I can choose ELK - because I know a lot about its internals, have good source code knowledge and will go with low "price" - self-supported it for free. Also can get open-source recognition (helping community) and getting help from open source community. I also can choose Splunk, because I know a lot about how to deploy it, and I already know I will get value from it from the first day and will be able to progress pretty quickly. ... View more

@daniel333 you can aways attach to the running container with docker exec -it [container_name] bash and after that check this configurations as you always do by ps aux Find splunkd process and do cd /proc/[splunkd]/ cat limits cat /sys/kernel/mm/transparent_hugepage/enabled ... View more

@wsanderstii, correct Splunk Forwarder does support this. Any reason why you cannot install Balancer in Front? On side note. Just want to share, that I am author of Splunk Logging Driver. I left Splunk year ago and co-founded own company, which is focused on Monitoring Container environments in Splunk. We have our own solution for forwarding Logs and Metrics from Docker to Splunk. We do not have this feature yet, but currently we are working on v3, which will include a lot of features in our applications and collector (Indexer Acknowledgment support, Capacity planning monitoring, Security Monitoring and more). It should not be hard for us to implement balancing on Splunk output as well. If you will be interested to give it a try https://www.outcoldsolutions.com and send me email denis@outcoldsolutions.com ... View more

That is not supported by Splunk Logging Driver. You need to setup your own Reverse Proxy (HAProxy or Nginx will work) in front of HEC to do that. ... View more

Thank you! Glad that it was useful! ... View more

@daniel333 I am co-presenter on the link posted below by felipesewaybricker. I left Splunk year ago and co founded https://www.outcoldsolutions.com, our company is specialized on providing Monitoring Solutions for Containers and consulting around Splunk + Containers. I am author of Splunk images, so I know them from the ground. Feel free to shoot me email, especially if your company is looking for professional consulting on this topic. ... View more

@mazzy89 could you share the code of application you are a reference to? on a side note, have you seen our solution https://www.outcoldsolutions.com/ to send logs and metrics to Splunk? ... View more

Just FYI for other folks, who are looking for solution to monitor Windows Containers - we have finished our solution for Monitoring Windows Containers. We are planning to publish it on SplunkBase and our docs in following days/week. Please send us email contact@outcoldsolutions.com if you want to get access to beta version. ... View more

@splunksundar sorry, I am a little bit confused by your question. You cannot run Linux containers on Windows Containers, and most of the software built for collecting metrics from Linux will not work for Windows, because of the OS differences. And we do not provide currently our containers for Windows, because of that. Depends on demands, probably we will at some point, but not right now. I would suggest to just try to install Splunk UF directly on Windows Host, where you are going to run containers, with Splunk Add-on for Microsoft Windows (https://splunkbase.splunk.com/app/742/), considering that all processes (including the one from containers) will run on the same kernel (if you aren't going to run them as Hyper-V containers) - you should get all the information in Splunk. ... View more

@splunksundar Splunk does not provide Windows images (as far as I know). I am original author of Splunk Docker images (ex-splunker). Our company (https://www.outcoldsolutions.com) provides tools for monitoring Docker, Kubernetes, OpenShift clusters in Splunk, and also consulting for topics related to Splunk + Docker/Kubernetes/OpenShift. I have researched a while ago images for Windows, but because there were no need in them, have not continued that. Feel free to send me an email denis@outcoldsolutions.com, possible I will be able to help you with getting these images for Windows. ... View more

Hi @zielkepham, I have not seen yet ready to use solution for Windows. Our company (https://www.outcoldsolutions.com) is focused on providing monitoring solutions for Docker, OpenShift, Kubernetes environments. We are just starting with Windows support, and have some early prototypes. Feel free to send us an email https://www.outcoldsolutions.com/contact/ so we can talk and understand your needs. ... View more

Hi @raffinyc We have developed an application for Monitoring OpenShift clusters, including forwarding logs (container, host and openshift components) and monitoring stats (CPU, Memory, IO, etc) on level of processes, containers, pods, hosts. You can get the application from Splunkbase https://splunkbase.splunk.com/app/3836/ and find installation instructions on this page https://www.outcoldsolutions.com/docs/monitoring-openshift/ ... View more

We have developed an application for Monitoring OpenShift clusters, including forwarding logs (container, host and openshift components) and monitoring stats (CPU, Memory, IO, etc) on level of processes, containers, pods, hosts. You can get the application from Splunkbase https://splunkbase.splunk.com/app/3836/ and find installation instructions on this page https://www.outcoldsolutions.com/docs/monitoring-openshift/ ... View more

I have updated my answer, thank you for clarification! ... View more

You are looking on something like ... | eval InvalidCompositeMsmtA=floor(InvalidCompositeMsmtA) You can find the documentation about floor http://docs.splunk.com/Documentation/Splunk/7.0.1/SearchReference/MathematicalFunctions#floor.28X.29 ... View more

Seems like somewhere is a mistake, please send a feedback ... View more

Actually docs say that it should be available with 6.4.x http://dev.splunk.com/view/event-collector/SP-CAAAE8Y#basicauth ... View more

Which Splunk version are you using? I believe another two were added later, not on the initial release. ... View more