Your search checks every minute in the time range, not only the last one. When CPU is back to normal, the minutes from the high period are still inside the time range, and where high_cpu=1 AND consecutive_count >= 5 still returns them, so alert triggers again until they go out of the time range. Keep only the last minute for every host: | mstats avg(_value) as avg_idle WHERE index=linux_metrics metric_name=cpu_metric.pctIdle span=1m BY host | eval cpu_usage = round(100 - avg_idle, 2) | eval high_cpu = if(cpu_usage >= 90, 1, 0) | sort 0 host _time | streamstats count as consecutive_count by host, high_cpu reset_on_change=1 | eventstats max(_time) as last_minute by host | where _time=last_minute AND high_cpu=1 AND consecutive_count >= 5 | table _time host cpu_usage consecutive_count I tested it with makeresults: host that was high for 8 minutes and normal for last 7 minutes is not returned anymore, host that is high for last 6 minutes is returned. Schedule it every minute with time range earliest=-15m@m latest=@m, so the last minute is always complete one. And if you want one alert per incident, not every minute while CPU stays high, use consecutive_count = 5 instead of >= 5, or throttle the alert by host. ... View more

It is not normal. Default inputs.conf has   [monitor://$SPLUNK_HOME/var/log/splunk] index = _internal   without any whitelist or blacklist, so every file in this folder is indexed. I just tried on 10.4.3, created test_create_incident_modalert.log in var/log/splunk and it was in _internal in few seconds. But there is no props for these files, so Splunk picks sourcetype by itself from the file name, for my small file it was test_create_incident_modalert-too_small. So if you search by sourcetype, you will not find them. Try   index=_internal source=*modalert.log   If they are still not there, check on the search head how tailing processor sees this file:   | rest /services/admin/inputstatus/TailingProcessor:FileStatus splunk_server=local | transpose 0 | search column="*modalert*"   It shows status for each file (like "finished reading" or "open file"), so you will see if Splunk even knows about it. ... View more

We (Outcold Solutions https://www.outcoldsolutions.com/) provide a whole complete set of tools for Monitoring Docker (including UCP, ECS), Kubernetes (self-provisioned, EKS, GCP, AKS, Rancher) and OpenShift. It takes about 5 to 10 minutes to set up the application and you will get logs, metrics, alerts, a lot of dashboards for various use cases, like performance monitoring, cluster health monitoring, application monitoring, network monitoring, security, audit, allocations, and more. We have pre-built more than 35 alerts, that are working out of the box and help you to stay ahead of any issue with your clusters. All our applications available on SplunkBase: https://splunkbase.splunk.com/app/3723/ https://splunkbase.splunk.com/app/3836/ https://splunkbase.splunk.com/app/3743/ Our company is Splunk Technology Alliance Partner. You can try it out, it is all automated process, no need to contact us. If you wish a demo, send us an email to [email protected] ... View more

Depends on how you ingest docker logs, with the collectord you can annotate to drop some log lines, see https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#example-2-dropping-messages ... View more

Hello, Application "Monitoring OpenShift - Metrics and Log Forwarding" is distributed under the same license as Collectord (https://www.outcoldsolutions.com/docs/license-agreement/), if you have not obtained the license for our solution, using just the application without the license is a violating the license agreement of the application. If you are looking for free solutions, please take a look at the https://github.com/splunk/splunk-connect-for-kubernetes If you are interested to look at the performance comparison between Collectord and Fluentd - you can look at our blog post https://www.outcoldsolutions.com/blog/2018-11-19-performance-collectord-fluentd-fluentbit/ Thank you for understanding. ... View more

@sathwikr076 to be able to resolve this issue - will be better to send a support ticket to [email protected] The reason for this error is that HEC does not have access to write to this index, or this index does not exist on Splunk. Depending on the version of Collectord you are using, you can configure incorrect index behavior with the configuration, see https://www.outcoldsolutions.com/docs/monitoring-kubernetes/v5/splunk-output/#http-event-collector-incorrect-index-behavior ... View more

Hello @mgherman HTTP Event Collector expects all the values to be prepared in the right format. Fluent-bit does not do that, connect for kubernetes does solve this problem. We can also offer our solution https://www.outcoldsolutions.com/ with the pre-build dashboards and alerts https://splunkbase.splunk.com/app/3743/, it solves the problem of forwarding logs in the expected format. You can also define fields extraction on the source level by specifying the annotations, see https://www.outcoldsolutions.com/docs/monitoring-kubernetes/v5/annotations/#extracting-fields-from-the-container-logs We have also a performance comparison between collectord (our solution), fluent-bit and fluentd https://www.outcoldsolutions.com/blog/2018-11-19-performance-collectord-fluentd-fluentbit/ ... View more

I would highly suggest looking for the alternatives for forwarding Kubernetes logs to Splunk: Open Source Free version of Splunk Connect for Kubernetes - this is a recommended by Splunk option for forwarding logs, built on top of FluentD. Monitoring Kubernetes with Collectord by Outcold Solutions (I represent this company) - this is a proprietary software, takes 10 minutes to set up to forward logs and collect metrics. This is a complete solution for forwarding container, application, host logs, monitor performance of containers and health of the clusters. ... View more

I would highly suggest looking for the alternatives for forwarding Kubernetes logs to Splunk: Open Source Free version of Splunk Connect for Kubernetes - this is a recommended by Splunk option for forwarding logs Monitoring Kubernetes with Collectord by Outcold Solutions (I represent this company) - this is a proprietary software, takes 10 minutes to set up to forward logs and collect metrics. ... View more

@petreb I believe line-merge is only available for the raw endpoint on HTTP Event Collector http://dev.splunk.com/view/event-collector/SP-CAAAE6P#raw, so the events need to be merged before you send them to HEC. I can recommend you to look at our solution for forwarding docker logs to Splunk https://www.outcoldsolutions.com/docs/collectorfordocker/#join-rules, that allows you to predefine advanced patterns how lines need to be merged before we send them to HEC https://www.outcoldsolutions.com/blog/2018-03-11-forwarding-pretty-json-logs-to-splunk/ ... View more

From my experience, I needed to export several GB of data, and the best performance I got is from dump command. You need to do few things: Change the job Lifetime to 7 days. Move job to the background, so it will not depend on UI. With CLI my export was running for more than 24 hours, with dump I got my data out in 3-4 hours. ... View more

I have seen that the limit has been increased for all logging drivers. But not sure, why do you see different behavior on different types of logging formats, as this is handled by the driver itself. As I have mentioned in my previous reply - you can try our solution, that does not have this problem. ... View more

Our company provides a complete solution for Monitoring Kubernetes in Splunk https://www.outcoldsolutions.com/. We are Splunk Alliance Technology partners. You can download our certified application at https://splunkbase.splunk.com/app/3743/ and follow instructions on how to get our kubernetes native collector to be installed in your cluster https://www.outcoldsolutions.com/docs/monitoring-kubernetes/ Our solution forwards application logs, host logs, metrics from the hosts, pods, containers and the processes. We are working on the next version of our solution, that will include monitoring capabilities for the control plane, including etcd cluster, controller manager, kubelets, scheduler, and providing capabilities to explore audit logs. ... View more

Curious, have you seen our solutions for monitoring Docker, Kubernetes and OpenShift clusters? https://www.outcoldsolutions.com/ We also have a blog post explaining how to set up our solution on Tectonic https://www.outcoldsolutions.com/blog/2018-03-21-monitoring-tectonic-in-splunk/ ... View more

Let me start from the beginning. Running multiple processes in the same container is an anti-pattern. Try to avoid it as much as possible. Kubernetes, for example, have a great solution for your case, where they can deploy two containers in the same Pod and just setup communication between the containers on the same loopback network interface (127.0.0.1), so for the processes, it will look like they are running in the same container. See https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/ for details. If you still want to have all three processes in the same container, you have two options to get the logs in different indices: routing on indexer If you can identify logs on side of indexer, you can forward logs as you usually do and using transforms.conf on the indexer side route them to specific index, see http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf [nginx_route] DEST_KEY = _MetaData:Index REGEX = nginx .* FORMAT = index_nginx avoiding container logs Another option. You can create a volume for logs, that you will share between your container and forwarder (Splunk Universal Forwarder) or our collector (https://www.outcoldsolutions.com). And in the configuration define to which index you want to forward these logs. In you container you will need to change how you write logs, instead of stdin, write them to the files. ... View more

It is not clear, are you running one container with 3 processes or 3 different containers? ... View more

It is a hard question, as it depends on how you setup the Supervisor. Let me start from the beginning. Running multiple processes in the same container is an anti-pattern. Try to avoid it as much as possible. Kubernetes, for example, have a great solution for your case, where they can deploy two containers in the same Pod and just setup communication between the containers on the same loopback network interface (127.0.0.1), so for the processes, it will look like they are running in the same container. See https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/ for details. Unfortunately, Docker does not have this out of the box, and configuring the same can be problematic. There are two options. You can redirect stdout and stderr from the processes to the supervisord stdout/stderr (took an example from https://github.com/CachetHQ/Docker/blob/master/conf/supervisord.conf#L18-L25) [program:nginx] command=/usr/sbin/nginx stdout_events_enabled=true stderr_events_enabled=true stdout_logfile_maxbytes=0 stderr_logfile_maxbytes=0 stdout_logfile=/dev/stdout stderr_logfile=/dev/stderr In that case, all the logs will end up in container stdout/stderr. You can use Docker Logger Drivers to send these logs to Splunk. The problem with this approach - all the logs, from supervisord, from nginx and all other processes, will end up in container stdout/stderr, it could be hard to distinguish them. Another approach is to have a data volume for the application logs. Attach the same volume to the Splunk Universal Forwarder or another process (like our collector https://www.outcoldsolutions.com) and forward these logs as you usually do. Don't forget to set the log rotation. ... View more

You can use HTTP Event Collector: Instance dashboard in Monitoring Console to find which tokens were used (received data, etc). I don't know of any automatic way to delete these tokens. But you can just open the inputs.conf file and remove these tokens from here and reload input settings after that https://yoursplunk.com/en-US/debug/refresh (if that does not work, you will need to restart the splunk). Another option is to use Http Event Collector CLI http://dev.splunk.com/view/event-collector/SP-CAAAE7D ... View more

Having multiple indexer will help with the indexer availability, but will not solve the networking problem. You can also have Heavy Weight Forwarders installed on the same node, so you will not have networking issues anymore. And that forwarders will send data to indexers, when they are available. The hang you are experiencing is unexpected, and I assume that it is possible that Splunk Logging Driver does not set the read timeout, and the connection is just getting disconnected from one end but does not close it on Splunk Logging Driver, so it indefinitely waits for a response. It does not seem like Splunk Logging Driver sets the ReadTimeout to the http.Client https://github.com/moby/moby/blob/master/daemon/logger/splunk/splunk.go#L223, so you can send a PR to add a timeout https://golang.org/pkg/net/http/#Client That should solve this problem partially. But again, I will suggest you take a look on our solution, as our log forwarding does not depend on Splunk log driver, you will write the logs in JSON, our collector tails JSON logs and forwards them to Splunk. We have a free trial for 30 days. Give a try, send us an email to [email protected] to learn more, we can schedule a call and discuss all the issues you experience. ... View more