Replying also here, referring to this post: https://community.splunk.com/t5/Getting-Data-In/Why-has-the-index-process-paused-data-flow-How-to-handle-too/m-p/633554/highlight/true#M108467   We ended up with this configuration: In server.conf [queue=indexQueue] maxSize=500MB In indexes.conf [default] throttleCheckPeriod=5 maxConcurrentOptimizes=2 maxRunningProcessGroups=32 processTrackerServiceInterval=0   In this way we have both the benefits: if we restart the cluster we don't have anymore the IndexWriter message during the normal running we don't have the HealthChangeReporter OR PeriodicHealthReporter messages anymore Thanks a lot for your suggestion! ... View more

Hi @hrawat_splunk  We ended up with this configuration: In server.conf [queue=indexQueue] maxSize=500MB In indexes.conf [default] throttleCheckPeriod=5 maxConcurrentOptimizes=2 maxRunningProcessGroups=32 processTrackerServiceInterval=0   In this way we have both the benefits: if we restart the cluster we don't have anymore the IndexWriter message during the normal running we don't have the HealthChangeReporter OR PeriodicHealthReporter messages anymore Thanks a lot for your suggestion! ... View more

Thanks for your reply! We have already followed the guide you provided and also the one listed here below. By the way I opened an Idea called "Virtualization and Performance guide for deploying Splunk" https://ideas.splunk.com/ideas/EID-I-1008   Here below some example of case-study we have followed:   https://core.vmware.com/resource/splunk-vmware-vsan   https://www.dell.com/community/s/vjauj58549/attachments/vjauj58549/storage-and-data-protection-wiki-ch/3903/1/h15604-splunk-enterprise-and-vmax-all-flash.pdf   https://www.delltechnologies.com/resources/en-us/asset/offering-overview-documents/products/storage-2/h15699-splunk-vxrail-sg.pdf   https://www.intel.com/content/dam/www/public/us/en/documents/reference-architectures/high-performance-data-analytics-with-splunk-brief.pdf   ... View more

Hello @hrawat_splunk  Really thanks for your reply. We applied the suggested configuration both on server.conf and indexes.conf So basically from my understanding the aim is to check for index throttling more often (throttleCheckPeriod=5), with only 1 splunk-optimize running over a bucket (maxConcurrentOptimizes=1), spawning several child processes over it (maxRunningProcessGroups=32) and checking every second if any other child process can be launched (processTrackerServiceInterval=0). Therefore the purpose is to concentrate all the optimize resources on a single bucket per time. What we observed is the following: if we put the Cluster Master in maintenance and stop an Indexer we do not see anymore the messages on the Monitoring Console. The Indexing Queue fill anyway to 100% on all remaining Indexers even if we have increase the size to 500MB, the Indexing Rate decrease from 15-20 MB/s to 1MB/s, but in short time (approximately few minutes) the problem is solved The side effect we observed is that since we applied the modification we see in Indexers _internal index the following messages (20-30 events per hour): 02-16-2023 17:15:33.556 +0100 INFO HealthChangeReporter - feature="Index Optimization" indicator="concurrent_optimize_processes_percent" previous_color=green color=yellow due_to_threshold_value=100 measured_value=1 reason="The number of splunk optimize processes is at 100% of the maximum. As a result, the index processor has paused data flow." 02-16-2023 17:15:47.753 +0100 INFO PeriodicHealthReporter - feature="Index Optimization" color=yellow indicator="concurrent_optimize_processes_percent" due_to_threshold_value=100 measured_value=1 reason="The number of splunk optimize processes is at 100% of the maximum. As a result, the index processor has paused data flow." node_type=indicator node_path=splunkd.index_processor.index_optimization.concurrent_optimize_processes_percent And we also see the following "original" message (1-2 events per day): 02-16-2023 14:45:38.658 +0100 INFO IndexWriter [12974 indexerPipe] - The index processor has paused data flow. Too many tsidx files in idx=_internal bucket="/xxxxxxx/xxxx/xxxxxxxxxx/splunk/db/_internaldb/db/hot_v1_1928" , waiting for the splunk-optimize indexing helper to catch up merging them. Ensure reasonable disk space is available, and that I/O write throughput is not compromised.   It seems to me the direction you given us is the correct one to solve the problem, in fact now it is fine if the Cluster takes just few minutes to recover, before was much longer. What we would like to improve is to avoid during the normal running the PeriodicHealthReporter and HealthChangeReporter messages that inform us the indexing has stopped. Do you think that we can increase the maxConcurrentOptimizes value to avoid that? I think in this way we could better balance the "brute force" over more buckets, probably we will loose something when an Indexer is stopped but we gain in normal running.   For reference here the indexes.conf specification: throttleCheckPeriod = <positive integer> * How frequently, in seconds, that splunkd checks for index throttling conditions. * NOTE: Do not change this setting unless a Splunk Support professional asks you to. * The highest legal value is 4294967295. * Default: 15 maxConcurrentOptimizes = <nonnegative integer> * The number of concurrent optimize processes that can run against a hot bucket. * This number should be increased if: * There are always many small tsidx files in the hot bucket. * After rolling, there are many tsidx files in warm or cold buckets. * You must restart splunkd after changing this setting. Reloading the configuration does not suffice. * The highest legal value is 4294967295. * Default: 6 maxRunningProcessGroups = <positive integer> * splunkd runs helper child processes like "splunk-optimize", "recover-metadata", etc. This setting limits how many child processes can run at any given time. * This maximum applies to all of splunkd, not per index. If you have N indexes, there will be at most 'maxRunningProcessGroups' child processes, not N * 'maxRunningProcessGroups' processes. * Must maintain maxRunningProcessGroupsLowPriority < maxRunningProcessGroups * This is an advanced setting; do NOT set unless instructed by Splunk Support. * Highest legal value is 4294967295. * Default: 8 processTrackerServiceInterval = <nonnegative integer> * How often, in seconds, the indexer checks the status of the child OS processes it has launched to see if it can launch new processes for queued requests. * If set to 0, the indexer checks child process status every second. * Highest legal value is 4294967295. * Default: 15   Thanks a lot, Edoardo ... View more

    @hrawat_splunk thanks for the update we have the same exact issue. I see you mentioned it has been fixed with 9.1, do you mean 9.0.1? The latest available is 9.0.3 We are on prem with 9.0.2 and still facing it despite we already put the indicated set-up in indexes.conf see my question here https://community.splunk.com/t5/Splunk-Enterprise/The-index-processor-has-paused-data-flow-How-to-optimize/m-p/629520 Do you suggest increasing maxRunningProcessGroups?   ... View more

Exactly, I confirm in our environment the issue was mainly due to thruput constraint, but we put 25 MB/s instead of unlimited. The only strange thing that didn't allowed us to quickly identify the root cause was that, for the windows events locally generated by the WEC server itself, the Splunk Universal Forwarder had no delay collecting them. The only delay was observed on forwarding with the Splunk Universal Forwarder the events stored by the Windows Event Collector (WEC) coming from the other machines through Windows Event Forwarding (WEF).   limits.conf [thruput] maxKBps = 25600     To understand the thruput limit in your environment you can use this query (stay quite higher than the maximum you observe)   index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) hostname=your_WEC_host | timechart minspan=30s max(eval(tcp_KBps)) as "KB/s", max(tcp_eps) as "Events/s"     ... View more

@andrewtrobec : exactly, you only need a serverclass that allows to deploy the app on 1 HF only. In this way you don't need to deploy it on all the HF and then SSH into your single target HF to modify the local folder. You basically define the correct configs (see for example your_app_name/local/inputs.conf) in the deployment server and then you deploy it just on 1 HF. ... View more

Hi Andrew, Hope you are fine! About your Use Case I think the best solution is to define on your severclass where the app have to be deployed, so basically on one HF only. In serverclass.conf: [serverClass:YOUR_SERVER_CLASS:app:your_app_name] restartSplunkd = 1 stateOnClient = enabled [serverClass:YOUR_SERVER_CLASS] disabled = false whitelist.0 = <clientName> | <IP address> | <hostname> | <instanceId> see: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverclassconf   Best Regards, Edoardo ... View more

Hi @gcusello  Thanks again for your feedback. We already do route and filtering for several sourcetype. Anyway for Firewall logs the scenario we want to achieve  is different. Logs are mostly similar and they contains the following fields: FirewallID, sourceIP, destinationIP, destinationPort, action etc... So every time an host attempt a connection it is logged by the Firewall. What we want to do is to summarize the data as follows before indexing in Splunk FirewallID, sourceIP, destinationIP, destinationPort, action etc... <count> So basically if an host attempt to connect, let's say 100 time, to the same destinationIP and destinationPort and the connection is always dropped instead of having 100 lines of logs we want 1 line of log with the count. It is like doing a group by The only way I see to do that is having a "Summarization tool" before Splunk index the data ... View more

Hi @gcusello  Thanks for your feedback. Yes I know summary index runs on Splunk over indexed data. My aim here is to summarize the data with a tool before they are indexed, to save license and storage. Just posted here to know if someone already did something similar, or have some suggestions on tools that can fit my needs etc... ... View more

Hi @prakash007  You probably don't need to declare the port in uri config, the 443 is the default one for https connection. By the way, even with the correct configuration I posted previously we were getting an HTTP 502 Bad Gateway error. Our Use Case was to export some logs from an on premise Data Center to a third party Splunk installation hosted in AWS. The target was hosted in AWS, with a Load Balancer and a WAF in front but the modification were in charge to the third party admin, and as far as I know they did some modification in the WAF rules to avoid the HTTP 502. ... View more

We have solved the issue with this config. Note: in server.conf better to first test with proxy_rules = * and then restrict server.conf [proxyConfig] http_proxy = http://ip:port https_proxy = http://ip:port proxy_rules = * no_proxy = localhost, 127.0.0.1, ::1 outputs.conf [httpout] httpEventCollectorToken = XXXX-XXXX-XXXX-XXXX-XXXX uri = https://yourdomain.com   We had to put Splunk UF in DEBUG mode and it seems Splunk by itself append the “/services/collector/s2s”, so there is no need to add it in the httpout uri config: 12-21-2021 19:01:38.193 +0100 DEBUG S2SOverHttpOutputProcessor - S2SHttp Json transaction uri=https://yourdomain.com/services/collector/s2s, with sending size: 373645 ... View more

Thanks for the suggestion, I will probably install a Splunk server instance on RHEL 8 and check first with SELinux in permissive mode to see if something is blocked. Anyway do you know if, as of today, the current situation is changed and now Splunk rpm fully support SELinux? ... View more

Same here, the UF was working fine on a Windows server than suddenly started to not ingest the file anymore. To see the error I had to put in debug mode the universal forwarder. The strange thing is that other files are still monitored and forwarded correctly. I have even added followSymlink=false and CHECK_METHOD=modtime without any luck #inputs.conf [monitor://D:\path\to_*\file-*_*.csv] disabled=0 index=myindex sourcetype=mysourcetype crcSalt = <SOURCE> followSymlink=false #props.conf [source::D:\path\to_*\file-*_*.csv] CHECK_METHOD=modtime   ... View more