Hi @gcusello , Yes, this parameter maxKBps = 0 has been configured from the start - even still we have observed delays - queues as I shared in an earlier post.  So am not sure where else to look into to resolve this issue - only area I was thinking was the delay of the ACK from Indexers to the Syslog Server (UF) as useACK=true was configured.  Kind regards   ... View more

Hi @gcusello , I have been trying to figure out how to tune the servers and see if there are any other challenges. The findings have been that even though useACK=true has been set in outputs.conf even still logs are being missed in Splunk - this was identified when it was identified that Splunk did not receive logs for a specific day and after checking on the Syslog Server I found there were logs for that day but they were not received in Splunk.  When you said to tune the hosts - there is only one host (Syslog Server - UF) which is sending the logs to Splunk and the settings of this host are as follows, limits.conf [thruput] maxKBps = 0 outputs.conf [tcpout] forceTimebasedAutoLB = true maxQueueSize = 7MB useACK = true The host has about 64 GB RAM. Can you advise if possible on what settings to tune in the host limits.conf etc and about what could be the reason logs are missing even though useACK=true? (Can it be that the delay is so long in receiving the ACK from the Indexers that the Queue is filled up and older data is removed from Queue before being sent to the indexers?) Kind regards ... View more

I've accepted this as the solution since distributing the load would definitely help resolve the issue but since I was looking for a workaround to avoid that - had kept this open and kind of forgot about this thread later on - but for anyone else having the same issue - we did end up dividing the load on two Syslog Servers and the issue has accordingly reduced - there is still some delay some times but that is probably due to still the load being too much for even two Syslog Servers sometimes during peak. ... View more

Hi @gcusello , I was able to finally run the search you provided, can you help me understand the snippet from the output pasted below,           It seems that there is a lot of queues in the parsing queue - fill percentage is on average around 96% approx.   Kind regards ... View more

Hi @gcusello  Thank you for the answer and the SPL Query - it is much appreciated. I have one more question - related to useACK = true [tcpout] defaultGroup = groupA useACK = true [tcpout:groupA] server = x.x.x.x:9999 sslVersions = TLS1.2 [tcpout:groupB] server = x.x.x.x:9997, x.x.x.x:9997, x.x.x.x:9997, x.x.x.x:9997   Can I set useACK = false for groupB? I feel maybe because the groupB indexers are under a huge load and in a different network therefore the acknowledgement would take time or be delayed that's why the data to groupB is not being sent in real-time and has delays because it is waiting for the acknowledgement. What would you recommend regarding this configuration - setting useACK = false for groupB? Thank you. ... View more

Hi @gcusello , Thank you for confirming about the approach. Yes, there is an issue that right after I configured the _TCP_ROUTING for the specific log sources (log paths) the groupB did not receive any events from the log sources even though the log files on the Syslog Server were being updated - the events / logs were received in groupB after some time. Further, for example, I have 50 different hosts sending logs in one log path on the Syslog Server which is being sent to both groupA and groupB - the groupB received logs from only 5 even though all 50 log files were being updated. Summary of Issues, 1. How to identify the reason for the delay in logs being received in groupB (configured via _TCP_ROUTING) 2. Is there a chance for logs to be missing? How to identify if this is the case? Steps taken to identify whether logs are being sent, I've tried to read the metrics.log file on the UF to see if any data is being sent to the groupB which showed that right after the configuration was applied no logs were being sent but after some time some logs were being sent, sample snippet pasted below, tail /opt/splunkforwarder/var/log/splunk/metrics.log | grep groupB 09-01-2022 13:59:32.194 +0500 INFO Metrics - group=queue, name=tcpout_groupB, max_size=7340032, current_size=2322132, largest_size=2322132, smallest_size=2322132 09-01-2022 13:59:32.194 +0500 INFO Metrics - group=tcpout_connections, name=groupB:x.x.x.x:9997:0, sourcePort=8089, destIp=x.x.x.x, destPort=9997, _tcp_Bps=3967.13, _tcp_KBps=3.87, _tcp_avg_thruput=3.87, _tcp_Kprocessed=116, _tcp_eps=0.10, kb=116.22, max_ackq_size=22020096, current_ackq_size=118360 I did find these from the health.log file pasted below which signify the issue might be with load - however I'm not sure about this one, if you can confirm about this, tail /opt/splunkforwarder/var/log/splunk/health.log 09-01-2022 13:46:35.112 +0500 INFO HealthChangeReporter - feature="TailReader-0" indicator="data_out_rate" previous_color=green color=yellow due_to_threshold_value=1 measured_value=1 reason="The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data." 09-01-2022 13:46:40.112 +0500 INFO HealthChangeReporter - feature="BatchReader-0" indicator="data_out_rate" previous_color=yellow color=red due_to_threshold_value=2 measured_value=2 reason="The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data." 09-01-2022 13:56:34.898 +0500 INFO HealthChangeReporter - feature="TailReader-0" indicator="data_out_rate" previous_color=red color=green measured_value=0 ... View more

Apologies for not clarifying it - all data is being sent to one indexer group however some data is being sent to both the indexer groups so that's why tried to use _TCP_ROUTING in inputs.conf The default group is groupA (one indexer) so all data is being sent to this group however for specific data I've added _TCP_ROUTING = groupA, groupB so that it is also sent to groupB (multiple indexers) as well.   groupA is a different Splunk Deployment and groupB is a different Splunk Deployment so I am sending the data from this Syslog Server to two different Splunk Deployments. ... View more

Hi @gcusello , I've configured it as per the article you mentioned but I've used just the outputs.conf and inputs.conf because the article mentions the props.conf and transforms.conf method for heavy forwarders. Since I have a UF that's why I've just defined the group in outputs.conf and then the specific monitor stanza has been updated with _TCP_ROUTING to forward to both groups. Is this the correct approach? This configuration has been performed but the issue is that not all the logs are not being received in the second group of indexers e.g. is there any way I can check from the UF level what data it is sending to which group and if there are any errors? Both the indexers groups are separate Splunk Deployments - I have access to the UF to make the changes. ... View more

An update, This issue was fixed I believe by using newer version (8.1.1) of the Splunk Agent and the Apps were successfully downloaded from the Splunk DS.  ... View more

Hi @richgalloway , An update, we have been trying out different options such as increasing the parsingQueue from10 MB to about 100 MB as per the conversations with support. However, the delay is still there.  Coming to your two options,  (a) running multiple syslog servers with UF to distribute load - that is being considered however might take time so I'm thinking of trying out your second option (b) running multiple (2) UF on a single syslog server with each UF monitoring different/separate directories.  What I'm most curious and apprehensive about with the option (b) is if another UF is installed on the syslog server and tries to monitor a separate directory that was previously being monitored by the existing UF then how do we avoid re-indexing/duplication of events? Since, there were some events that were already onboarded from that directory by the existing UF and due to the delay the remaining were not yet onboarded while we disabled monitoring for the existing UF and enabled the new UF to monitor that directory. Is there a way to avoid re-indexing/duplication of the events (if my understanding is correct and this scenario can occur). To add here, the directory contains multiple log files for each hour of the day as and when events received from the network devices. Thank you for your time and advice, much appreciated.  ... View more

Hi @richgalloway , Yes the maxKBps value has been set to zero. Further, a case was opened with support and it was shared with them that we were getting lots of Enqueuing a very large file with bytes_to_read events and they suggested to increase the value to a higher value or larger than the file size as a workaround that would help out with the delays issue.  Response by support Splunk by default when forwarding large events/logs, they will stop using the tailreader for data ingestion, and will pass the event/log to the batch reader for forwarding. The batch reader by default has a limit set for 20,971,520 bytes, after which will handle the events and cause enqueuing of events.  Workaround To overcome this, We increased the min_batch_size_bytes in limits.conf to a higher value or larger than the file size to have the events being handled by trailing processor instead than batch reader which was causing the stop this resolved the situation. Current contents of limits.conf file  [thruput] maxKBps = 0 [default] min_batch_size_bytes = 1000000000 Even now we are still getting the Enqueuing a very large file events in splunkd.log because some of the log files on the Syslog Server have a size of 17 GB almost.  I'm thinking of increasing the min_batch_size_bytes to maybe 10 GB, have you any suggestions regarding this method? ... View more

There are 5 indexers that the syslog server UF is forwarding to.  I do see messages in splunkd that  TailingProcessor - Could not send data to output queue (parsingQueue), retrying... We are looking at the multiple syslog servers option but could you let me know more about the multiple UF on a single syslog server option? Each splunkd process would be taking CPU resources then and so the specs of the server if increases can let us proceed with the multiple UF option?   ... View more

Well, It is my understanding that the apps are pushed from the deployment server to the universal forwarders when they phone home. For example, a newly deployed universal forwarder once it phones home and the deployment server has assigned it to a server class then all the apps of that server class are pushed to the new universal forwarder, is that right? Also, I tried manually configuring the outputs.conf and inputs.conf in /opt/splunkforwarder/etc/system/local and have verified that I can successfully receive the logs being monitored.  The only major issue right now is that the universal forwarder is still not able to download the apps from the deployment server even though both telnet and the command that you shared earlier show that connectivity is successful. The below command results in output of "CONNECTED" .... "Verify Return Code: 19 (Self-Signed Certificate in Certificate Chain)" .... "DONE" echo '\n' | openssl s_client -connect <deployment-server>:8089 Can there be any issue due to the universal forwarder being version 8.2.2 and deployment server being version 8.1.1 ?  ... View more