About ahmadgul21

ahmadgul21 · ‎11-10-2021

An update, This issue was fixed I believe by using newer version (8.1.1) of the Splunk Agent and the Apps were successfully downloaded from the Splunk DS.

ahmadgul21 · ‎11-10-2021

Hi @richgalloway , An update, we have been trying out different options such as increasing the parsingQueue from10 MB to about 100 MB as per the conversations with support. However, the delay is still there. Coming to your two options, (a) running multiple syslog servers with UF to distribute load - that is being considered however might take time so I'm thinking of trying out your second option (b) running multiple (2) UF on a single syslog server with each UF monitoring different/separate directories. What I'm most curious and apprehensive about with the option (b) is if another UF is installed on the syslog server and tries to monitor a separate directory that was previously being monitored by the existing UF then how do we avoid re-indexing/duplication of events? Since, there were some events that were already onboarded from that directory by the existing UF and due to the delay the remaining were not yet onboarded while we disabled monitoring for the existing UF and enabled the new UF to monitor that directory. Is there a way to avoid re-indexing/duplication of the events (if my understanding is correct and this scenario can occur). To add here, the directory contains multiple log files for each hour of the day as and when events received from the network devices. Thank you for your time and advice, much appreciated.

ahmadgul21 · ‎10-21-2021

Hi @richgalloway , Yes the maxKBps value has been set to zero. Further, a case was opened with support and it was shared with them that we were getting lots of Enqueuing a very large file with bytes_to_read events and they suggested to increase the value to a higher value or larger than the file size as a workaround that would help out with the delays issue. Response by support Splunk by default when forwarding large events/logs, they will stop using the tailreader for data ingestion, and will pass the event/log to the batch reader for forwarding. The batch reader by default has a limit set for 20,971,520 bytes, after which will handle the events and cause enqueuing of events. Workaround To overcome this, We increased the min_batch_size_bytes in limits.conf to a higher value or larger than the file size to have the events being handled by trailing processor instead than batch reader which was causing the stop this resolved the situation. Current contents of limits.conf file [thruput] maxKBps = 0 [default] min_batch_size_bytes = 1000000000 Even now we are still getting the Enqueuing a very large file events in splunkd.log because some of the log files on the Syslog Server have a size of 17 GB almost. I'm thinking of increasing the min_batch_size_bytes to maybe 10 GB, have you any suggestions regarding this method?

ahmadgul21 · ‎10-19-2021

There are 5 indexers that the syslog server UF is forwarding to. I do see messages in splunkd that TailingProcessor - Could not send data to output queue (parsingQueue), retrying... We are looking at the multiple syslog servers option but could you let me know more about the multiple UF on a single syslog server option? Each splunkd process would be taking CPU resources then and so the specs of the server if increases can let us proceed with the multiple UF option?

ahmadgul21 · ‎10-18-2021

Hi, I would like some advice on how to best resolve my current issue with regards to the Logs being delayed that are being sent from Syslog Server via a Universal Forwarder. Situation Details: We have a central Syslog Server (Linux Based) that is handling all network devices Logs and storing them locally. The average size of daily log volume on the Syslog Server is about 800 GB to 1 TB approx. We have configured retention policies such that only 2 days of Logs are kept locally on the Syslog Server. To onboard the logs into Splunk we have deployed a Universal Forwarder on the Syslog Server and configured the below to optimize it, server.conf [general] parallelIngestionPipelines = 2 [queue=parsingQueue] maxSize = 10MB limits.conf [thruput] maxKBps = 0 However, even now we are getting many events in splunkd.log of WARN Tail Reader - Enqueuing a very large file=/logs/.... in the batch reader, with bytes_to_read=xxxxxxx, reading of other large files could be delayed The effect of this is that even though Logs are being forwarded to Splunk but there is a large delay time associated with when the logs are received on Syslog Server and when they are indexed/searchable in Splunk. There are many dips in logs received in Splunk when investigating and usually these dips recover as soon as we start receiving the logs that are sent by Splunk UF. However in some cases, the logs are missed because it seems by the time the Splunk agent reaches the log file to read 2 days have passed so that the log file is no longer available as per retention policies. Would appreciate any helpful advice on how to go about addressing this issue. Thank you.

ahmadgul21 · ‎08-24-2021

Well, It is my understanding that the apps are pushed from the deployment server to the universal forwarders when they phone home. For example, a newly deployed universal forwarder once it phones home and the deployment server has assigned it to a server class then all the apps of that server class are pushed to the new universal forwarder, is that right? Also, I tried manually configuring the outputs.conf and inputs.conf in /opt/splunkforwarder/etc/system/local and have verified that I can successfully receive the logs being monitored. The only major issue right now is that the universal forwarder is still not able to download the apps from the deployment server even though both telnet and the command that you shared earlier show that connectivity is successful. The below command results in output of "CONNECTED" .... "Verify Return Code: 19 (Self-Signed Certificate in Certificate Chain)" .... "DONE" echo '\n' | openssl s_client -connect <deployment-server>:8089 Can there be any issue due to the universal forwarder being version 8.2.2 and deployment server being version 8.1.1 ?

ahmadgul21 · ‎08-24-2021

Hi, The issue is that some servers with universal forwarder agent deployed on them are not being able to successfully download the apps from the deployment server. Environment Details: Server: Linux RHEL 7.9 (3.x Kernel) Deployment Server: Splunk Enterprise 8.x Splunk Universal Forwarder: 8.2.2 for Linux The agent is successfully installed and connected to the deployment server using the below command ./splunk set deploy-poll depoloyment-server:8089 And it is showing up successfully on the deployment server as well however when I push apps to the server via the deployment server they aren't successfully downloaded. From the universal forwarder splunkd.log, ERROR HttpClientRequest *** - HTTP client error=Connection closed by peer while accessing server=*** for request=*** From the deployment server splunkd.log, What can be the possible reason for this behavior? Since the communication seems fine (we've opened uni-directional communication from server to deployment-server on port 8089). Kind regards

Posts	7
Solutions	0
Karma Given	1
Karma Received	0
Member Since	‎06-24-2021

Online Status	Offline
Date Last Visited	‎03-21-2022 09:12 AM

UF on Syslog Server - Enqueuing Large Files - Logs...

Splunk Forwarder - Deployment Apps not being downl...

Re: Splunk Forwarder - Deployment Apps not being d...

Re: UF on Syslog Server - Enqueuing Large Files - ...

Re: UF on Syslog Server - Enqueuing Large Files - ...

Re: UF on Syslog Server - Enqueuing Large Files - ...

UF on Syslog Server - Enqueuing Large Files - Logs...

Re: Splunk Forwarder - Deployment Apps not being d...

Splunk Forwarder - Deployment Apps not being downl...