A year after, well, better late than never 😆 unfortunately, I didn't found a solution to this case, so I just moved on 😢  ... View more

I have reverted back yo using UDP, and how everything is back to normal again. ... View more

Unfortunately, I couldn't resolve this, I gave up trying to integrate this with Splunk 😔 ... View more

The network connection between this HF and the two indexers is a VPN site-to-site connection, as this HF is on the customer site and the two indexers are on our site. Regarding your second point, we are using HFs in this case because: The UFs are on the customer side and the indexers are on our site, we can't just simply send all the UF data directly to the indexers, we needed an HF in the middle. We installed syslog-ng on the HF to collect the syslog logs from the network devices (firewalls, routers, switches, etc.) that are on the customer side, then we created a local inputs.conf on the HF and in it a monitor stanza to monitor the syslog logs directory and send them to the indexers. We also installed and configured the db connect app on the HF to get the database logs from the customer also. What I have done lately is adding another new HF alongside this one hoping somehow to eliminate the issues, then I configured all the UFs to send the logs only to the new HF, leaving the old HF to handle only the syslog logs and the databases, yet the issue with the old HF is still there! Isn't having the below prevent the HF from parsing the logs?  [indexAndForward] index = false ... View more

@PickleRick  On the HF, under the limits.conf: [thruput] maxKBps = 0 [inputproc] max_fd = 900 and under the outputs.conf: [tcpout] defaultGroup = primary_indexers maxQueueSize = 100MB useACK = false forceTimebasedAutoLB = true autoLBFrequency= 30 autoLBVolume = 15728640 indexAndForward = false forwardedindex.filter.disable = true compressed = false forwardedindex.2.whitelist = (_audit|_introspection|_internal) [tcpout:primary_indexers] server = x.x.x.10:9997, x.x.x.11:9997 [indexAndForward] index = false Please note that even though I have increased the number of "parallelIngestionPipelines" to 12, the server specs were not impacted! nothing is fully saturated as I expected it to do! the CPU utilization is 2% and the RAM is 7 GB out of 16 GB! ... View more

Hi @scelikok  Thank you very much. ... View more

Hi @scelikok  I have already set this setting on the HF the minute I converted the firewall to send TCP instead of UDP. Please check the below screenshot from the props.conf file on the HF:   ... View more

@PickleRick  Ok 1. oops my bad 😂😂 2. thanks for the information 👍 3. I have taken a chunk of the logs from the written file and put it in regex101.com and tested the LINE_BREAKER regex and it worked as expected, please see below: Also, my setup is as follows:  I have 2 Splunk machines, one acts as HF & DS, and the other acts as IDX & SH, they are all in the same subnet including the firewall. On the HF/DS machine I have installed syslog-ng on it to receive the syslog logs on it, the syslog-ng writes the logs in a directory on the machine, and I have inputs.conf file that monitors this directory with its contents, and outputs.conf file to forward the logs to the SH/IDX machine. Also, on both Splunk machines, they have the same "Fortinet FortiGate Add-On for Splunk" with the same configurations.   ... View more

@PickleRick  Just adding more information here: When the firewall was sending UPD, the logs were separated by a new line and all worked just fine without modifying anything in the props.conf Here is a sample of the firewall logs that were sent using UPD:   ... View more

Hi @diogofgm_h2i  I have done a tcpdump and put the pcap file in Wireshark and did not see any issues in the logs. In that same period of time i have done a "tail -f" command to the written .log file and found cuts in the logs same as the picture in my original question. So in this case I'm almost certain that this issue is caused by the syslog-ng itself.  Moreover, I have worked on Fortinet FortiGate Add-On for Splunk document in the "details" page that recommends to modifies the props.conf as below: "Note: the UDP port, 514 in this example should be opened in firewall for logs to pass through. If you choose TCP input and on FortiGate use "reliable"(tcp) mode for syslog setting, you will need to add the following in local/props.conf because tcp tranported syslog will have xxx <yyy> header as line indicator.(8514 below is an example of TCP port, you can choose your own. There is no timestamp header like UDP so you can specify the timestamp field in the fortigate log, in our case the precision is in nanoseconds so the time format is %s%9N. If your FOS version has time stamp in different precision"   [source::tcp:8514] SHOULD_LINEMERGE = false LINE_BREAKER = (\d{2,3}\s+<\d{2,3}>) TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N   The above note can be found here: https://splunkbase.splunk.com/app/2846 And regarding the receiving of the logs, The firewall is in the same subnet that the syslog-ng machine is in.  Regarding the link you have provided, I will look into it and get back to you thanks.  More on that,  when the firewall was sending UPD, the logs were separated by a new line and all worked just fine without modifying anything in the props.conf Here is a sample of the firewall logs that were sent using UPD:   ... View more

Dear @PickleRick  I have done a tcpdump and put the pcap file in Wireshark and did not see any issues in the logs. In that same period of time i have done a "tail -f" command to the written .log file and found cuts in the logs same as the picture in my original question. So in this case I'm almost certain that this issue is caused by the syslog-ng itself.  Moreover, I have worked on Fortinet FortiGate Add-On for Splunk document in the "details" page that recommends to modifies the props.conf as below: "Note: the UDP port, 514 in this example should be opened in firewall for logs to pass through. If you choose TCP input and on FortiGate use "reliable"(tcp) mode for syslog setting, you will need to add the following in local/props.conf because tcp tranported syslog will have xxx <yyy> header as line indicator.(8514 below is an example of TCP port, you can choose your own. There is no timestamp header like UDP so you can specify the timestamp field in the fortigate log, in our case the precision is in nanoseconds so the time format is %s%9N. If your FOS version has time stamp in different precision" [source::tcp:8514] SHOULD_LINEMERGE = false LINE_BREAKER = (\d{2,3}\s+<\d{2,3}>) TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N The above note can be found here: https://splunkbase.splunk.com/app/2846 ... View more

Hi @PickleRick  I have noticed something here,  Between two cuts there are exactly 131072 characters, which is exactly the number in log_msg_size(131072)!! Now as per the syslog-ng website below, it is not recommended to set log-msg-size() higher than 10 MiB. https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/log-msg-size So I have increased the number to around 10, this did reduce the frequency of the cut, but the issue is still there.  I don't think that the solution is to keep increasing the number! Any idea? Thanks ... View more

I have checked but nope, it's not suppressed. And yes the search produces results in a tabular format ... View more

I have checked but nope, it's not suppressed ... View more

That's the one! Thank you 👍 ... View more