Getting Data In

Syslog ng cut the logs of Fortigate?

muradgh
Path Finder

I have a Fortigate firewall that was configured to send UDP logs, lately, I have configured it to send TCP logs instead of UDP, then I have started to see something wrong with the way the logs are received, I have noticed that the logs are being cut in random locations within the single log and continue writing the rest of the log after adding a new line and a time stamp.

muradgh_0-1673777798117.png

 

The above screenshot is just one example, it happens regularly.

I have tried to modify the syslog-ng.conf configuration file, in the options to be specific:

keep_timestamp (yes); ---> keep_timestamp (no);

log_msg_size(65536); ---> log_msg_size(131072);

But the issue still persists!

Can anybody please help me with this?

Labels (1)
Tags (2)

mjdramos
Observer

hi @muradgh i’m having the same issue on my fortigate logs using TCP but we’re using splunk cloud so modifying the props.conf file i think is not a straightforward task for us to do so i’m planning to use UDP instead.. 

are you able to share with me your syslog-ng.conf for fortigate logging if that’s ok with you? i also need inputs on setting up the correct filters to make the raw output readable and one line per event.

did you also set the log format on fortigate firewall to use rfc 5424 when sending to syslog-ng?

thank you in advance! 🙏

0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you're on Cloud, you can't send your syslog directly to cloud and need a local forwarder (or SC4S instance)  anyway. So it doesn't matter much whether it's TCP or UDP (at least in terms of on-site vs. Cloud).

jrmurray
Explorer

I found that the solution to this is using the syslog-ng syslog() driver instead of the network() driver as your source.

Here is an example configuration:

source "s_fortigate" {
  syslog( 
    transport("tcp")
    port(1514)
  );
};

template t_msgheader_msg {
template("$MSGHDR $MSG\n");
};

destination "d_fortigate" {
file("/data/syslog/fortigate.log"
owner(splunk)
group(splunk)
dir_owner(splunk)
dir_group(splunk)
create_dirs(yes)
dir_perm(0770)
perm(0660)
template(t_msgheader_msg)
);
};

log {
source("s_fortigate");
destination("d_fortigate");
};

@muradgh @idiota 

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @muradgh,

In order to LINE_BREAKER work, you should use the line breaker setting below to fortigate_log  sourcetype on HF.

[fortigate_log]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\d{2,3}\s+<\d{2,3}>)
TIME_PREFIX = eventtime=
TIME_FORMAT = %s%9N

This should work better but you may have similar problem again if Syslog-ng puts a new line.

It is better to tell syslog-ng event boundaries to make it write one event per line. I don't have experience on syslog-ng that is why I don't know how to do it. 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

muradgh
Path Finder

Hi @scelikok 

I have already set this setting on the HF the minute I converted the firewall to send TCP instead of UDP.
Please check the below screenshot from the props.conf file on the HF:

muradgh_0-1678287787739.png

 

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Go back to using UDP.  The authors of Splunk Connect for Syslog (SC4S) found UDP to be better for syslog than TCP.  See https://splunk.github.io/splunk-connect-for-syslog/main/architecture/#udp-vs-tcp

---
If this reply helps you, Karma would be appreciated.
0 Karma

PickleRick
SplunkTrust
SplunkTrust

@richgallowayWhile I agree that sometimes UDP is better for syslog (sometimes it's the only option that the source can use but it's another story), the linked "article" grossly oversimplifies the matter. And complaining about overhead and then suggesting running TLS... That's hilarious. True, UDP is "faster" but several types of syslog sources I know of will happily send events over UDP without caring whether anyone receives them (ignoring ICMP unreachable completely) but will buffer quite a lot of data in case of TCP problems. So it's not that easy.

@muradghDo a tcpdump on your event stream and see what is actually being sent over the wire. You'll see whether it's the sender's fault or something on the receiver end.

0 Karma

muradgh
Path Finder

Hi @PickleRick 
I have noticed something here, 
Between two cuts there are exactly 131072 characters, which is exactly the number in log_msg_size(131072)!!
Now as per the syslog-ng website below, it is not recommended to set log-msg-size() higher than 10 MiB.
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-...

So I have increased the number to around 10, this did reduce the frequency of the cut, but the issue is still there. 
I don't think that the solution is to keep increasing the number!
Any idea?

Thanks

Tags (2)

PickleRick
SplunkTrust
SplunkTrust

OK. Did you do the tcpdump?

It's always the first thing to do in such cases.

Since it's a TCP stream and the problem is with wrong detection of event boundaries I'd suspect that there is a mismatch between sent and expected line breaks. For example, receiver expects \n to delimit events but source sends \r. (happens sometimes with some more "exotic" solutions).

0 Karma

muradgh
Path Finder

Dear @PickleRick 

I have done a tcpdump and put the pcap file in Wireshark and did not see any issues in the logs.
In that same period of time i have done a "tail -f" command to the written .log file and found cuts in the logs same as the picture in my original question.

So in this case I'm almost certain that this issue is caused by the syslog-ng itself. 

Moreover, I have worked on Fortinet FortiGate Add-On for Splunk document in the "details" page that recommends to modifies the props.conf as below:

"Note: the UDP port, 514 in this example should be opened in firewall for logs to pass through. If you choose TCP input and on FortiGate use "reliable"(tcp) mode for syslog setting, you will need to add the following in local/props.conf because tcp tranported syslog will have xxx <yyy> header as line indicator.(8514 below is an example of TCP port, you can choose your own. There is no timestamp header like UDP so you can specify the timestamp field in the fortigate log, in our case the precision is in nanoseconds so the time format is %s%9N. If your FOS version has time stamp in different precision"

[source::tcp:8514]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\d{2,3}\s+<\d{2,3}>)
TIME_PREFIX = eventtime=
TIME_FORMAT = %s%9N


The above note can be found here:
https://splunkbase.splunk.com/app/2846

Tags (2)
0 Karma

muradgh
Path Finder

@PickleRick 
Just adding more information here:

When the firewall was sending UPD, the logs were separated by a new line and all worked just fine without modifying anything in the props.conf
Here is a sample of the firewall logs that were sent using UPD:

muradgh_0-1678278127292.png

 

Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. Wait a second.

1. UDP, not UPD 😉

2. With UDP transmission each event is sent in a separate datagram so there is no need for explicit breaking into events. With TCP you have a continuous stream of data which needs to be broken into events.

3. As you can see from the quoted excerpt from props.conf, the LINE_BREAKER in case of this TCP transmission it looks... strange. Since you seem to be using some intermediate layer to receive syslog messages and write it to files, that syslog daemon (syslog-ng, right?) might not be able to split your stream by this "delimiter". You could try to use this LINE_BREAKER in your props.conf on the indexers/HFs so that the written files will be read and broken into events properly (you should also set proper EVENT_BREAKER on the UF which reads this file).

0 Karma

muradgh
Path Finder

@PickleRick 

Ok
1. oops my bad 😂😂
2. thanks for the information 👍

3. I have taken a chunk of the logs from the written file and put it in regex101.com and tested the LINE_BREAKER regex and it worked as expected, please see below:

muradgh_0-1678282802070.png


Also, my setup is as follows: 
I have 2 Splunk machines, one acts as HF & DS, and the other acts as IDX & SH, they are all in the same subnet including the firewall.
On the HF/DS machine I have installed syslog-ng on it to receive the syslog logs on it, the syslog-ng writes the logs in a directory on the machine, and I have inputs.conf file that monitors this directory with its contents, and outputs.conf file to forward the logs to the SH/IDX machine.
Also, on both Splunk machines, they have the same "Fortinet FortiGate Add-On for Splunk" with the same configurations.

 

Tags (2)
0 Karma

diogofgm_h2i
Engager

Have you done a a tcpdump? just to make sure the issue is not with the sender.
What's  interesting is the logs in the image seem to be missing several line breaks between events. the <189>date=... should be in the beginning of the event and you have more than 1 occurrence per line meaning more than one event in the same line. Also the number before the <189> (e.g. 605) looks like a line/event count and I don't think that's expected from this source. Are you receiving this directly from the Fortigate or do you have anything in between?

Something I found that seems to be related but ended up not being a syslog-ng issue: https://lists.balabit.hu/pipermail/syslog-ng/2022-August/026516.html

0 Karma

muradgh
Path Finder

Hi @diogofgm_h2i 

I have done a tcpdump and put the pcap file in Wireshark and did not see any issues in the logs.
In that same period of time i have done a "tail -f" command to the written .log file and found cuts in the logs same as the picture in my original question.

So in this case I'm almost certain that this issue is caused by the syslog-ng itself. 

Moreover, I have worked on Fortinet FortiGate Add-On for Splunk document in the "details" page that recommends to modifies the props.conf as below:

"Note: the UDP port, 514 in this example should be opened in firewall for logs to pass through. If you choose TCP input and on FortiGate use "reliable"(tcp) mode for syslog setting, you will need to add the following in local/props.conf because tcp tranported syslog will have xxx <yyy> header as line indicator.(8514 below is an example of TCP port, you can choose your own. There is no timestamp header like UDP so you can specify the timestamp field in the fortigate log, in our case the precision is in nanoseconds so the time format is %s%9N. If your FOS version has time stamp in different precision"

 

[source::tcp:8514]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\d{2,3}\s+<\d{2,3}>)
TIME_PREFIX = eventtime=
TIME_FORMAT = %s%9N

 


The above note can be found here:
https://splunkbase.splunk.com/app/2846

And regarding the receiving of the logs, The firewall is in the same subnet that the syslog-ng machine is in. 

Regarding the link you have provided, I will look into it and get back to you thanks. 

More on that,  when the firewall was sending UPD, the logs were separated by a new line and all worked just fine without modifying anything in the props.conf
Here is a sample of the firewall logs that were sent using UPD:

muradgh_0-1678277997768.png

 




Tags (2)
0 Karma

idiota
Loves-to-Learn Lots

Hi @muradgh 
I had the same issue.

After you use the UDP port to delivery Fortigate syslog, was the issue permanently resolved?

0 Karma

muradgh
Path Finder

I have reverted back yo using UDP, and how everything is back to normal again.

0 Karma

idiota
Loves-to-Learn Lots

Thanks so much for your feedback, I will go to try the solution.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...