Using Ubuntu with Universal Forwarder to sent logs in SPLUNK - Dilemmas and Questions

[ornaldo]

Hi community,

There are a lot of articles videos in youtube etc but at some point it is becoming so so confusing so i'm asking for a little help here.

Topic: I want to use a syslog ng server in Ubuntu in order Trimming  and send logs to SPLUNK

What i have done so far: 

•  Installed an Ubuntu Server (Ubuntu 22.04.2 LTS)
• Installed the universal forwarder (splunkforwarder-9.1.0.1-77f73c9edb85-linux-2)
• Installed syslog-ng
• Configured inputs.conf in /opt/splunkforwarder/etc/apps/search/local

# FortiGate
[monitor:///root/syslog/logs/fortinet/fortigate/*/*.log]
sourcetype = fgt_log
index = fortigate
disabled = false
host_segment = 6

• Configured outputs.conf in /opt/splunkforwarder/etc/system/local

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = X.X.X.X.9997

[tcpout-server://1X.X.X.X.:9997]
useACK = true
useSSL = true
clientCert = $SPLUNK_HOME/etc/auth/XXX/server.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/XXX/cacert.pem[tcpout-server://X.X.X.X:9997]
sslVerifyServerCert = true
sslAltNameToCheck = XXXX

• Configured the certs

What is wrong?: 

• There are a lot of output.conf and input.conf directories. Which is the correct one ?
• In SPLUNK i can see that logs are coming using index=_internal  but there are not the logs of fortigate

07-11-2023 12:48:48.579 +0200 INFO AutoLoadBalancedConnectionStrategy [2746 TcpOutEloop] - Found currently active indexer. Connected to idx=10.10.10.203:9997:0, reuse=1.
date_hour = 12date_mday = 11date_minute = 48date_month = julydate_second = 48date_wday = tuesdaydate_year = 2023date_zone = 120eventtype = splunkd-loghost = syslogsrvindex = _internallinecount = 1punct = --_::._+____[_]_-____.___=...::,_=.source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkdsplunk_server = splunktimeendpos = 29timestartpos = 0

• Also i noticed that some of the logs of fortigate are under index=main.

[ornaldo]

I'm sharing here the final version of syslog-ng.conf that worked on trimming part of fortigate logs. However i noticed that license is being consumed like before 😞 

@version: 3.35

@include "scl.conf"

# Syslog-ng configuration file, compatible with default Debian syslogd

# installation.

# First, set some global options.

options { 

    chain_hostnames(off); 

    flush_lines(0); 

    use_dns(no); 

    use_fqdn(no);

    owner("root"); 

    group("adm"); 

    perm(0640); 

    stats_freq(0);

    bad_hostname("^gconfd$");

    create_dirs(yes);

    keep_hostname(yes);

    log_fifo_size(2048);

    log_msg_size(8192);

    time_reopen(10);

};

# Adding param to make syslog-ng listen on udp/514 for syslog

source s_net {

    udp(port(514));

};

# Adding destination for local file to receive FortiGate logs

destination d_fortinet_fortigate {

    file("/root/syslog/logs/fortinet/fortigate/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortigate.log" create_dirs(yes));

};

destination d_fortinet_fortiweb {

    file("/root/syslog/logs/fortinet/fortiweb/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortiweb.log" create_dirs(yes));

};

destination d_fortinet_fortiauthenticator {

    file("/root/syslog/logs/fortinet/fortiauthenticator/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortiauthenticator.log" create_dirs(yes));

};

destination d_fortinet_fortimail {

    file("/root/syslog/logs/fortinet/fortimail/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortimail.log" create_dirs(yes));

};

# Filter to instruct syslog-ng how to identify FortiGate syslog

filter f_fortinet_fortigate {

    match("devid=\"FG[A-Z0-9]+\"" value("MESSAGE"));

};

filter f_fortinet_fortiweb {

    match("device_id=FV[A-Z0-9]+" value("MESSAGE"));

};

filter f_fortinet_fortiauthenticator {

    match("subcategory=\"Authentication\"" value("MESSAGE"));

};

filter f_fortinet_fortimail {

    match("device_id=FE[A-Z0-9]+" value("MESSAGE"));

};

# Trimming different parts together for logging

rewrite rewrite_fortigate {

 #subst ( '^time=\d\d:\d\d:\d\d(\.\d{3,6})? ', "", value("MESSAGE") );

 subst ( ' devid="[A-Z0-9]{16}" ', " ", value("MESSAGE") );

 subst ( ' logid="[0-9]{10}" ', " ", value("MESSAGE") );

 subst ( ' srcname="([\w-]+)\.example\.local" ', ' srcname="$1" ', value("MESSAGE") );

 subst ( ' srcintfrole="[a-z]+" ', " ", value("MESSAGE") );

 subst ( ' dstname="([\w-]+)\.example\.local" ', ' dstname="$1" ', value("MESSAGE") );

 subst ( ' dstintfrole="[a-z]+" ', " ", value("MESSAGE") );

 subst ( ' poluuid="[a-z0-9-]{36}" ', " ", value("MESSAGE") );

 subst ( ' sessionid=[0-9]+ ', " ", value("MESSAGE") );

 subst ( ' policytype="policy" ', " ", value("MESSAGE") );

 subst ( ' appcat="unscanned"', "", value("MESSAGE") );

 subst ( ' crscore=[0-9]+ craction=[0-9]+ crlevel="[a-z]+"', "", value("MESSAGE") );

 subst ( ' attackid=[0-9]+ ', " ", value("MESSAGE") );

 subst ( ' incidentserialno=[0-9]+ ', " ", value("MESSAGE") );

 subst ( ' ref="http:\/\/www\.fortinet\.com\/ids\/VID[0-9]+"', " ", value("MESSAGE") );

 subst ( ' msg="application[s]?([0-9]+)?:[^.]+.[^.]+.[^.]+.[^.]+.[^.]+"', " ", value("MESSAGE") );

};

# Creating the different parts together for logging

log {

    source(s_net);

    filter(f_fortinet_fortigate);

rewrite(rewrite_fortigate);

    destination(d_fortinet_fortigate);

};

log { 

    source(s_net);

    filter(f_fortinet_fortiweb);

    destination(d_fortinet_fortiweb);

};

log { 

    source(s_net);

    filter(f_fortinet_fortiauthenticator);

    destination(d_fortinet_fortiauthenticator);

};

log {

    source(s_net);

    filter(f_fortinet_fortimail);

    destination(d_fortinet_fortimail);

};

###

# Include all config files in /etc/syslog-ng/conf.d/

###

@include "/etc/syslog-ng/conf.d/*.conf"

[venky1544]

Hi @ornaldo 

• Configure the  inputs.conf in splunk/etc/system/local/inputs.conf

if you use the btool precedence command you get to know why the data is going in main index

[ornaldo]

I have used a default syslog-ng.conf that i have found in the internet:

@version: 3.13
@include "scl.conf"

# Syslog-ng configuration file, compatible with default Debian syslogd
# installation.

# First, set some global options.
options {
chain_hostnames(off);
flush_lines(0);
use_dns(no);
use_fqdn(no);
owner("root");
group("adm");
perm(0640);
stats_freq(0);
bad_hostname("^gconfd$");
create_dirs(yes);
keep_hostname(yes);
log_fifo_size(2048);
log_msg_size(8192);
time_reopen(10);
};
# Adding param to make syslog-ng listen on udp/514 for syslog
source s_net {
udp(port(514));
};
# Adding destination for local file to receive FortiGate logs

destination d_fortinet_fortigate {
file("/root/syslog/logs/fortinet/fortigate/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortigate.log" create_dirs(yes));
};

destination d_fortinet_fortiweb {
file("/root/syslog/logs/fortinet/fortiweb/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortiweb.log" create_dirs(yes));
};

destination d_fortinet_fortiauthenticator {
file("/root/syslog/logs/fortinet/fortiauthenticator/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortiauthenticator.log" create_dirs(yes));
};

destination d_fortinet_fortimail {
file("/root/syslog/logs/fortinet/fortimail/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortimail.log" create_dirs(yes));
};
# Filter to instruct syslog-ng how to identify FortiGate syslog
filter f_fortinet_fortigate {
match("devid=\"FG[A-Z0-9]+\"" value("MESSAGE"));
};

filter f_fortinet_fortiweb {
match("device_id=FV[A-Z0-9]+" value("MESSAGE"));
};

filter f_fortinet_fortiauthenticator {
match("subcategory=\"Authentication\"" value("MESSAGE"));
};

filter f_fortinet_fortimail {
match("device_id=FE[A-Z0-9]+" value("MESSAGE"));
};
# Creating the different parts together for logging

log {
source(s_net);
filter(f_fortinet_fortigate);
destination(d_fortinet_fortigate);
};

log {
source(s_net);
filter(f_fortinet_fortiweb);
destination(d_fortinet_fortiweb);
};

log {
source(s_net);
filter(f_fortinet_fortiauthenticator);
destination(d_fortinet_fortiauthenticator);
};

log {
source(s_net);
filter(f_fortinet_fortimail);
destination(d_fortinet_fortimail);
};
###
# Include all config files in /etc/syslog-ng/conf.d/
###
@include "/etc/syslog-ng/conf.d/*.conf"

[ornaldo]

Hi, thank you for your reply. 

Sorry but i'm still confused.  In SPLUNK i can see that logs are comming under index="main" sourcetype="fortigate_traffic"

[GaetanVP]

Yes when you create/modified an inputs.conf file, and do not specify anything in front of the "index" key, Splunk will by default forward the logs to the main index.

I suppose you missed to configure the targeted index in one of your inputs.conf file, to find it you can once again use the btool command to locate the conf file that you will need to modify !

/opt/splunkforwarder/bin/splunk btool inputs list --debug | grep -i fortigate
/opt/splunkforwarder/bin/splunk btool sourcetypes list --debug | grep -i fortigate

This should return the path where you specified the sourcetype for incoming fortigate data.

Let me know how it went !
GaetanVP 

[ornaldo]

The output of the first debug is:

/opt/splunkforwarder/etc/apps/search/local/inputs.conf                     [monitor:///root/syslog/logs/fortinet/fortigate/*/*.log]
/opt/splunkforwarder/etc/apps/search/local/inputs.conf                     index = fortigate

The output of the second debug is empty

thnx 

[GaetanVP]

Hello @ornaldo ,

Here are some thoughts regarding your porst :

- There are a lot of output.conf and input.conf directories. Which is the correct one ?

Indeed, you can have multiple inputs.conf and outputs.conf Splunk files configured on a machine. Splunk will basically merged all of those file with a specific precedence.

More info here : https://docs.splunk.com/Documentation/Splunk/9.1.0/Admin/Wheretofindtheconfigurationfiles

In your case, I think this is precedence that will be "follow"

It is not really the best practice to put inputs.conf and outputs.conf directly in /opt/splunkforwarder/etc/system/local of a Universal Forwarder because some configuration could be overridden by same *.conf file located in /opt/splunkforwarder/etc/apps/<my_apps>/...

Another way to list the final merged that Splunk did for a specific configuration is the following commands :

/opt/splunkforwarder/bin/splunk btool inputs list --debug
/opt/splunkforwarder/bin/splunk btool outputs list --debug

This will display what configuration are applied and where it is located, can be useful to debug.

- In SPLUNK i can see that logs are coming using index=_internal  but there are not the logs of fortigate

Yes by default after the UF installation, Splunk will try to forward all the Splunk internal related logs to the tcpout defined. The logs forwarded are for instance located in /opt/splunkforwarder/var/log/splunk

Using the first point, if you want to learn more about what is monitored and how it's forward to your indexer, you can use : 

/opt/splunkforwarder/bin/splunk btool inputs list --debug | grep -i _internal

You will be able to locate all the apps containing conf files responsible of sending logs to the "_internal" index.

Hope it helps !
GaetanVP