FortiGate Firewall is consuming the license

[ornaldo]

Dears,

After the forward of the logs from FortiGate toward SPLUNK we noticed that the license is being consumed recently. Can you help us with trimming and additional advice ?

Thank You

[gcusello]

Hi @ornaldo,

firewall logs are usually the most important license consumer!

Anyway, you can filter your events deleting some events before indexing: in this way you limit the license consuption, but you loose data that could be useful.

So you have to analyze your logs and identify the ones that you think aren't so useful for your purposes, in details, you have to find one or more regexes to filter these events.

Then you have to apply the procedure described at https://docs.splunk.com/Documentation/Splunk/9.0.5/Forwarding/Routeandfilterdatad#Filter_event_data_... on the first Heavy Forwarder you logs pass through or on Indexers (if there isn't ant HF).

in few words,

in props.conf:

[your_sourcetype]
TRANSFORMS-null = setnull

and in transforms.conf:

[setnull]
REGEX = <your_regex_for_filtering>
DEST_KEY = queue
FORMAT = nullQueue

beware choosing the sourcetype that the Fortigate Add-On make a sourcetype transformation, so use the correct one that you can identify running a search on your logs.

Ciao.

Giuseppe

[ornaldo]

Thank You for your reply.

So using this method it can work without adding another syslog in the middle.

Is there any guide for the REGEX language that should i use for Fortinet ?

Thank You 

[PickleRick]

Receiving syslogs directly on your Splunk instance is not a very good idea in a production environment. It has scalability issues and you lose information about where the event really came from (which might be important if you have more than one firewall and - for example - receive logs from many different network segments in which the local firewall is always named "firewall1" so the host field parsed out from the event itself makes the sources indistinguishable from one another.

[gcusello]

Hi @ornaldo,

you can find many sites that exèòain hot to use regex, and sites that help you in regex testing, I use regex101.com, but it isn't the only one.

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[isoutamo]

Hi

It's just like @gcusello said. One thing what I have seen on Fortinet logs that those have basically same information several times on event. You probably could save some license if you remove those "duplicate" parts on logs. You could use props.conf + SEDCMD for this. BUT f/when you are using Forti's own app & ta to analyse those logs, I don't know are those working after that or not?

r. Ismo

[ornaldo]

In some articles in the community i have seen that they are suggesting trimming firewall logs using syslog-ng to conserve license Solved: Trimming firewall logs using syslog-ng to conserve... - Splunk Community

I do not know if this is a good idea and i was looking for something less time consuming using props.conf.

But i do not know if anyone had this experience before and can share some information how they have tailored the props.conf 

[isoutamo]

This is exactly what I mean. Definitely you should collect those logs with real syslog server instead of using Splunk's TCP listener. Both syslog-ng and rsyslog could do that trimming. Both of those could also send events directly with http(s) to HEC. Other option is use local UF on your syslog servers to collect logs. 

If you have working syslog server you should use it, if not then consider use traditional version vs SC4S (https://github.com/splunk/splunk-connect-for-syslog).

[ornaldo]

I understand. Thank You