Getting Data In

What are some Syslog storage recommendations?

willspk
Loves-to-Learn Lots

Hey all,

I need some advice regarding our syslog storage facility. We're using rsyslog and at the moment we've got all firewall logs going into a single log file, which is getting pretty large at this point. I'm then using the universal forwarder to send this over to Splunk. The log file at the moment is around 150gb and growing. We've got plenty of space but I was wondering, is there a better way I should be approaching this? For example, should I break the logs up so that each Firewall has it's own directory and new sub directories per day?  

Any insight would be appreciated. 

Thanks,

Will

Labels (3)
Tags (2)
0 Karma

PickleRick
Ultra Champion

Rotating log files is a good idea in general - it's easier to maintain and manage disk space if you have your logs divided into separate files. There's one caveat though - if you have too many of the files splunk can take significant time to catch-up with their state on forwarder restart. So having files rotated daily is a reasonable compromise.

Anyway, if you don't need storing logs for other purposes on the syslog server, you can consider sending the logs by rsyslog to HEC input instead of writing them down and ingest with UF. It gives you some additional possiibilities like easy adjusting metadata.

0 Karma

gcusello
Legend

Hi @willspk,

yes it's better to write a different file e.g. every day ot every week to reduce the file dimension.

In addition in this way you can remove older files saving disk space.

I don't hint to use aldo a different folder, it isn't mandatory, but if you like, you're free to do it.

As I said, I'd rotate to a new file e.g. every night and I'd schedule a script that delets files older e.g. of one week or two o three days.

In addition I suppose that you are using two Universal Forwarding to take syslogs with a Load Balancer as a front end, to avoid to loose syslogs during maintenence or fault, if not take in consideration this choice.

Only for your knowledge, having to manage so huge volume of data, take also in cosideration Splunk Connect for Syslog App (https://splunkbase.splunk.com/app/4740/).

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...