Getting Data In

What are some Syslog storage recommendations?

willspk
Engager

Hey all,

I need some advice regarding our syslog storage facility. We're using rsyslog and at the moment we've got all firewall logs going into a single log file, which is getting pretty large at this point. I'm then using the universal forwarder to send this over to Splunk. The log file at the moment is around 150gb and growing. We've got plenty of space but I was wondering, is there a better way I should be approaching this? For example, should I break the logs up so that each Firewall has it's own directory and new sub directories per day?  

Any insight would be appreciated. 

Thanks,

Will

Labels (3)
Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Rotating log files is a good idea in general - it's easier to maintain and manage disk space if you have your logs divided into separate files. There's one caveat though - if you have too many of the files splunk can take significant time to catch-up with their state on forwarder restart. So having files rotated daily is a reasonable compromise.

Anyway, if you don't need storing logs for other purposes on the syslog server, you can consider sending the logs by rsyslog to HEC input instead of writing them down and ingest with UF. It gives you some additional possiibilities like easy adjusting metadata.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @willspk,

yes it's better to write a different file e.g. every day ot every week to reduce the file dimension.

In addition in this way you can remove older files saving disk space.

I don't hint to use aldo a different folder, it isn't mandatory, but if you like, you're free to do it.

As I said, I'd rotate to a new file e.g. every night and I'd schedule a script that delets files older e.g. of one week or two o three days.

In addition I suppose that you are using two Universal Forwarding to take syslogs with a Load Balancer as a front end, to avoid to loose syslogs during maintenence or fault, if not take in consideration this choice.

Only for your knowledge, having to manage so huge volume of data, take also in cosideration Splunk Connect for Syslog App (https://splunkbase.splunk.com/app/4740/).

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...