Getting Data In

What are some Syslog storage recommendations?

willspk
Engager

Hey all,

I need some advice regarding our syslog storage facility. We're using rsyslog and at the moment we've got all firewall logs going into a single log file, which is getting pretty large at this point. I'm then using the universal forwarder to send this over to Splunk. The log file at the moment is around 150gb and growing. We've got plenty of space but I was wondering, is there a better way I should be approaching this? For example, should I break the logs up so that each Firewall has it's own directory and new sub directories per day?  

Any insight would be appreciated. 

Thanks,

Will

Labels (3)
Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Rotating log files is a good idea in general - it's easier to maintain and manage disk space if you have your logs divided into separate files. There's one caveat though - if you have too many of the files splunk can take significant time to catch-up with their state on forwarder restart. So having files rotated daily is a reasonable compromise.

Anyway, if you don't need storing logs for other purposes on the syslog server, you can consider sending the logs by rsyslog to HEC input instead of writing them down and ingest with UF. It gives you some additional possiibilities like easy adjusting metadata.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @willspk,

yes it's better to write a different file e.g. every day ot every week to reduce the file dimension.

In addition in this way you can remove older files saving disk space.

I don't hint to use aldo a different folder, it isn't mandatory, but if you like, you're free to do it.

As I said, I'd rotate to a new file e.g. every night and I'd schedule a script that delets files older e.g. of one week or two o three days.

In addition I suppose that you are using two Universal Forwarding to take syslogs with a Load Balancer as a front end, to avoid to loose syslogs during maintenence or fault, if not take in consideration this choice.

Only for your knowledge, having to manage so huge volume of data, take also in cosideration Splunk Connect for Syslog App (https://splunkbase.splunk.com/app/4740/).

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...