Getting Data In

What are recommendations for our current file monitoring setup?

some_guy
Path Finder

My setup looks like this:
syslog collector with UF -> HF -> Index cluster

File input in a directory on the syslog collector: /log/logfile.log

logfile.log is continuously monitored. File size approaching 22GB as I write this.

This was the approach recommended by our Splunk subject-matter-expert. However, Splunk Education tells me that tiny logfiles are always preferable (I can split the logfiles up into smaller files with logrotate).

What would you do? Thanks.

Tags (3)
0 Karma
1 Solution

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

View solution in original post

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...