What are recommendations for our current file monitoring setup?

[some_guy]

My setup looks like this:
syslog collector with UF -> HF -> Index cluster

File input in a directory on the syslog collector: /log/logfile.log

logfile.log is continuously monitored. File size approaching 22GB as I write this.

This was the approach recommended by our Splunk subject-matter-expert. However, Splunk Education tells me that tiny logfiles are always preferable (I can split the logfiles up into smaller files with logrotate).

What would you do? Thanks.

[aaronkorn · Accepted solution]

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.