Getting Data In

What are recommendations for our current file monitoring setup?

some_guy
Path Finder

My setup looks like this:
syslog collector with UF -> HF -> Index cluster

File input in a directory on the syslog collector: /log/logfile.log

logfile.log is continuously monitored. File size approaching 22GB as I write this.

This was the approach recommended by our Splunk subject-matter-expert. However, Splunk Education tells me that tiny logfiles are always preferable (I can split the logfiles up into smaller files with logrotate).

What would you do? Thanks.

Tags (3)
0 Karma
1 Solution

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

View solution in original post

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

Get Updates on the Splunk Community!

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...

Major Splunk Upgrade – Prepare your Environment for Splunk 10 Now!

Attention App Developers: Test Your Apps with the Splunk 10.0 Beta and Ensure Compatibility Before the ...

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...