Getting Data In

What are recommendations for our current file monitoring setup?

some_guy
Path Finder

My setup looks like this:
syslog collector with UF -> HF -> Index cluster

File input in a directory on the syslog collector: /log/logfile.log

logfile.log is continuously monitored. File size approaching 22GB as I write this.

This was the approach recommended by our Splunk subject-matter-expert. However, Splunk Education tells me that tiny logfiles are always preferable (I can split the logfiles up into smaller files with logrotate).

What would you do? Thanks.

Tags (3)
0 Karma
1 Solution

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

View solution in original post

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...