Getting Data In

What are recommendations for our current file monitoring setup?

some_guy
Path Finder

My setup looks like this:
syslog collector with UF -> HF -> Index cluster

File input in a directory on the syslog collector: /log/logfile.log

logfile.log is continuously monitored. File size approaching 22GB as I write this.

This was the approach recommended by our Splunk subject-matter-expert. However, Splunk Education tells me that tiny logfiles are always preferable (I can split the logfiles up into smaller files with logrotate).

What would you do? Thanks.

Tags (3)
0 Karma
1 Solution

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

View solution in original post

aaronkorn
Splunk Employee
Splunk Employee

I would modify your syslog-ng.conf to rotate the files every hour with something like this
destination d_cisco_switches { file("/opt/syslog-data/cisco_switches/$HOST/$YEAR-$MONTH-$DAY/$HOUR-cisco-switches.log" create_dirs(yes)); };.
Depending on how chatty your source is you can also use logrotate but a good rule of thumb is to keep it below 1GB per file.

Also, what is the use case for the HF between the UF and IDX's? You can get away with just UF -> IDX.

Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...