topic Re: Syslog path (/opt/syslog) is full on rsyslog server? in Getting Data In

Syslog path (/opt/syslog) is full on rsyslog server?

aalhabbash1 — Sun, 25 Aug 2019 06:23:31 GMT

Hi Splunker;

The syslog server store any logs coming to it by syslog on files as .log file then Splunk read this logs from this file and store the logs, when this file start to full Splunk convert this file to .gz file to available the space for restore another logs on .log file another time.

Can I make (/opt/syslog) path 80% used not 100% to avoid such alerts and focus on real issues.

Best Regards;
Abdullah Al-Habbash

Re: Syslog path (/opt/syslog) is full on rsyslog server?

richgalloway — Sun, 25 Aug 2019 12:45:47 GMT

This is not a Splunk problem. The .gz files are created by Linux utilities, not by Splunk. You must employ other Linux utilities (perhaps Logrotate) to ensure disk space does not become 100% utilized.

Re: Syslog path (/opt/syslog) is full on rsyslog server?

jkat54 — Sun, 25 Aug 2019 12:52:11 GMT

There is 0 correct way to use logrotate with syslog AND splunk. It just wasn't designed for the chore.

Better to write your own script for log rotation. those who have found "success" with this have data loss and don't know it.

Re: Syslog path (/opt/syslog) is full on rsyslog server?

DavidHourani — Sun, 25 Aug 2019 13:33:37 GMT

@jkat54 interesting that you would say that. Could you please give us some more details and references ?

Re: Syslog path (/opt/syslog) is full on rsyslog server?

jkat54 — Sun, 25 Aug 2019 14:12:31 GMT

I may have jumped the gun here. Every implementation I've seen had a step to restart syslog.

I suppose if you told logrotate to only rotate files seen more than once AND you make syslog write files with date time stamps, you could have success. But in my experience there were file handles open everywhere, racing conditions between splunk monitor and syslog and logrotate, etc.

I've never seen one setup with all three that wasn't dropping data at some point or causing other unforeseen issues.

I just steer clear, my opinion I suppose.

Re: Syslog path (/opt/syslog) is full on rsyslog server?

DavidHourani — Sun, 25 Aug 2019 14:46:47 GMT

Yeah totally agree with you on this : "in my experience there were file handles open everywhere, racing conditions between splunk monitor and syslog and logrotate, etc." ... and the lower the time interval is on the rotation the more chance data loss could occur.

Re: Syslog path (/opt/syslog) is full on rsyslog server?

aalhabbash1 — Mon, 26 Aug 2019 05:11:53 GMT

Under /etc/logrotate.d/splunk on syslog server I have the below configuration:

/data/syslog/network////.log
/data/syslog/network/////.log
/data/syslog/security////.log
/data/syslog/security/////.log
/data/syslog/security//////.log {
daily
rotate 1
compress
missingok
notifempty
nocreate
postrotate
systemctl reload-or-restart rsyslog.service
systemctl reset-failed rsyslog.service
endscript
}

And I have size for /opt/syslog partition 296G.

How can change this configuration when this partition arrived 200G for make the logrotate?

appreciate your support in that?

Re: Syslog path (/opt/syslog) is full on rsyslog server?

DavidHourani — Mon, 26 Aug 2019 07:06:05 GMT

You can use the size option :
https://linux.die.net/man/8/logrotate

Re: Syslog path (/opt/syslog) is full on rsyslog server?

jkat54 — Mon, 26 Aug 2019 11:46:07 GMT

You can't use logrotate to move data once the filesystem is at x GB in size.

You can write a shell script that uses a combination of df -h, grep, awk, mv, gzip, etc. though. I doubt anyone here is going to write that for you though.

You should try a Linux forum, not a splunk forum.