where i can find the retention configuration for Splunk syslog receiver and how to change the logs for 1 day only

[pacifikn]

Greetings!!

I need your help on where I can find the retention configuration for Splunk syslog receiver through command line and how can change to only receive the logs for 1day not 2days, here I mean to do retention so that I can change where it is receiving and store logs for 2 days, NOW I WANT to do retention to only receive for 1 day only, KINDLY HELP ME AND GUIDE ME HOW I CAN DO THIS?

Splunk receiver it is in opt directory where i receive the syslog logs for different network  devices , storing those logs for days then after logs are deleted, BUT I want only to receive all logs coming from different devices and when day finished at midnight can delete that logs in splunk receiver after being indexed into indexers.

Kindly help me on this as I want to avoid that the receiver storage run out of space.

Thank you in advance!

[PickleRick]

What do you mean by "syslog receiver"? And what do you mean by "retention"?

There are many different approaches to receiving syslog alerts. One (not recommended for production environments) is to listen on a network port directly with the splunk forwarder and forward received events to the indexers. In this case there is no "retention" as such as you're only ingesting the data as it's being sent to you directly.

The other one is to use an intermediate layer of some syslog solution (rsyslog, syslog-ng, maybe something else) to receive event sent from network and "repack" them and send into splunk input (possibly a HEC one). In this case, of course the syslog daemon can also write the events to local log files but it's up to the syslog solution configuration as well as your local logrotate or similar tool to clean up the logs.

The third possible approach is also to use some syslog daemon but make it write to local files and then read events from files with the forwarder. Here also it's up to external tools (i.e. logrotate) configuration to make sure that the logs are removed after some time.

[pacifikn]

Thank you @PickleRick  for your response, 

What I mean Syslog receiver , Splunk syslog receiver is the opt/ directory where we receive data from different source, but this store logs for three days, AND we want to adjust it and store only data for 1 day, not keep storing data from previous days, we want after being parsed and indexed to indexers we want that this data remain in splunk syslog server for day ago to be deleted not keep saved in the opt/syslog/catch-all directory?

For now if we check in directory where we receive data from different sources, we contains the data for 3days and this cause the space to run out, so we want to free space and configure/or make script so that it will only save/store the current day only not days ago?

How we can do that? kindly guide me with the command?

Thank you in advance

[PickleRick]

Honestly, I have no idea what kind of syslog daemon is in your system installed under /opt. Typically, system-wide syslog is configured with files in /etc and holds files in /var (especially /var/log). So if you have other solution which saves files in /opt, sorry, it's up to whoever installed it there and knows how it works.

[Stefanie · Accepted solution]

Sure, I can add comments to your script and it should make sense. This is not where the retention of logs are made. However, it is part of it. It looks like there is something somewhere which makes this script run at the current interval it is now. I think you mentioned every two days.

Since you have found a script, there is a high chance there is a cron job that is calling this script.

#!/bin/bash


#Uses the find command to search in /opt/syslog/ for files that end in .log. It then removes the files as Splunk

/bin/find "/opt/syslog/" -type f -name \*.log -mtime +3 -exec rm {} \;
su splunk <<'EOF'


#Uses the find command in /opt/syslog/ to look for directories and redirects what it finds to /opt/syslog/current_folders

find "/opt/syslog/" -type d > "/opt/syslog/current_folders"


#This grep command looks for files that are not in the past_folders but are in the current_folders. It then makes a list of the new_devices that have sent logs to your syslog.

grep -Fxv -f "/opt/syslog/past_folders" "/opt/syslog/current_folders" >> "/opt/syslog/new_devices.log"


#After it gets it's list. It copies over the current_folders to the past_folders

cp -f "/opt/syslog/current_folders" "/opt/syslog/past_folders"


#It then makes a list of the removed_devices 

find "/opt/syslog/" -empty -type d >> "/opt/syslog/removed_devices.log"
EOF

[pacifikn]

Thank you so much  @Stefanie for your kind response and explanation,

May you please help me and share with me the command to locate the configuration where retention is?

and show me the way I can do this to change to only receive data from different sources AND only store in syslog server receiver the data for only one day, because currently it store the received data for two days then it deleted when another day added.

i.e. when this day end it will delete the stored data for wednesday(13/10/2021.log) and remain with the data for two days(14/10/2021.log and 15/10/2021.log) and so on, 

Kindly help :

- How to find this config where it is located?

-  how to make it so that it could only receiving data for current day not keeping storing the data for yesterday because it is already indexed in indexers, no need to continue storing data for previous day in syslog receiver.

Help me how I can do this, Thank you in advance.

[Stefanie]

---

- How to find this config where it is located?

 The configuration for your retention seems to be a script that is ran from cron (a "service" on your server that runs things on a frequency you specify). I could be wrong but that is likely. Did you run the commands I mentioned from my first reply?

There could also be that the script is ran by Splunk. In that case, you might have an app that is named something related to Syslog. It all depends.

---

-  how to make it so that it could only receiving data for current day not keeping storing the data for yesterday because it is already indexed in indexers, no need to continue storing data for previous day in syslog receiver.

 You're going to need to find where this setting for the frequency to run that script. What's the name of that script? Where is it located? That should give you some clues.

[Stefanie]

You can configure your syslog server to rotate daily. 

Depending on how your server is set up, you can check the settings for it in cron. Try a command like 

ls /etc/cron.daily/ | grep logrotate

You can also check your settings in /etc/logrotate.d/rsyslog .

For more information, google search " How to setup and manage log rotation using logrotate in Linux"

[pacifikn]

Dear @Stefanie , This is what i get when run the command you specified,

ls /etc/cron.daily/ | grep logrotate

output:     logrotate

and when I read content gives me this:

# cat logrotate
#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

AND when I list the cron.daily directory:

[root xxx   cron.daily]# ls

output this below:  
delete_old_syslog_data.sh     logrotate      man-db.cron        mlocate

I can't find where this config that delete data received in Splunk instance /opt directory is located, if there's any command that do this or if there is any extension file that are common in this exercices maybe we can locate it,

OR you can help me with the command as you understand my scenario, what are the command or other way to do so that you can configure to only storing the received logs for the current day only in /opt/syslog/catch-all directory that I receive logs from different sources AND avoid keep storing in this opt/syslog/catch-all directory the data for the days ago? I only want to receive data then after being indexed , I don't want them to continue being stored in this opt/syslog/catchall directory I want that the data can be stored only 1 day and when it pass for midnight it can be deleted,  how to do this ? Thank you in advance

[Stefanie]

try running the command:

crontab -e

Do you see anything there?

[pacifikn]

Dear @Stefanie  after running crontab -e 

Nothing found in the editor

no root ...

[Stefanie]

I gave it some more thought and I think that I've might have found your issue. In the script you posted, in the first command:

/bin/find "/opt/syslog/" -type f -name \*.log -mtime +3 -exec rm {} \;
su splunk <<'EOF'

Change the mtime to +0. This will delete files over 1 day old. If you want to be safe you can set it to +1 before setting it to +0. Don't forget to make a backup.

Your script should look like this 

#!/bin/bash
/bin/find "/opt/syslog/" -type f -name \*.log -mtime +0 -exec rm {} \;
su splunk <<'EOF'
find "/opt/syslog/" -type d > "/opt/syslog/current_folders"
grep -Fxv -f "/opt/syslog/past_folders" "/opt/syslog/current_folders" >> "/opt/syslog/new_devices.log"
cp -f "/opt/syslog/current_folders" "/opt/syslog/past_folders"

find "/opt/syslog/" -empty -type d >> "/opt/syslog/removed_devices.log"
EOF

[pacifikn]

Greetings!!!

Thank you so much @Stefanie  for your prompt response, but I still confused, 

I got where the script is ,but it seems  a little hard to understand, can you help me understand the below script ?

#!/bin/bash
/bin/find "/opt/syslog/" -type f -name \*.log -mtime +3 -exec rm {} \;
su splunk <<'EOF'
find "/opt/syslog/" -type d > "/opt/syslog/current_folders"
grep -Fxv -f "/opt/syslog/past_folders" "/opt/syslog/current_folders" >> "/opt/syslog/new_devices.log"
cp -f "/opt/syslog/current_folders" "/opt/syslog/past_folders"

find "/opt/syslog/" -empty -type d >> "/opt/syslog/removed_devices.log"
EOF

Kindly help me understand the above script , if it is where the retention of logs received are made , if so how can I reduce the days to only receive one day and delete the previous  day and only receive today logs not yesterday and so on.. thank you