Write Cron Expression to Schedule Python Scripts

[SplunkDash]

Hello,

I have 4 python scripts  to parse data that we receive in Linux machine once a day where HF has installed. Currently, I am running my python scripts manually every day  in that Linux machine to perform that task . Is there any ways, I can  write Cron Expression  to automate my python scripts so that python scripts will run automatically once a day in that Linux machine where HF has installed. Thank you so much, any help will be highly appreciated. 

[kamlesh_vaghela]

@SplunkDash 

If your script is independent and not belongs to Splunk then you should use system level crons to schedule your script. Please refer below links.

https://opensource.com/article/17/11/how-use-cron-linux

https://phoenixnap.com/kb/set-up-cron-job-linux

If your script is bundled into any Splunk app then I would suggest to setup a script as scripted input and define interval to execute script.

Please refer below links.

https://docs.splunk.com/Documentation/SplunkCloud/latest/AdvancedDev/ScriptSetup

Thanks
KV
▄︻̷̿┻̿═━一   😉

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

[SplunkDash]

Thank you so much Kamlesh_Vaghela for sending me this resourceful info, really appreciate it. 

My Cron expression is 0 23 * * * (runs every day at 11pm ). How would  I  incorporate this expression with my requirements, what file name should I use to store this, and where I need to store that, so system can read my Cron expression along with other info and runs the scripts  automatically once a day at 11pm.  Thank you again and any help will be highly appreciated. 

Here is other info:

1.  Location of my scripts in Linux machine: /opt/splunk/etc/apps/main-apps/python_scripts

2. File names : pyscriptcsv-1.py, pyscriptcsv-2.py, pyscripttxt-1.py, and pyscripttxt-2.py 

[PickleRick · Accepted solution]

This is completely out of scope of splunk administration itself. And no, argumenting that you want to run a splunk-related script doesn't make it a splunk issue. This way you could argue that creating policies for firewalls is a splunk issue because you're logging events from those policies in splunk. No, it isn't.

But I'll give you some things to consider.

Firstly, cron configuration might differ slightly depending on what cron your linux distro uses (there are different implementations around).

Secondly - it can be affected by your environment configuration policies (I've known companies which, for example, didn't allow users to create their own cron jobs).

There are also some other things regarding the scripts you run as cron jobs - logging, preventing concurrent runs (if necessary).

Oh, and remember that job spawned by cron can and probably will have a different environment variables than when you run the script from interactive shell. It might not matter much in case of python script but this issue alone has claimed many manhours of debugging across the world.

So, back to your original question - the easiest way (but ugliest) to add such cronjob is to run

crontab -e

Which will run your default $EDITOR (probably vi-clone if you haven't configured one so be warned) and will let you configure cron jobs for your user. If you want to run the script from another user, you must run the command from another user's session.

Into this file you add a line

4 23 * * *  /opt/splunk/etc/apps/main-apps/python_scripts/script1
5 23 * * *  /opt/splunk/etc/apps/main-apps/python_scripts/script2
6 23 * * *  /opt/splunk/etc/apps/main-apps/python_scripts/script3
7 23 * * *  /opt/splunk/etc/apps/main-apps/python_scripts/script4

It's not a good idea to run all scripts at once and run them at the top of the hour since many other cron jobs might be running at that time.

But it's usually a better idea not to define this as a user's job (because it's getting defined in /var/spool/cron which is not that obvious as part of system's configuration), but as a system-wide configuration. With a most-commonly crons you do it either as part of /etc/crontab or as a file in /etc/cron.d. But in this case, since it's a system-wide configuration, you need to add one more field - a username with which the command is to be executed. So if you want it run as root (definitely not recommended! You should as little as root as possible), you write it as

4 23 * * * root /opt/splunk/etc/apps/main-apps/python_scripts/script1
5 23 * * * root /opt/splunk/etc/apps/main-apps/python_scripts/script2
6 23 * * * root /opt/splunk/etc/apps/main-apps/python_scripts/script3
7 23 * * * root /opt/splunk/etc/apps/main-apps/python_scripts/script4