Getting Data In

Calling a sequence of commands

tristanmatthews
Path Finder

Hi,

I'm taking over splunk management for a company I just joined and have found some errors in the way the data was being processed. Consequently I've needed to reindex the whole thing.

Is there any easy way to schedule a sequence of commands to run one after the other? I'd like to step through the whole pipeline as if the normally daily cron jobs are being run day1, day2, day3... the way they were schedule, but I'm not seeing an easy way to do that. I mean I can schedule them to run more often, but I don't see an easy way to set the time ranges, that way.

Thanks,
Tristan

Tags (2)
0 Karma
1 Solution

grijhwani
Motivator

1) You don't specify what platform you are running on, or your Splunk version.

2) You don't specify if you mean the Splunk internal cron-style scheduling, or if the answer to 1) is some unix-style o/s system cron jobs.

3) The question is vague as to what you are trying to index.

If this is a unix-based installation you should be able to use awk or perl to separate your data into daily chunks and then run Splunk one-shot imports on each segment. Of course if your time reference is not inherent in the log data you can only approximate.

You probably want to be looking at
Documentation > Splunk Enterprise > Getting Data In > Use the CLI and looking for the oneshot sub-command

View solution in original post

grijhwani
Motivator

1) You don't specify what platform you are running on, or your Splunk version.

2) You don't specify if you mean the Splunk internal cron-style scheduling, or if the answer to 1) is some unix-style o/s system cron jobs.

3) The question is vague as to what you are trying to index.

If this is a unix-based installation you should be able to use awk or perl to separate your data into daily chunks and then run Splunk one-shot imports on each segment. Of course if your time reference is not inherent in the log data you can only approximate.

You probably want to be looking at
Documentation > Splunk Enterprise > Getting Data In > Use the CLI and looking for the oneshot sub-command

lukejadamec
Super Champion

You need fill_summary_index.py! I love that script.
It will backfill your data in your summary indexes, and it will do it automatically based on what ever cron schedule you like. Just tell it what search, how far back to look, what index to populate, how many concurrent searches to run at a time, who owns the search, and what the auth credentials are... It comes standard in Splunk/etc/bin.
Going forward, why not just use a cron schedule for your searches?

tristanmatthews
Path Finder

Sorry, yeah that was a bit vague.

1) Its a splunk install on Google Compute Engine Linux, Splunk version 5.0.4
2 ) The commands are saved searches for summary indexing. I'm fairly new to splunk and have just been scheduling them through the web interface under manager > Searches and Reports, which is where I found them.
3) I had splunk reindex all the raw data files, and now I'm trying to schedule the summary indexing commands, which have been built to be run in a particular order to build up new user counts and error counts based on the other summary indexes and the results of previous days.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...