Reporting

missing ~49 minutes of events out of 1 hour scheduled cron job with savedsearches

Helios
Loves-to-Learn Everything

wanted to reach out for help regarding an issue we have been experiencing on one of our customers.  We build an app that exports events from a standalone customer using the Splunk Enterprise instance.  We have that box gather the logs and hold them until it can be exported out of the box manually.

We used the savedseaches.conf file to schedule a search query script (export.py) to pull events.  The problem is that on this particular customer he is only getting like 11 minutes worth of logs, but the file is scheduled to pull all index events from lets say 3:30pm-4:30pm, but the events start loading only from 4:19pm-4:30pm.   It does this across all times consistently.

example, missing the first like 49 minutes of events: 

4:19pm-430pm

5:19pm-5:30pm

6:19pm-6:30pm

 We have a export.py script that goes out and gathers all index=* events according to the cron specified.

savedsearches.conf

cron_schedule = 30 */1 * * * 

enablesched = 1

dispatch.ttl = 1800

allow_skew = 10m

search = | export

disable = 0

 To compensate for lags, we build into the |export.py script to pull the events 1 hour prior so like.  This is part of the script dealing with the specific search.

now = str(time.time()-3600).split(".")[0]

query = "search index=* earliest=" + last_scan + "  lastest=" + now + " 

once script is done, it creates a timestamp in a file of the now in epoch time, which is used for the next schedule time.

Any help would be appreciated

Labels (2)
0 Karma

Helios
Loves-to-Learn Everything

index=* OR index=_* earliest=" + last_scan + " lastest =" + now + " | append [ | mpreview index=*_metrics ] | fields - _bkt, _cd, _serial, _si

0 Karma

Helios
Loves-to-Learn Everything

forgot to put the query currently running

index=* OR index=_* | append [ | mpreview index=*_metrics ] | fields - _bkt, _cd, _serial, _si

0 Karma
Get Updates on the Splunk Community!

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...

Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation

As of Splunk Cloud Platform 9.3.2408 and Splunk Enterprise 9.4, classic dashboard export features are now ...