Reporting

Scheduled report returning incomplete results

dfurtaw
Path Finder

A scheduled report runs once every sunday and does not return all of the data that we need. I fear that it is hitting a quota or limit but I'm unable to find any noteworthy information after inspecting the job. This report takes 00:57:10 to run.

More context: The report shows a stats table and list events by chronological order for the previous week, it looks like most of the results returned were the most recent, it doesn't want to return the older events (longer than 2 days into the report). If I run an ad hoc search on results for a smaller time range, I return results that I should be seeing in the report.

Error I'm seeing in the search.log (although I see this in reports that complete)
01-05-2020 08:20:44.434 INFO PipelineComponent - Process delayed by 3429.056 seconds, perhaps system was suspended?

Any suggestions on troubleshooting techniques?

Events: 15,891,316 5.01 MB

Query:

(index=wineventlog sourcetype=wineventlog:security EventCode=numberofcode
[| inputlookup nameofcsv.csv
| fields Account_Name]) OR (index=linux sourcetype=linux_secure
[| inputlookup nameofcsv.csv
| rename Account_Name as user
| fields user]) OR (index=indexname sourcetype=sourcetypename authentication_category=login
[| inputlookup nameofcsv.csv
| rename Account_Name as src_user
| fields src_user]) OR (index=mssql sourcetype=mssql:audit
[| inputlookup nameofcsv.csv
| rename Account_Name as server_principal_name
| fields server_principal_name]) OR (index=indexname sourcetype=sourcetypename
[| inputlookup nameofcsv.csv
| rename Account_Name as user
| fields user])
| rename Account_Name as tmp_user user as tmp_user src_user as tmp_user server_principal_name as tmp_user
| mvexpand tmp_user
| search
[| inputlookup nameofcsv.csv
| rename Account_Name as tmp_user
| fields tmp_user]
| eval var_user=lower(tmp_user)
| stats max(_time) AS Last_Login by var_user
| convert ctime("Last_Login")
| lookup nameoflookup Account_Name as var_user OUTPUTNEW dn

0 Karma
1 Solution

woodcock
Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

View solution in original post

0 Karma

woodcock
Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

0 Karma

woodcock
Esteemed Legend

OK, so which was it?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Sharing the full query would be most helpful.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dfurtaw
Path Finder

Hey Rich - went ahead and updated the question with a query.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...