Reporting
Highlighted

Scheduled report returning incomplete results

Path Finder

A scheduled report runs once every sunday and does not return all of the data that we need. I fear that it is hitting a quota or limit but I'm unable to find any noteworthy information after inspecting the job. This report takes 00:57:10 to run.

More context: The report shows a stats table and list events by chronological order for the previous week, it looks like most of the results returned were the most recent, it doesn't want to return the older events (longer than 2 days into the report). If I run an ad hoc search on results for a smaller time range, I return results that I should be seeing in the report.

Error I'm seeing in the search.log (although I see this in reports that complete)
01-05-2020 08:20:44.434 INFO PipelineComponent - Process delayed by 3429.056 seconds, perhaps system was suspended?

Any suggestions on troubleshooting techniques?

Events: 15,891,316 5.01 MB

Query:

(index=wineventlog sourcetype=wineventlog:security EventCode=numberofcode
[| inputlookup nameofcsv.csv
| fields AccountName]) OR (index=linux sourcetype=linuxsecure
[| inputlookup nameofcsv.csv
| rename AccountName as user
| fields user]) OR (index=indexname sourcetype=sourcetypename authentication
category=login
[| inputlookup nameofcsv.csv
| rename AccountName as srcuser
| fields srcuser]) OR (index=mssql sourcetype=mssql:audit
[| inputlookup nameofcsv.csv
| rename Account
Name as serverprincipalname
| fields serverprincipalname]) OR (index=indexname sourcetype=sourcetypename
[| inputlookup nameofcsv.csv
| rename AccountName as user
| fields user])
| rename Account
Name as tmpuser user as tmpuser srcuser as tmpuser serverprincipalname as tmpuser
| mvexpand tmp
user
| search
[| inputlookup nameofcsv.csv
| rename AccountName as tmpuser
| fields tmpuser]
| eval var
user=lower(tmpuser)
| stats max(
time) AS LastLogin by varuser
| convert ctime("LastLogin")
| lookup nameoflookup Account
Name as var_user OUTPUTNEW dn

0 Karma
Highlighted

Re: Scheduled report returning incomplete results

SplunkTrust
SplunkTrust

Sharing the full query would be most helpful.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Scheduled report returning incomplete results

Path Finder

Hey Rich - went ahead and updated the question with a query.

0 Karma
Highlighted

Re: Scheduled report returning incomplete results

Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

View solution in original post

0 Karma
Highlighted

Re: Scheduled report returning incomplete results

Esteemed Legend

OK, so which was it?

0 Karma