Reporting

Scheduled report returning incomplete results

dfurtaw
Path Finder

A scheduled report runs once every sunday and does not return all of the data that we need. I fear that it is hitting a quota or limit but I'm unable to find any noteworthy information after inspecting the job. This report takes 00:57:10 to run.

More context: The report shows a stats table and list events by chronological order for the previous week, it looks like most of the results returned were the most recent, it doesn't want to return the older events (longer than 2 days into the report). If I run an ad hoc search on results for a smaller time range, I return results that I should be seeing in the report.

Error I'm seeing in the search.log (although I see this in reports that complete)
01-05-2020 08:20:44.434 INFO PipelineComponent - Process delayed by 3429.056 seconds, perhaps system was suspended?

Any suggestions on troubleshooting techniques?

Events: 15,891,316 5.01 MB

Query:

(index=wineventlog sourcetype=wineventlog:security EventCode=numberofcode
[| inputlookup nameofcsv.csv
| fields Account_Name]) OR (index=linux sourcetype=linux_secure
[| inputlookup nameofcsv.csv
| rename Account_Name as user
| fields user]) OR (index=indexname sourcetype=sourcetypename authentication_category=login
[| inputlookup nameofcsv.csv
| rename Account_Name as src_user
| fields src_user]) OR (index=mssql sourcetype=mssql:audit
[| inputlookup nameofcsv.csv
| rename Account_Name as server_principal_name
| fields server_principal_name]) OR (index=indexname sourcetype=sourcetypename
[| inputlookup nameofcsv.csv
| rename Account_Name as user
| fields user])
| rename Account_Name as tmp_user user as tmp_user src_user as tmp_user server_principal_name as tmp_user
| mvexpand tmp_user
| search
[| inputlookup nameofcsv.csv
| rename Account_Name as tmp_user
| fields tmp_user]
| eval var_user=lower(tmp_user)
| stats max(_time) AS Last_Login by var_user
| convert ctime("Last_Login")
| lookup nameoflookup Account_Name as var_user OUTPUTNEW dn

0 Karma
1 Solution

woodcock
Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

View solution in original post

0 Karma

woodcock
Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

View solution in original post

0 Karma

woodcock
Esteemed Legend

OK, so which was it?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Sharing the full query would be most helpful.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

dfurtaw
Path Finder

Hey Rich - went ahead and updated the question with a query.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!