Reporting

Scheduled report returning incomplete results

dfurtaw
Path Finder

A scheduled report runs once every sunday and does not return all of the data that we need. I fear that it is hitting a quota or limit but I'm unable to find any noteworthy information after inspecting the job. This report takes 00:57:10 to run.

More context: The report shows a stats table and list events by chronological order for the previous week, it looks like most of the results returned were the most recent, it doesn't want to return the older events (longer than 2 days into the report). If I run an ad hoc search on results for a smaller time range, I return results that I should be seeing in the report.

Error I'm seeing in the search.log (although I see this in reports that complete)
01-05-2020 08:20:44.434 INFO PipelineComponent - Process delayed by 3429.056 seconds, perhaps system was suspended?

Any suggestions on troubleshooting techniques?

Events: 15,891,316 5.01 MB

Query:

(index=wineventlog sourcetype=wineventlog:security EventCode=numberofcode
[| inputlookup nameofcsv.csv
| fields Account_Name]) OR (index=linux sourcetype=linux_secure
[| inputlookup nameofcsv.csv
| rename Account_Name as user
| fields user]) OR (index=indexname sourcetype=sourcetypename authentication_category=login
[| inputlookup nameofcsv.csv
| rename Account_Name as src_user
| fields src_user]) OR (index=mssql sourcetype=mssql:audit
[| inputlookup nameofcsv.csv
| rename Account_Name as server_principal_name
| fields server_principal_name]) OR (index=indexname sourcetype=sourcetypename
[| inputlookup nameofcsv.csv
| rename Account_Name as user
| fields user])
| rename Account_Name as tmp_user user as tmp_user src_user as tmp_user server_principal_name as tmp_user
| mvexpand tmp_user
| search
[| inputlookup nameofcsv.csv
| rename Account_Name as tmp_user
| fields tmp_user]
| eval var_user=lower(tmp_user)
| stats max(_time) AS Last_Login by var_user
| convert ctime("Last_Login")
| lookup nameoflookup Account_Name as var_user OUTPUTNEW dn

0 Karma
1 Solution

woodcock
Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

View solution in original post

0 Karma

woodcock
Esteemed Legend

It should not be that it being scheduled has anything to do with it. Are you sure that it does? Some thing to do/check.
1: You should always be setting schedule windows for your saved searches as wide as possible to allow the scheduler to de-clump them.
2: Try not to use subsearches, join, or transaction because these do not scale well.
3: If you are using sort, be sure to ALWAYS use a number after it, otherwise it truncates (e.g. do sort 0 foo).
4: Check for latency (difference between _time and _indextime); it could be that events are arriving late and you will have to either fix this or shift your time windows backwards and run it later.

0 Karma

woodcock
Esteemed Legend

OK, so which was it?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Sharing the full query would be most helpful.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dfurtaw
Path Finder

Hey Rich - went ahead and updated the question with a query.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...