Splunk Alert only includes first 1000 results of search. Why?

[the_wolverine]

Search head is version 4.1.5. I have a scheduled search that emails the report of all searches run in the past 7 days. The scheduled search runs just fine, the text of the alert indicates that several thousands results were returned by the search but the csv file only contains 1000 results.

Is this a simple configuration issue? What needs to be configured to increase this number to say, 10000 ? What are the ramifications of increasing said number?

[the_wolverine]

I found the following entry in default/alert_actions.conf:

maxresults = * Set the global maximum number of search results sent via alerts. * Defaults to 100.

It seems to be a typo as the the maxresults is really 1000. Still, after setting this to 50000, I'm only seeing 1000 results in my alerts when there should be more.

Note, my changes were made on a search head (version 4.1.5). Do I also need to update the indexers directly?

[the_wolverine]

It appears to be working now. Looks like partial PEBKAC (did not restart Splunk?) and partial bug in documentation. Thanks for your assistance, Lowell.

[Lowell]

I update my original answer (not sure if you saw that or not). But I think support is really your only option at this point. Something very strange is going on here.

[the_wolverine]

Thanks for the suggestion, Lowell. I haven't tried the command edit but this appears to be a bug so I'll followup with support on it.

[Lowell]

I'm thinking that the search head is the correct location. Have you tried bumping the global value in alert_actions.conf and seeing if that makes a difference? Or even hard-coding a higher value on the "command" entry.

[Lowell · Accepted solution]

Looks like you can change this with the following config entry:

alert_actions.conf:

[email]
maxinputs = 10000

Unfortunately, it looks like this is a global value so all email alerts would be affected by this change. (I suppose you could also make this a per-user or per-app setting with the right meta settings, so I guess my point is really that it's not per-savedsearch.)

---

If you wanted to be tricky (and this could break in newer releases), you could update the email action's "command" to use a setting associated with individual saved searches, like this:

alert_actions.conf:

[email]
command = $action.email.preprocess_results{default=""}$ | sendemail "to=$action.email.to$" "server=$action.email.mailserver{default=localhost}$" "from=$action.email.from{default=splunk@localhost}$" "subject=$action.email.subject{recurse=yes}$" "format=$action.email.format{default=csv}$" "sssummary=Saved Search [$name$]: $counttype$($results.count$)" "sslink=$results.url$" "ssquery=$search$" "ssname=$name$" "inline=$action.email.inline{default=False}$" "sendresults=$action.email.sendresults{default=False}$" "sendpdf=$action.email.sendpdf{default=False}$" "pdfview=$action.email.pdfview$" "searchid=$search_id$" "graceful=$graceful{default=True}$" maxinputs="$action.email.maxinputs{default=1000}$" maxtime="$action.email.maxtime{default=5m}$"

(I simply changed "$maxinputs" to "$action.email.maxinputs", which makes this setting per-savedsearch instead of global. So then you could update your saved search like this:

savedsearches.conf

[my_big_export_saved_search]
action.email = 1
action.email.sendresults = 1
action.email.to = users@example.com
action.email.maxinputs = 10000
...

---

Update:

After looking over the sendemail.py code, I don't see any references to the maxinputs or maxtime command line arguments. So I don't think they are used at all. I don't see any hard-coded limits set within this script either (other than an email server socket timeout.) So I'm wondering if perhaps "maxinputs" and "maxtime" are left overs from a previous version? So I'm now thinking that my solution suggested above will not work at all.

Per the docs (and the Wolverine's comments below), it sounds like the total amount of results is actually capped by the maxresults variable set in the alert_actions.conf file. In my system/default/alert_actions.conf, this is in fact set to 100 on my system, just as alert_actions.conf.spec indicates. However, this doesn't make any sense either. If the alerting subsystem is really using this value, then how are you seeing 1000 entries in your email alert? You should only be getting 100, based on this setting. Furthermore, since maxresults is set globally (before any named stanza), it would be applied to all stanzas and therefore alert actions such as summary_index and populate_lookup would also be subject to this limit. But that doesn't seem right. I know I have summary index generating saved searches that use the "summary_index" alerting mechanism, and they generate more than 100 (probably more than 1000) events within a single search.

This is weird, I can't figure out what's going on here. My best conclusion is that while the alerting process seems to be setup in a generic and mostly reusable way, there seems to be some hard-coded voodoo going on behind the scenes. And for whatever reason, that voodoo wants to keep your from sending more than 1,000 events in your email.

I'm guessing that the "script" alerting action has a bigger limit than this. You could probably use that to build your own email-sending mechanism as a work around; but that's no fun.

[the_wolverine]

I thought that this was working but its not for me in 4.1.5 (setting maxinputs). My resultset is still capped at 1000 results.

Further, I don't even see that "maxinputs" exists in the alert_actions.conf.spec file.