This is an ancient thread, but bears updating. As of 9.1.x, this is what I see: Default is 10K lines.  You need to change it in alert_actions.conf.  Any change there will propagate to "Advanced Edit" for a saved search; there is no need to change the value there. If the mail doesn't come through, look through splunkd logs for "Message size exceeds fixed limit" as many mailers reject attachments larger than a given size (10M is typical) ... View more

This is an ancient thread, but bears updating. As of 9.1.x, this is what I see: Default is 10K lines.  You need to change it in alert_actions.conf.  Any change there will propagate to "Advanced Edit" for a saved search; there is no need to change the value there. If the mail doesn't come through, look through splunkd logs for "Message size exceeds fixed limit" as many mailers reject attachments larger than a given size (10M is typical) ... View more

Don't you mean | rename licenseGB as GB ... View more

This is incorrect; the text above is from an SPL2 page, not an SPL page.  The correct info is this: Proper field name syntax Field names must conform to the field name syntax rules. Valid characters for field names are a-z, A-Z, 0-9, . , :, and _. Field names cannot begin with 0-9 or _ . Leading underscores are reserved for Splunk Enterprise internal variables. which can be found here: https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Aboutregularexpressionswithfieldextractions ... View more

Sigh. The problem with this solution is that the flow log JSON events can be massive (>500K) and spath /mvexpand can't handle it.  Also the best way to use this data would be for each individual flow log entry to have all associated data (Rule included).  There are some other techniques discussed in this 2020 blog post that use an Azure Function to send HEC events. https://www.splunk.com/en_us/blog/platform/splunking-azure-nsg-flow-logs.html ... View more

You were absolutely right; unprintable characters.  cat -v of the savedsearches.conf file revealed the problem, and a few tries in the GUI I was able to reset the search to remove the characters.  Thanks! ... View more

I realize this is an old thread; but I just hit the same problem with sendresults, splunk 8.1.5, sendresults 5.0.3. I was using a long comma separated list of email addresses in email_to. Saw errors like: " error="'ascii' codec can't encode character '\u202c' in position 29: ordinal not in range(128)" The fix was to put a semicolon at the end of the email_to value. So  [email protected],[email protected],[email protected];   ... View more

We had this problem on our monitoring console after an upgrade to 8.1.5. Root cause was the DMC Alert - Missing forwarders search - in our very large environment, it was running every 15 minutes and generating 250M of results. This search is set to keep results for 24 hours.  So that's 24G of results in a day.  Changed the settings to keep results for only one hour, and the solved the problem. In any case, look at the "jobs" on the host and sort by size. You may see something similar - a frequently running job that is saving all results for too long. ... View more

I'm stuck here - do I have to get some certificate from my Azure portal and append it to this file?  I'm getting this same error stream that ends in "KeyError: 'account_class_type not exists'" ... View more

I don't believe this is correct anymore for current releases of Splunk.  The embed URL has changed drastically. ... View more

This is the solution from Splunk as well:   https://splunkcommunities.force.com/customers/apex/ArticleDetailPage?URLName=Scheduled-Search-Alert-not-Triggering-Emails     ... View more

As of now we have not had to repeat the process;  I think the problem existed for a very long time before we detected it.  Latest from Splunk Support is a patch will be in a 7.3.x version for this issue. ... View more

My approach would be to blow them all away. They are regenerated nearly immediately, there is no side-effect or problem with blowing them away, and it's easy. Note that there are some manifest.csv files that do not exist under the datamodel_summary directories, so it's important not to remove those. If you have concerns you can move them sideways rather than delete them, but I'm not sure that works as well - if splunk has a filehandle open on the file, it may update the moved files. find $SPLUNK_HOME/var/lib/splunk/\*/datamodel_summary -type f -name manifest.csv -exec rm {} \; ... View more

See my answer below. I think this is the root cause. Would be curious to see if you have similar experience. ... View more

This problem appears to be connected to out of date and/or incorrect manifest.csv files that live under the datamodel_summaries directories in every index directory. Remove those files with a find/exec rm. Splunk will rebuild this file in a few minutes, and it does not require a restart of the indexer or any cluster maint mode. The error message should then cease. If it doesn't, try removing them a second time. It also led, in our case, to incomplete data models due to Splunk thinking that the datamodel volume sizes were, in some cases 10x their actual size. Here's SPL to find this error. Note the two spaces between "WARN and CMslave" index=_internal sourcetype=splunkd "WARN CMSlave" *datamodel_summary* It seems that it can take multiple removals of the manifest.csv files to persuade some indexers to rewrite them correctly. This rest call shows _splunk_summaries volume sizes: | rest splunk_server=<something to identify your indexers> /services/data/index-volumes | fields splunk_server, title, total_size, max_size, volume_path |search title=_splunk_summaries | eval total_size_gb = if(isnull(total_size), "-", round(total_size / 1024, 2)) | eval max_size_gb = if(isnull(max_size) OR max_size = "infinite", "unlimited", round(max_size / 1024, 2)) | eval disk_usage_gb = total_size_gb | fields splunk_server, title, disk_usage_gb, max_size_gb, volume_path | sort - disk_usage_gb | rename title as Volume, disk_usage_gb as "Volume Usage (GB)", max_size_gb as "Volume Capacity (GB)", volume_path as "Volume Path" ... View more

Any updates to this issue? I'm seeing this - I think it relates to incorrect volume sizes reported for _splunk_summaries and missing data model information. ... View more

This also seems to happen if your client secret is incorrect. ... View more

My observation is the timestamps of all the directories in srtemp realate to when splunkd was shut down or crashed (or was killed by an OS out of memory kill process). So in general a find command such as: find $SPLUNK_HOME/var/run/splunk/srtemp -type d -mtime +30 | xargs rm -rf would be a fine thing to run weekly on any highly used Splunk search head. ... View more

In a similar vein, if you are not using a stats comment, you can simply append / makeresults to create a dummy result to feed to | map. | append [| makeresults | eval siteID="DUMMY"] ... View more

Yessir! Done and done! ... View more

Exactly right Martin, thank you. <td data-cell-index="1" class="string" tabindex="> Andrew Gerber </td> ... View more

Yes, I believe so... using a different account, or I would edit it. Corrected below: | rest /services/data/indexes | search totalEventCount > 0 | eval now=strftime(now(), "%Y-%m-%d") | stats first(minTime) as "Earliest Event Time" first(maxTime) as "Latest Event Time" first(now) as "Current Date" first(currentDBSizeMB) as currentDBSizeMB by title | rename title as "Index" | sort - currentDBSizeMB | eval "Index Size in GB"= round(currentDBSizeMB/1000,2) | table Index "Earliest Event Time" "Latest Event Time" "Current Date" "Index Size in GB" ... View more

The code contains several references to a person with email "[email protected]"; I've emailed him with this comment as well. (If anyone from Qualys reads this). The problem I see in the current add-on is it overwrites the qualys_kb.csv lookup file with a zero length file on failure. It would make more sense if it would abort and keep the original copy on failure, or at least make a backup. ... View more

Also a problem with 1.2.3. Solution mentioned below fixes it fine. ... View more