This article is a bit old (2010) but it's high-level enough to give you an idea of what might be happening.
What is the memory usage on your Splunk server? Is it regularly paging to virtual memory? From the description of the error, Splunk may determine that it's running short on memory, so will attempt to conserve memory by parsing time-stamps only down to the second when running a search. This may not actually affect you (depending on the granularity of the timestamps in your events), but is definitely indicative of an issue that needs investigation.
EDIT: There's also this article on Memory Pressure - same recommendation (more RAM).
Actually, it's barely touching the 48G of RAM in the box. I suspect it's the per-search configured limits (which are default), and the search is returning hundreds of thousands or millions of results when I get this error.
Hi jtrucks, have you made a tuning of this parameter? Which value is ok for your configuration? Thanks
This is a message indicating that we are throttling the rate at which a search process reads event raw data (the
_raw field) to keep the memory usage of the search process low.
We've had issues of high memory usage with searches encountering patches of large (~10KB or more)
_raw fields, so when that happens, we slow down the rate at which we read those events.
That being said, we also discovered that:
We are adjusting both of those things in a future release.
In the meantime, if you have enough memory available that you feel comfortable allowing searches to use more of it, you can increase this threshold by changing the value of
max_rawsize_perchunk in limits.conf:
max_rawsize_perchunk = <integer> * Maximum raw size of results per call to search (in dispatch). * 0 = no limit. * Defaults to 100000000 (100MB) * Not affected by chunk_multiplier
hi hexx, how did you slow down the rate at which events are read?
if i understood correctly, with "maxrawsizeperchunk" you basically increase the memory that the search process can use, but how do you tune the read rate? thanks!
We have an app which is dumping approximately 100,000 events at a time into the index, all with the same timestamp, and each event is quite large, many with hundreds of lines. When I would receive the above error, the search would perform very slowly, taking 30+ minutes to finish. I started slowly increasing the size of maxrawsizeperchunk, without any change, and then jumped up to default*10, putting it to 1,000,000,000. The error disappeared, the search completed in 3 minutes, and there was minimal noticeable memory and CPU usage on the search head/indexer (the same machine).
Default*10 worked well for me, so far.
In my own testing I'm suspecting that this is due to large numbers of events with the exact same timestamp -- possibly caused by un-timestamped events that are being timestamped by Splunk as they are indexed. This was not an issue in version 4.3.x .. it just started occurring in version 5.0.3. Setting the maxrawsizeperchunk may alleviate the symptom but is not addressing the issue.
Splunk 6.1.1, I ran across this for the first time when I indexed a new source type with larger events than my other source types. I have some events that are many Kilobytes in this source type, and there aren't that many. Increasing the maxrawsizeperchunk resolved it and doesn't appear to have caused any change in the memory load that I can tell.