SbrkSysAllocator failed

[ttrolf]

Hi,

since we upgraded to Splunk 4.3 from 4.2 we keep seeing this WARN coming up:

01-13-2012 14:49:35.988 +0100 WARN  ProcessDispatchedSearch - PROCESS_SEARCH src/system-alloc.cc:423] SbrkSysAllocator failed.

We see this popping also in Splunkd.log before the former message kicks in:

01-13-2012 14:44:05.902 +0100 WARN  DistributedBundleReplicationManager - bundle replication to 2 peer(s) took too long (13054ms), bundle file size=28390KB, replication_id=1326462232

We run Splunk 64bit on RHEL5 64bit Linux.

UPDATE: The server seems to allocate large amounts of memory, up to 10Gb and lots of swap space too. Is this some memory bug?

[hodsonc]

I just got this error while running fsck.

I upgraded to 4.3 and after doing the indexer it told me I should run an fsck on the indexes. After shutting down splunk and starting the fsck I got the above error.

  $ /opt/splunk/bin/splunk fsck --mode metadata --all --repair
bucket=/opt/splunk/var/lib/splunk/_internaldb/db/db_1327351718_1327338541_3791 file='/opt/splunk/var/lib/splunk/_internaldb/db/db_1327351718_1327338541_3791/Sources.data' code=16 contains recover-padding, repairing...
src/system-alloc.cc:423] SbrkSysAllocator failed.

other buckets then continue to be worked on.

[hexx]

This might be a different problem, I would suggest that you ask a new question. Some of the stuff we'll want to find out :

• Does this happen consistently and always with the same bucket?
• How much physical and virtual memory is available on your system at the time that you get the error?

[dwaddle · Accepted solution]

It's possible you are seeing this issue: http://splunk-base.splunk.com/answers/37809/halp-consulting-the-summary-dashboard-of-the-search-app-... . Can you please follow its advice and report back? Thanks.

[ttrolf]

Hi following the link you sent me duckfez solved the issue, thank you! Awaiting Splunk 4.3.1 then.

[hexx]

I agree with @duckfez I believe the message you are reporting is merely a symptom of a memory exhaustion problem. If you'd like to see if Splunk is responsible for that, I would suggest to install the Splunk on Splunk app and to turn on the ps_sos.sh scripted input to track memory usage by Splunk process (splunkd, splunkweb, searches).