Collect jemalloc heap data to troubleshoot high memory growth contributed by search process

[keio_splunk]

Splunk search process is consuming high memory causing splunkd to crash. How to collect jemalloc heap data for Splunk Technical Support to troubleshoot high memory growth issues contributed by the specific search process? 

[keio_splunk · Accepted solution]

1. Start the search process. 

2. Run top command to list the PID of the search process consuming high memory.

3. Run ps command on the search process PID to list the search artifact directory.

ps -ef | grep <PID>

4. Go to the search artifact directory path and execute the "touch save" command to be sure that it is not reaped by the dispatch reaper.

cd $SPLUNK_HOME/var/run/splunk/dispatch/<sid>; touch save

5. Create a directory to save the jemalloc heap data.

mkdir -p /tmp/heap

6. Enable jemalloc heap collection from the search artifact directory path.

MALLOC_CONF="prof:true,prof_accum:true,prof_leak:true,lg_prof_interval:28,prof_prefix:/tmp/heap/heap_data" $SPLUNK_HOME/bin/splunk cmd splunkd search $(cat args.txt)

7. Stop the search once the issue has been reproduced. 

8. Tar up the heap data files using the tar command. 

tar cvzf heap.tar.gz /tmp/heap

9. Tar up search artifact folder. 

tar cvzf search-artifact.tar.gz $SPLUNK_HOME/var/run/splunk/dispatch/<sid>