Monitoring Splunk

Collect jemalloc heap data to troubleshoot high memory growth contributed by search process

keio_splunk
Splunk Employee
Splunk Employee

Splunk search process is consuming high memory causing splunkd to crash. How to collect jemalloc heap data for Splunk Technical Support to troubleshoot high memory growth issues contributed by the specific search process? 

 

Labels (3)
0 Karma
1 Solution

keio_splunk
Splunk Employee
Splunk Employee

1. Start the search process. 

2. Run top command to list the PID of the search process consuming high memory.

3. Run ps command on the search process PID to list the search artifact directory.

 

ps -ef | grep <PID>

 

4. Go to the search artifact directory path and execute the "touch save" command to be sure that it is not reaped by the dispatch reaper.

 

cd $SPLUNK_HOME/var/run/splunk/dispatch/<sid>; touch save

 

5. Create a directory to save the jemalloc heap data.

 

mkdir -p /tmp/heap

 

6. Enable jemalloc heap collection from the search artifact directory path.

 

MALLOC_CONF="prof:true,prof_accum:true,prof_leak:true,lg_prof_interval:28,prof_prefix:/tmp/heap/heap_data" $SPLUNK_HOME/bin/splunk cmd splunkd search $(cat args.txt)

 

7. Stop the search once the issue has been reproduced. 

8. Tar up the heap data files using the tar command. 

 

tar cvzf heap.tar.gz /tmp/heap

 

9. Tar up search artifact folder. 

 

tar cvzf search-artifact.tar.gz $SPLUNK_HOME/var/run/splunk/dispatch/<sid>

 

View solution in original post

0 Karma

keio_splunk
Splunk Employee
Splunk Employee

1. Start the search process. 

2. Run top command to list the PID of the search process consuming high memory.

3. Run ps command on the search process PID to list the search artifact directory.

 

ps -ef | grep <PID>

 

4. Go to the search artifact directory path and execute the "touch save" command to be sure that it is not reaped by the dispatch reaper.

 

cd $SPLUNK_HOME/var/run/splunk/dispatch/<sid>; touch save

 

5. Create a directory to save the jemalloc heap data.

 

mkdir -p /tmp/heap

 

6. Enable jemalloc heap collection from the search artifact directory path.

 

MALLOC_CONF="prof:true,prof_accum:true,prof_leak:true,lg_prof_interval:28,prof_prefix:/tmp/heap/heap_data" $SPLUNK_HOME/bin/splunk cmd splunkd search $(cat args.txt)

 

7. Stop the search once the issue has been reproduced. 

8. Tar up the heap data files using the tar command. 

 

tar cvzf heap.tar.gz /tmp/heap

 

9. Tar up search artifact folder. 

 

tar cvzf search-artifact.tar.gz $SPLUNK_HOME/var/run/splunk/dispatch/<sid>

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...