Monitoring Splunk

Why is the Splunk search head eating up swap space?

Path Finder

Our 6.6.2 search head (linux 2.6.32-573.18.1.el6.x86_64) is constantly low on free swap space,

I tried swapoff -a (clears swap), swapon -a

but after some time, splunk eats up almot 4 gigs of swap. Its also consistently maxing out CPU use

where would bea good place to troubleshoot this? Its not an indexer so not sure why such high CPU use here.

alt text

alt text

Tags (3)
0 Karma

Path Finder

checked any running jobs, have a few searches, but nothing major,

/opt/splunk/bin/splunk list jobs

Asynchronous jobs:
    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517928000_19793, ttl: 438

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517927700_19745, ttl: 139

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517927400_19669, ttl: 1042

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517926500_19506, ttl: 141

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517855400_10230, ttl: 13641

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517853600_9906, ttl: 11845

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517846400_8549, ttl: 4641

    Job id: 1517428177.10568377, ttl: 0
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!