Monitoring Splunk

Why is the Splunk search head eating up swap space?

perfecto25
Path Finder

Our 6.6.2 search head (linux 2.6.32-573.18.1.el6.x86_64) is constantly low on free swap space,

I tried swapoff -a (clears swap), swapon -a

but after some time, splunk eats up almot 4 gigs of swap. Its also consistently maxing out CPU use

where would bea good place to troubleshoot this? Its not an indexer so not sure why such high CPU use here.

alt text

alt text

Tags (3)
0 Karma

perfecto25
Path Finder

checked any running jobs, have a few searches, but nothing major,

/opt/splunk/bin/splunk list jobs


Asynchronous jobs:
    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517928000_19793, ttl: 438

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517927700_19745, ttl: 139

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517927400_19669, ttl: 1042

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517926500_19506, ttl: 141

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517855400_10230, ttl: 13641

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517853600_9906, ttl: 11845

    Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517846400_8549, ttl: 4641

    Job id: 1517428177.10568377, ttl: 0
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!