Looking at your free -m output, we can see that very little physical memory remains free on your server. Oddly enough, most of it (Almost 13GB out of your total of 16) appears to be held by the operating system for the purpose of caching.
Although I would expect things to be resolved organically given that you have enough swap, it's possible that the kernel scheduler is having trouble deciding who to push into swap to execute new processes that may require more than the remaining 1GB of free physical memory, which may result in the memory allocation error you report.
For sure, 13GB of kernel cache appears excessive. Rebooting the box to reset that and attempting splunk fsck again seem like the reasonable way to go.
... View more