If you are running Splunk with systemd and have ever upgraded your host memory, it is possible that Splunk is not using it! When you install Splunk and have completed your build process, if you are like me, one of the last things I do is: ./splunk enable boot-start -systemd-managed 1 -user splunk reboot the box and pat yourself on the back, maybe grab a beer. At some point later, you add more memory to the host (or in my case, resize the EC2 instance) EC2 is a special case which is what highlighted this for me - more on that later.. Assuming your host has swap memory, everything will work fine. However, what you may not notice is that Splunk is heavily using swap instead of real memory. If you disable swap swapoff -a you will very likely see Splunk dies rapidly with OOM. What happens is this: When you run splunk enable boot-start Splunk takes note of the current memory limit and writes this into the Splunkd.service unit file. Unless configured otherwise, default linux behaviour is to allow memory overcommit using cgroups, meaning that if Splunk wants more memory than it has allocated by cgroups, all sorts of magic happens which ultimately means that swap memory gets allocated to Splunk. If your swap is on fast IO disks, you may not notice straight away, however Splunk will never be able to use more 'real' memory than was present when you initially ran enable boot-start Check if you are affected: cat /etc/systemd/system/Splunkd.service Look for the line [Service] ... MemoryLimit= x Then check your system memory cat /proc/meminfo Look the value for MemTotal= y (kB) Note that the unit file is in bytes, and MemTotal is in kB, but check if these values match. If the MemoryLimit is below MemTotal then you very likely have been affected by this issue. To Resolve: 1 Stop Splunk 2 Disable boot-start ./splunk disable boot-start 3 Re-enable boot-start ./splunk enable boot-start -systemd-managed 1 -user splunk 4 Restart Splunk Now if you check the unit file, you should have the correct MemoryLimit applied and Splunk should be able to use all of it. Documentation: UPDATE: Following my submission, the documentation has been updated to reflect this. It now reads: The MemoryLimit value is set to the total system memory available in bytes when the service unit file is created. The MemoryLimit value will not update if the total available system memory changes. To update the MemoryLimit value in the unit file, you can manually edit the value or use the boot-start command to disable and re-enable systemd. http://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/RunSplunkassystemdservice The Splunk Docs team are still reviewing this, but this revised wording certainly makes it clearer. Thanks Edward K! EC2 corner case: Generally speaking "swap" on ec2 sucks! Most hosts running in AWS have storage volumes which have a limitation on IOPS. This can make it very apparent if your system has swap on an EBS and is using lots of IO because after a period of time, your host will have consumed all of the IOP credits, then IO grinds to a halt causing high iowait. (Tip, use htop and enable display of iowait - You will see high load averages but comparatively low CPU use unless iowait display is on) In my case this was exactly the problem I was experiencing. The obvious solution is more ram, less swap. So I upgraded the host, and disabled the swap. Instantly this caused OOM and so began much head scratching, digging and ultimately discovery of where this was occurring ... View more

I have an interesting problem which I can’t work out with the AWS-TA specifically for an S3 input. I am collecting CloudTrail logs from an S3 bucket (no SQS, because the existing environment was preconfigured) I am collecting the logs with a Generic S3 input, but I have limited the collection window to only events that have been written to S3 in the last week (ish). As an aside, the bucket is already lifecycle managed and only has 90 days of logs within it. I am running the TA on an AWS HF instance with the necessary IAM roles, and data is collected, however.. there is a significant delay between logs being written to S3, collected and indexed. After some investigation, I have discovered that the HF is chewing through its entire Swap disk, whilst the physical (well, virtual) memory is max ~4gb of total available. I have been debugging this issue on a variety of ec2 instance types (c5.4xl/2xl/xl) and the system memory/core count has no impact on the behaviour. The behaviour of this was such that the swap file located on the / volume would be heavily written to, this is in turn would eat the available IOPS on the volume, which when depleted cause high IOwait and eventually the system becomes unresponsive. To check if this was a problem with the process needing a high amount of virt. mem for the initial ‘read’ I have moved the instance to an m5d, which provides an ephemeral 140GB ssd, which I have used as a dedicated swap device as it is not IOP limited (except by the underlying hardware device). As predicted, this has solved the IOPS depleting and the iowait condition is prevented. The box has a steady load avg of ~3 (yes I know it’s only got 4 cores) but is otherwise quite happy. However, the S3 python process has consumed 124GB of Virtual Memory, of which 123GB is on Swap. I have never seen anything like this before with this TA across many deployments on AWS. The logs from the TA report nothing untoward in the relevant log file, and whilst CT logs are getting in, they are taking 2-4 hours to arrive, rather than 30mins as configured in the input. I have dumped the /swap partition with strings , and I can see that the swap file contains the data from the CT log files read from s3, so my present assumption is that the entire buckets log files are being read into swap (multiple times, as there is only 7-8GB of logs in the bucket), it seems that once swap is full, the oldest logs are evicted from swap, and the HF finally processes them, and sends them to the indexers. Side note – if I run the HF without swap, it gets oomkilled and restarts. No crashlog is generated. Even so, physical memory usage never peaks at more than 3GB. Does anyone have any ideas what could be up? ... View more