Using Splunk on t2.micro Linux instance, why does the splunkd service need to be restarted to keep it running and how do I resolve this?

[prasasthi001]

Hi,

I have a t2.micro Linux instance running as a Splunk node. The Splunk instance sometimes doesn't pass status checks on AWS. When I stop and restart the instance again, it works. I SSH into the instance and check the status every time I cannot access the home page. It shows that the splunkd is not running. I restart the process and then I can access the Splunk page on port 8000 again. Please help me resolve this issue.

Thank you.
Sai

[Jeremiah]

The t2.micro instance has 1 (burstable) cpu and 1 GB of memory, which barely meet the Splunk minimum hw requirements. How much data are you pushing onto this system? How many users are accessing the UI? It's likely the process is crashing due to resource constraints. There are a couple of ways you can check this.

First, look at /opt/splunk/var/log/splunk and check for crash files. These files indicate the process crashed unexpectedly. If you have a support contract Splunk can use these files to help determine the cause of the crash.

Look at the sourcetype=splunkd log files from your instance at the time of the crash. Are there any errors or warnings that might indicate a problem?

Check the cloudwatch metrics for this instance. How is the CPU utilization? Disk and network IO? If you have the CW agent enabled, check memory utilization. You can also look at detailed host metrics collected by Splunk in the _introspection index. Check the DMC for any indications of resource constraints, especially memory.