All Apps and Add-ons

Using Splunk on t2.micro Linux instance, why does the splunkd service need to be restarted to keep it running and how do I resolve this?

prasasthi001
New Member

Hi,

I have a t2.micro Linux instance running as a Splunk node. The Splunk instance sometimes doesn't pass status checks on AWS. When I stop and restart the instance again, it works. I SSH into the instance and check the status every time I cannot access the home page. It shows that the splunkd is not running. I restart the process and then I can access the Splunk page on port 8000 again. Please help me resolve this issue.

Thank you.
Sai

0 Karma

Jeremiah
Motivator

The t2.micro instance has 1 (burstable) cpu and 1 GB of memory, which barely meet the Splunk minimum hw requirements. How much data are you pushing onto this system? How many users are accessing the UI? It's likely the process is crashing due to resource constraints. There are a couple of ways you can check this.

First, look at /opt/splunk/var/log/splunk and check for crash files. These files indicate the process crashed unexpectedly. If you have a support contract Splunk can use these files to help determine the cause of the crash.

Look at the sourcetype=splunkd log files from your instance at the time of the crash. Are there any errors or warnings that might indicate a problem?

Check the cloudwatch metrics for this instance. How is the CPU utilization? Disk and network IO? If you have the CW agent enabled, check memory utilization. You can also look at detailed host metrics collected by Splunk in the _introspection index. Check the DMC for any indications of resource constraints, especially memory.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...