Deployment Architecture

Suddenly, Splunk stops working

rubeniturrieta
Communicator

Hi all,

I have a Splunk server, that suddenly stops working, and i don't know why.
I've added the 'Splunk enable boot start' command, but anyways Splunk don't starts alone.

I'll be grateful for your help

Regards

Tags (3)
0 Karma
1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

There are a variety of reasons your Splunk could be crashing.

  1. Not enough memory (check /var/log/messages for OOM errors) - per @richgalloway
  2. SELINUX, (did you disable SELINUX? CentOS 7, by default, enforces it and running as root might be causing badness)
  3. ES could be causing significant load, but typically won't crash the system (unless it forces OOM killer at the OS level)

Did you figure out the issue?

View solution in original post

ephemeric
Contributor

Check your logs. Check your logs. Check your logs.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

There are a variety of reasons your Splunk could be crashing.

  1. Not enough memory (check /var/log/messages for OOM errors) - per @richgalloway
  2. SELINUX, (did you disable SELINUX? CentOS 7, by default, enforces it and running as root might be causing badness)
  3. ES could be causing significant load, but typically won't crash the system (unless it forces OOM killer at the OS level)

Did you figure out the issue?

rubeniturrieta
Communicator

Thanks you, alercercogitatus, i've reintalled ES, and i'm studying a week the Splunk behavior, it's working for now.
If Splunk stop working again, i'll see the other reasons that you suggested.

Thanks you, again

Regards

0 Karma

rubeniturrieta
Communicator

I've reinstalled ES, and Splunk didn't stopped working anymore.

Regards

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Perhaps the OS is shutting down Splunk because of a lack of resources. Check your OS logs.

---
If this reply helps you, Karma would be appreciated.

shandman
Path Finder

What OS are you using? Is the hardware stable? How does it shut down? Just shuts off? or Hangs?

0 Karma

rubeniturrieta
Communicator

I'm using Centos 7.1. The hardware is stable. Splunk just shuts off (for example, Splunk is today with no search, but tomorrow stops, and nobody did nothing)

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you looked at splunkd.log to see if there's anything that explains why Splunk won't start?

---
If this reply helps you, Karma would be appreciated.
0 Karma

koshyk
Super Champion

how did you start splunk? Please put the command here . Also which OS you using? Which User/group if Unix?

0 Karma

rubeniturrieta
Communicator

I start splunk with ./start splunk , in the splunk_home/bin folder. The OS is Centos 7.1, and root/root user/group. Splunk starts with no problem using the command, but suddenly stops.

0 Karma

koshyk
Super Champion

I think you are trying to put Splunk Enterprise Security or PCI app. They are heavy apps and is causing crashing.
Try removing whole of "Splunk ES or PCI app" and restart SPlunk

rubeniturrieta
Communicator

Yes, i have Splunk ES. I'll try it. Thanks you Koshyk

0 Karma

rubeniturrieta
Communicator

Hi richgalloway, thanks for your reply:

splunkd.log shows only warning, but nothing about why Splunk won't start.

06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list

0 Karma

rubeniturrieta
Communicator

Hi richgalloway, thanks for your reply,

splunkd.log only shows warnings, nothing about why Splunk won't start (06-24-2015 Splunk stops working)

06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/services_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/system_version_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/update_signature_reference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccounts_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.551 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/ids_category_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.554 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vuln_signature_reference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_cloud_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_email_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_web_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_dns_reply_codes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.858 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.858 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:55.814 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/DA-ESS-NetworkProtection/lookups/whois_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.006 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/listeningports_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/localprocesses_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_operations_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/services_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/system_version_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/update_signature_reference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.047 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccounts_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.305 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/ids_category_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.359 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vuln_signature_reference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_cloud_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_email_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_web_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_dns_reply_codes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:57.134 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:57.135 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.079 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/DA-ESS-NetworkProtection/lookups/whois_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/listeningports_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/localprocesses_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_operations_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/services_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/system_version_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/update_signature_reference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccounts_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.117 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/ids_category_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.117 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vuln_signature_reference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_cloud_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_email_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_web_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_dns_reply_codes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...