topic Re: Suddenly, Splunk stops working in Deployment Architecture

Suddenly, Splunk stops working

rubeniturrieta — Wed, 01 Jul 2015 13:34:41 GMT

Hi all,

I have a Splunk server, that suddenly stops working, and i don't know why.
I've added the 'Splunk enable boot start' command, but anyways Splunk don't starts alone.

I'll be grateful for your help

Regards

Re: Suddenly, Splunk stops working

richgalloway — Wed, 01 Jul 2015 13:37:02 GMT

Have you looked at splunkd.log to see if there's anything that explains why Splunk won't start?

Re: Suddenly, Splunk stops working

rubeniturrieta — Mon, 28 Sep 2020 20:26:18 GMT

Hi richgalloway, thanks for your reply,

splunkd.log only shows warnings, nothing about why Splunk won't start (06-24-2015 Splunk stops working)

06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/services_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/system_version_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/update_signature_reference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccounts_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.551 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/ids_category_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.554 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vuln_signature_reference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_cloud_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_email_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_web_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_dns_reply_codes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.858 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.858 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:55.814 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/DA-ESS-NetworkProtection/lookups/whois_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.006 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/listeningports_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/localprocesses_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_operations_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/services_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/system_version_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/update_signature_reference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.047 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccounts_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.305 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/ids_category_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.359 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vuln_signature_reference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_cloud_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_email_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_web_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_dns_reply_codes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:57.134 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:57.135 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.079 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/DA-ESS-NetworkProtection/lookups/whois_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/listeningports_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/localprocesses_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_operations_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malware_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/services_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/system_version_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/update_signature_reference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccounts_tracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.117 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/ids_category_tracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.117 -0300 WARN SearchResults - /home/nuevo_splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vuln_signature_reference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_cloud_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_email_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_corporate_web_domains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_dns_reply_codes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list

Re: Suddenly, Splunk stops working

rubeniturrieta — Mon, 28 Sep 2020 20:26:23 GMT

Hi richgalloway, thanks for your reply:

splunkd.log shows only warning, but nothing about why Splunk won't start.

06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim_email_protocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid_lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list

Re: Suddenly, Splunk stops working

koshyk — Wed, 01 Jul 2015 14:07:54 GMT

how did you start splunk? Please put the command here . Also which OS you using? Which User/group if Unix?

Re: Suddenly, Splunk stops working

rubeniturrieta — Wed, 01 Jul 2015 14:11:44 GMT

I start splunk with ./start splunk , in the splunk_home/bin folder. The OS is Centos 7.1, and root/root user/group. Splunk starts with no problem using the command, but suddenly stops.

Re: Suddenly, Splunk stops working

shandman — Wed, 01 Jul 2015 14:15:17 GMT

What OS are you using? Is the hardware stable? How does it shut down? Just shuts off? or Hangs?

Re: Suddenly, Splunk stops working

rubeniturrieta — Wed, 01 Jul 2015 14:20:37 GMT

I'm using Centos 7.1. The hardware is stable. Splunk just shuts off (for example, Splunk is today with no search, but tomorrow stops, and nobody did nothing)

Re: Suddenly, Splunk stops working

koshyk — Wed, 01 Jul 2015 14:38:54 GMT

I think you are trying to put Splunk Enterprise Security or PCI app. They are heavy apps and is causing crashing.
Try removing whole of "Splunk ES or PCI app" and restart SPlunk

Re: Suddenly, Splunk stops working

richgalloway — Wed, 01 Jul 2015 14:44:42 GMT

Perhaps the OS is shutting down Splunk because of a lack of resources. Check your OS logs.

Re: Suddenly, Splunk stops working

rubeniturrieta — Wed, 01 Jul 2015 14:46:26 GMT

Yes, i have Splunk ES. I'll try it. Thanks you Koshyk

Re: Suddenly, Splunk stops working

alacercogitatus — Mon, 06 Jul 2015 12:02:45 GMT

There are a variety of reasons your Splunk could be crashing.

Not enough memory (check /var/log/messages for OOM errors) - per @richgalloway
SELINUX, (did you disable SELINUX? CentOS 7, by default, enforces it and running as root might be causing badness)
ES could be causing significant load, but typically won't crash the system (unless it forces OOM killer at the OS level)

Did you figure out the issue?

Re: Suddenly, Splunk stops working

rubeniturrieta — Mon, 06 Jul 2015 21:17:22 GMT

Thanks you, alercercogitatus, i've reintalled ES, and i'm studying a week the Splunk behavior, it's working for now.
If Splunk stop working again, i'll see the other reasons that you suggested.

Thanks you, again

Regards

Re: Suddenly, Splunk stops working

rubeniturrieta — Wed, 15 Jul 2015 12:48:19 GMT

I've reinstalled ES, and Splunk didn't stopped working anymore.

Regards

Re: Suddenly, Splunk stops working

ephemeric — Mon, 23 Aug 2021 19:04:50 GMT

Check your logs. Check your logs. Check your logs.

Re: Suddenly, Splunk stops working

lespinosas — Fri, 16 May 2025 17:47:30 GMT

Wow @ephemeric thanks for the help!! -.-