Archive
Highlighted

Suddenly, Splunk stops working

Communicator

Hi all,

I have a Splunk server, that suddenly stops working, and i don't know why.
I've added the 'Splunk enable boot start' command, but anyways Splunk don't starts alone.

I'll be grateful for your help

Regards

Tags (3)
0 Karma
Highlighted

Re: Suddenly, Splunk stops working

SplunkTrust
SplunkTrust

Have you looked at splunkd.log to see if there's anything that explains why Splunk won't start?

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Communicator

Hi richgalloway, thanks for your reply,

splunkd.log only shows warnings, nothing about why Splunk won't start (06-24-2015 Splunk stops working)

06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/servicestracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/systemversiontracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/updatesignaturereference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.515 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccountstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.551 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-NetworkProtection/lookups/idscategorytracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.554 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vulnsignaturereference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimclouddomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimcorporateemaildomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim
corporatewebdomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimdnsreplycodes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.584 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim
emailprotocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.858 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid
lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:02:53.858 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sidlookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:55.814 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/DA-ESS-NetworkProtection/lookups/whoistracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.006 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/listeningportstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/localprocessestracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malwareoperationstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.007 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/malwaretracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/servicestracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/systemversiontracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.046 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/updatesignaturereference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.047 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccountstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.305 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-NetworkProtection/lookups/idscategorytracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.359 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vulnsignaturereference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimclouddomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimcorporateemaildomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim
corporatewebdomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimdnsreplycodes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:56.618 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim
emailprotocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:57.134 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid
lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:07:57.135 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sidlookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.079 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/DA-ESS-NetworkProtection/lookups/whoistracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/listeningportstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/localprocessestracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/malwareoperationstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.103 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/malwaretracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/servicestracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/systemversiontracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-EndpointProtection/lookups/updatesignaturereference.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.104 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-EndpointProtection/lookups/useraccountstracker.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.117 -0300 WARN SearchResults - /home/nuevosplunk/splunk/etc/apps/SA-NetworkProtection/lookups/idscategorytracker2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.117 -0300 WARN SearchResults - /home/nuevo
splunk/splunk/etc/apps/SA-NetworkProtection/lookups/vulnsignaturereference2.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimclouddomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimcorporateemaildomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.136 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim
corporatewebdomains.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimdnsreplycodes.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cim
emailprotocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guid
lookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid_lookup.csv in lookups.conf, cannot enumerate fields list

0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Communicator

Hi richgalloway, thanks for your reply:

splunkd.log shows only warning, but nothing about why Splunk won't start.

06-24-2015 22:08:00.137 -0300 WARN TransformsExtractionHandler - Unable to find stanza=cimemailprotocols.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=guidlookup.csv in lookups.conf, cannot enumerate fields list
06-24-2015 22:08:00.514 -0300 WARN TransformsExtractionHandler - Unable to find stanza=sid
lookup.csv in lookups.conf, cannot enumerate fields list

0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Super Champion

how did you start splunk? Please put the command here . Also which OS you using? Which User/group if Unix?

0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Communicator

I start splunk with ./start splunk , in the splunk_home/bin folder. The OS is Centos 7.1, and root/root user/group. Splunk starts with no problem using the command, but suddenly stops.

0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Super Champion

I think you are trying to put Splunk Enterprise Security or PCI app. They are heavy apps and is causing crashing.
Try removing whole of "Splunk ES or PCI app" and restart SPlunk

Highlighted

Re: Suddenly, Splunk stops working

Communicator

Yes, i have Splunk ES. I'll try it. Thanks you Koshyk

0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Path Finder

What OS are you using? Is the hardware stable? How does it shut down? Just shuts off? or Hangs?

0 Karma
Highlighted

Re: Suddenly, Splunk stops working

Communicator

I'm using Centos 7.1. The hardware is stable. Splunk just shuts off (for example, Splunk is today with no search, but tomorrow stops, and nobody did nothing)

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.