Deployment Architecture

How to get an answer from search command in Splunk Enterprise?

gjhaaland
Explorer

Hi,

Splunk has been working for a long period without any trouble. When I changed settings yesterday (can't remember what I did) the search command dos not work as before (no answer). 

If I go to settings - indexing   _audit, _internal , _introspection,  _telemtry, _history + main area all of them is disabled.

I also google, and it says that it perhaps has something to do identical id under db directory. We have same id on some files with .sentinel  

example:

db_123_345_12

db_123_345_12.rbsentinel 

 

If I run following command:

run netsat -an | grep 9997 we have many tcp session establised . 

 

Have of course rebooted, restarted splunk server several times.  It does not help much. 

Thanks in advance. Hope someone can give me a hint. 

 

Rgds

Geir

 

 

Labels (1)
Tags (1)
0 Karma

gjhaaland
Explorer

Forgot to mention

When I open Data Summary it says "Waiting for results" but it never get/receive any data. Only Waiting for Results without ending.

Rgds

Geir

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @gjhaaland ,

open a case To Splunk Support, it's the only way to have a quick answer.

ciao.

Giuseppe

0 Karma

gjhaaland
Explorer

Giuseppe,

Thanks again, 

Yes, If I run search command and/or old reports we get  no answer at all. The splunk gui is running, but we don't get any answer if we run search - index=*. Normally we will see a long listing with output. 

I have not deleted any files.  All I have done is  some settings regarding field extraction. After a while I discovered that we did not receive any data at all.  So I must be some connection between fields (enable/disable) and fields extraction.

Rgds

Geir 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @gjhaaland,

if you run a search on _internal, did you have results?

have you any messages from Splunk?

Ciao.

Giuseppe

0 Karma

gjhaaland
Explorer

Hi gcusello

 

Thanks for the answer.  No answer at all, even if I run “Usage Reporting Dashboard” the answer is empty. Since it work perfect yesterday I thinks/assume that some files are blocking stopping normal behavior .  

 

If I restart splunkd  I got following messages

 

1: Invalid key in stanza  [admin_external:configure]in /home/splunk/etc/apps/TA-eStreamer/default/restmap.conf, line 7: python.version

2: your indexes and inputs configurations are not internally consistent. For more info run splunk btool -check –debug

3: Validating installed files against hashes from /home/splunk/splunk/7.1……..-x86_64manifest’

Problems were found, please review your files and more customization to local

 

Starting splunk aerver deamon (splunkd)

Done

[OK}

 

Rgds

Geir

 

If I run splunk btool -check –debug

 

I got following error (cut/paste errors)

 

No spec file for: /home/splunk/etc/apps/Splunk_CiscoSecuritySuite/local/css_views.co

No spec file for: /home/splunk/etc/apps/TA-eStreamer/local/encore.conf

No spec file for: /home/splunk/etc/apps/eStreamer/local/estreamer.conf

No spec file for: /home/splunk/etc/apps/Splunk_CiscoSecuritySuite/default/css_views.conf

No spec file for: /home/splunk/etc/apps/Splunk_CiscoSecuritySuite/default/eventgen.conf

No spec file for: /home/splunk/etc/apps/TA-eStreamer/default/encore.conf

Invalid key in stanza [admin_external:configure] in /home/splunk/etc/apps/TA-eStreamer/default/restmap.conf, line 7: python.version  (value:  python3).

No spec file for: /home/splunk/etc/apps/eStreamer/default/estreamer.conf

No spec file for: /home/splunk/etc/apps/firepower_dashboard/default/appsetup.conf

No spec file for: /home/splunk/etc/apps/firepower_dashboard/default/umbrella.conf

No spec file for: /home/splunk/etc/system/default/conf.conf

No spec file for: /home/splunk/etc/system/local/migration.conf

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @gjhaaland,

the error messages aren't relevant.

Let me better understan: the search doesn't run or you have always no results?

When you say that yesterday worked perfectly, are you meaning: that yesterday the searches  run or that running  today a search on yesterday data the are ok?

Probably the only solution is to opena a case to Splunk Support that can access your system (with you) and debug the situation.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...