Splunk Enterprise

Cannot getting data into indexes

gjhaaland
Explorer

Hi,

For some reason we cannot receive data to _interal or other indexes(all of them). Old indexes are still available through database. It looks like a generic problem, not related to any specific index. All I can see is _audit.

Maybe it's ok to backup $SPLUNK_HOME/etc, and then reinstall splunk sw? or if possible restart some processes, or modify config file. input, output.conf

 

Rgds

Geir

Labels (1)
0 Karma

gjhaaland
Explorer

No problems with permissions, diskusage ++. I think it's a global problems. I know that for some days ago I tried to setup pkcs12 certificate (estreamer)  on splunk server.   But can't remember where I did these settings. 

Out form commands: 

$ source /home/splunk/bin/setSplunkEnv && df -H $SPLUNK_HOME $splunk_db
Tab-completion of "splunk <verb> <object>" is available.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-home 886G 587G 300G 67% /home
$


$sudo /home/splunk/bin/splunk btool indexes list volume |egrep '(\[|path)'
[volume:_splunk_summaries]
path = $SPLUNK_DB


$df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 11477568 40925632 22% /
devtmpfs 16312676 0 16312676 0% /dev
tmpfs 16329816 0 16329816 0% /dev/shm
tmpfs 16329816 10560 16319256 1% /run
tmpfs 16329816 0 16329816 0% /sys/fs/cgroup
/dev/sda3 1038336 173348 864988 17% /boot
/dev/mapper/centos-home 865131800 558906488 306225312 65% /home
tmpfs 3265964 12 3265952 1% /run/user/42
tmpfs 3265964 0 3265964 0% /run/user/1001
$

 

0 Karma

gjhaaland
Explorer

Thanks.  No problems with persmissions.  It could be something wrong with with some confiles.  But since the proplems  involves all indexfiles it must be something global settings, or some services/program not running. 

Do you thinks it's best to backup $SPLUNK/etc, run installation/upgrade and next copy etc files into new installation. 

Geir

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

If you cannot get any new data then mos obvious reason is that you have that disk space full. Second one is that for some reason your permissions / ownerships have changed on disk.

Please try "source /opt/splunk/bin/setSplunkEnv && df -H $SPLUNK_HOME $SPLUNK_DB" as a root on cmd line. Also check if you have volumes in use and check that disk space also.

To find volumes you should login as splunk user and then use

splunk btool indexes list volume|egrep '(\[|path)'

Which show those physical disk areas what those are using.

If there are enough space left then you should check ownership of those directories / files and change those if needed.

Did I understand right that you get some new data into _audit index, but not anywhere else?

r. Ismo 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What error messages do you see?  Are the indexes or the disk they're  on full?

Restarting or re-installing Splunk may help correct some causes of the problem, but not the most likely ones.

---
If this reply helps you, Karma would be appreciated.
0 Karma

gjhaaland
Explorer

Thanks for the answer.  Everyting seems to be ok. 

disk not full, licenses ok, rebooted several times, restarted splunk several times. But still we don't receive  data into indexes.  To save time, I wondered if it's possible to backup some files $SPLUNK_HOME/etc, and then reinstall splunk sw +  copy files into new installation. 

 

Do you think it will work?

Rgds

Geir 

0 Karma

SinghK
Builder

Did you try btool to check your configs, indexes.conf , inputs etc. may be there is a overlapping setting routing data somewhere else.

0 Karma

SinghK
Builder

or you are getting any permissions issue on splunk.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...