topic Re: Cannot getting data into indexes in Splunk Enterprise

Cannot getting data into indexes

gjhaaland — Thu, 12 Oct 2023 15:27:07 GMT

Hi,

For some reason we cannot receive data to _interal or other indexes(all of them). Old indexes are still available through database. It looks like a generic problem, not related to any specific index. All I can see is _audit.

Maybe it's ok to backup $SPLUNK_HOME/etc, and then reinstall splunk sw? or if possible restart some processes, or modify config file. input, output.conf

Rgds

Geir

Re: Cannot getting data into indexes

richgalloway — Thu, 12 Oct 2023 15:52:47 GMT

What error messages do you see? Are the indexes or the disk they're on full?

Restarting or re-installing Splunk may help correct some causes of the problem, but not the most likely ones.

Re: Cannot getting data into indexes

gjhaaland — Fri, 13 Oct 2023 07:16:30 GMT

Thanks for the answer. Everyting seems to be ok.

disk not full, licenses ok, rebooted several times, restarted splunk several times. But still we don't receive data into indexes. To save time, I wondered if it's possible to backup some files $SPLUNK_HOME/etc, and then reinstall splunk sw + copy files into new installation.

Do you think it will work?

Rgds

Geir

Re: Cannot getting data into indexes

SinghK — Fri, 13 Oct 2023 10:06:58 GMT

Did you try btool to check your configs, indexes.conf , inputs etc. may be there is a overlapping setting routing data somewhere else.

Re: Cannot getting data into indexes

SinghK — Fri, 13 Oct 2023 10:07:37 GMT

or you are getting any permissions issue on splunk.

Re: Cannot getting data into indexes

gjhaaland — Fri, 13 Oct 2023 10:28:03 GMT

Thanks. No problems with persmissions. It could be something wrong with with some confiles. But since the proplems involves all indexfiles it must be something global settings, or some services/program not running.

Do you thinks it's best to backup $SPLUNK/etc, run installation/upgrade and next copy etc files into new installation.

Geir

Re: Cannot getting data into indexes

isoutamo — Fri, 13 Oct 2023 10:40:16 GMT

If you cannot get any new data then mos obvious reason is that you have that disk space full. Second one is that for some reason your permissions / ownerships have changed on disk.

Please try "source /opt/splunk/bin/setSplunkEnv && df -H $SPLUNK_HOME $SPLUNK_DB" as a root on cmd line. Also check if you have volumes in use and check that disk space also.

To find volumes you should login as splunk user and then use

splunk btool indexes list volume|egrep '(\[|path)'

Which show those physical disk areas what those are using.

If there are enough space left then you should check ownership of those directories / files and change those if needed.

Did I understand right that you get some new data into _audit index, but not anywhere else?

r. Ismo

Re: Cannot getting data into indexes

gjhaaland — Fri, 13 Oct 2023 11:01:05 GMT

No problems with permissions, diskusage ++. I think it's a global problems. I know that for some days ago I tried to setup pkcs12 certificate (estreamer) on splunk server. But can't remember where I did these settings.

Out form commands:

$ source /home/splunk/bin/setSplunkEnv && df -H $SPLUNK_HOME $splunk_db
Tab-completion of "splunk <verb> <object>" is available.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-home 886G 587G 300G 67% /home
$

$sudo /home/splunk/bin/splunk btool indexes list volume |egrep '(\[|path)'
[volume:_splunk_summaries]
path = $SPLUNK_DB

$df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 11477568 40925632 22% /
devtmpfs 16312676 0 16312676 0% /dev
tmpfs 16329816 0 16329816 0% /dev/shm
tmpfs 16329816 10560 16319256 1% /run
tmpfs 16329816 0 16329816 0% /sys/fs/cgroup
/dev/sda3 1038336 173348 864988 17% /boot
/dev/mapper/centos-home 865131800 558906488 306225312 65% /home
tmpfs 3265964 12 3265952 1% /run/user/42
tmpfs 3265964 0 3265964 0% /run/user/1001
$