After changing Linux host name, Why do the logs not show up in new search engine?

[knieman9]

Hey there!

I used vmware to clone a host.

i tried changing server.conf and inputs.conf seven ways from Sunday. The process starts up without problems, but when i go to our local search engine: rjbandwpoc2 source="/var/log/secure".

nothing shows up.

thanks for any pointers.

[knieman9]

Thanks for the suggestions, but still no dice. I changed the inputs.conf file to use the new host name and restarted. i used splunk btool inputs list --debug on the original host and the one i cloned to. the only diff was the host name which is good.

< /opt/splunkforwarder/etc/system/local/inputs.conf host = rjbandwpoc2
---
> /opt/splunkforwarder/etc/system/local/inputs.conf host = rjbandwdev01

I am getting double-crossed somewhere else. the floor is official open for suggestions..

[PickleRick]

OK . The question is whether the logs are getting sent from this forwarder to the indexer at all.

Check /opt/splunkforwarder/var/log/splunk/splunkd.log for errors regarding upstream connections (or confirmation of connection).

Did you check with netstat or ss that there are connections established?

[knieman9]

interesting. the logs from the original host has a connection and some errors, the same messages in the hosts i cant find logs in my search engine. thanks for any pointers.

log snippet of the guy working:

03-24-2022 10:46:00.977 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:46:04.626 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh: Permission denied
03-24-2022 10:46:09.488 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::linux_secure. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:46:14.012 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:46:24.910 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh: Permission denied
03-24-2022 10:46:30.907 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.

log snippet of guy not working:

03-24-2022 10:44:53.096 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:44:57.012 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:44:57.014 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:07.035 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:07.038 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:22.933 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:45:25.054 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:25.059 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:33.523 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh: Per
mission denied
03-24-2022 10:45:35.088 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
[root@rjbandwpoc2 splunk]#

[PickleRick]

Well, there is generally quite a lot going on in your forwarders and not all of it is good. But it seems that both of those forwarders connect to the indexer.

03-24-2022 10:46:30.907 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.

You can verify from which hosts the events in _internal index come from.

| tstats count where index=_internal by host

You should get values for both of your forwarders (as well as another parts of your splunk infrastructure).

It's hard however to tell what else is going on in your setup since it clearly has some issues.

[gcusello]

Hi @knieman9,

if you clone a complete installation, in addition to the system hostname, you have also to change the hostname in :

• $SPLUNK_HOME/etc/system/local/server.conf
• $SPLUNK_HOME/etc/system/local/inputs.conf

and restart Splunk at the end.

Ciao.

Giuseppe

[PickleRick]

You might also have the hostname set in some other place. You probably don't if it's a fairly typical installation but in general - there is a possibility.

You can check it by calling your forwarder with

splunk btool inputs list --debug

This way you'll see if your hostname is overwritten somewhere and if so - in which file.