Installation

After changing Linux host name, Why do the logs not show up in new search engine?

knieman9
Loves-to-Learn Lots

Hey there!

I used vmware to clone a host.

i tried changing server.conf and inputs.conf seven ways from Sunday. The process starts up without problems, but when i go to our local search engine: rjbandwpoc2 source="/var/log/secure".

nothing shows up.

thanks for any pointers.

Labels (1)
0 Karma

knieman9
Loves-to-Learn Lots

Thanks for the suggestions, but still no dice. I changed the inputs.conf file to use the new host name and restarted. i used splunk btool inputs list --debug on the original host and the one i cloned to. the only diff was the host name which is good.

< /opt/splunkforwarder/etc/system/local/inputs.conf host = rjbandwpoc2
---
> /opt/splunkforwarder/etc/system/local/inputs.conf host = rjbandwdev01

I am getting double-crossed somewhere else. the floor is official open for suggestions..

0 Karma

PickleRick
Ultra Champion

OK . The question is whether the logs are getting sent from this forwarder to the indexer at all.

Check /opt/splunkforwarder/var/log/splunk/splunkd.log for errors regarding upstream connections (or confirmation of connection).

Did you check with netstat or ss that there are connections established?

0 Karma

knieman9
Loves-to-Learn Lots

interesting. the logs from the original host has a connection and some errors, the same messages in the hosts i cant find logs in my search engine. thanks for any pointers.

log snippet of the guy working:

03-24-2022 10:46:00.977 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:46:04.626 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh: Permission denied
03-24-2022 10:46:09.488 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::linux_secure. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:46:14.012 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:46:24.910 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh: Permission denied
03-24-2022 10:46:30.907 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.

log snippet of guy not working:

03-24-2022 10:44:53.096 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:44:57.012 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:44:57.014 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:07.035 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:07.038 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:22.933 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:45:25.054 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:25.059 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:33.523 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh: Per
mission denied
03-24-2022 10:45:35.088 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
[root@rjbandwpoc2 splunk]#

0 Karma

PickleRick
Ultra Champion

Well, there is generally quite a lot going on in your forwarders and not all of it is good. But it seems that both of those forwarders connect to the indexer.

03-24-2022 10:46:30.907 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.

You can verify from which hosts the events in _internal index come from.

| tstats count where index=_internal by host

You should get values for both of your forwarders (as well as another parts of your splunk infrastructure).

It's hard however to tell what else is going on in your setup since it clearly has some issues.

0 Karma

gcusello
Legend

Hi @knieman9,

if you clone a complete installation, in addition to the system hostname, you have also to change the hostname in :

  • $SPLUNK_HOME/etc/system/local/server.conf
  • $SPLUNK_HOME/etc/system/local/inputs.conf

and restart Splunk at the end.

Ciao.

Giuseppe

PickleRick
Ultra Champion

You might also have the hostname set in some other place. You probably don't if it's a fairly typical installation but in general - there is a possibility.

You can check it by calling your forwarder with

splunk btool inputs list --debug

This way you'll see if your hostname is overwritten somewhere and if so - in which file.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...