Installation

After changing Linux host name, Why do the logs not show up in new search engine?

knieman9
Loves-to-Learn Lots

Hey there!

I used vmware to clone a host.

i tried changing server.conf and inputs.conf seven ways from Sunday. The process starts up without problems, but when i go to our local search engine: rjbandwpoc2 source="/var/log/secure".

nothing shows up.

thanks for any pointers.

Labels (1)
0 Karma

knieman9
Loves-to-Learn Lots

Thanks for the suggestions, but still no dice. I changed the inputs.conf file to use the new host name and restarted. i used splunk btool inputs list --debug on the original host and the one i cloned to. the only diff was the host name which is good.

< /opt/splunkforwarder/etc/system/local/inputs.conf host = rjbandwpoc2
---
> /opt/splunkforwarder/etc/system/local/inputs.conf host = rjbandwdev01

I am getting double-crossed somewhere else. the floor is official open for suggestions..

0 Karma

PickleRick
Ultra Champion

OK . The question is whether the logs are getting sent from this forwarder to the indexer at all.

Check /opt/splunkforwarder/var/log/splunk/splunkd.log for errors regarding upstream connections (or confirmation of connection).

Did you check with netstat or ss that there are connections established?

0 Karma

knieman9
Loves-to-Learn Lots

interesting. the logs from the original host has a connection and some errors, the same messages in the hosts i cant find logs in my search engine. thanks for any pointers.

log snippet of the guy working:

03-24-2022 10:46:00.977 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:46:04.626 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh: Permission denied
03-24-2022 10:46:09.488 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::linux_secure. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:46:14.012 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:46:24.910 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh: Permission denied
03-24-2022 10:46:30.907 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.

log snippet of guy not working:

03-24-2022 10:44:53.096 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:44:57.012 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:44:57.014 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:07.035 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:07.038 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:22.933 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.
03-24-2022 10:45:25.054 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:25.059 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
03-24-2022 10:45:33.523 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh" /bin/sh: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh: Per
mission denied
03-24-2022 10:45:35.088 -0500 INFO ChunkedLBProcessor - Failed to find EVENT_BREAKER regex in props.conf for sourcetype::syslog. Reverting to the default EVENT_BREAKER regex for now.
[root@rjbandwpoc2 splunk]#

0 Karma

PickleRick
Ultra Champion

Well, there is generally quite a lot going on in your forwarders and not all of it is good. But it seems that both of those forwarders connect to the indexer.

03-24-2022 10:46:30.907 -0500 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.21.20.29:9997, reuse=1.

You can verify from which hosts the events in _internal index come from.

| tstats count where index=_internal by host

You should get values for both of your forwarders (as well as another parts of your splunk infrastructure).

It's hard however to tell what else is going on in your setup since it clearly has some issues.

0 Karma

gcusello
Esteemed Legend

Hi @knieman9,

if you clone a complete installation, in addition to the system hostname, you have also to change the hostname in :

  • $SPLUNK_HOME/etc/system/local/server.conf
  • $SPLUNK_HOME/etc/system/local/inputs.conf

and restart Splunk at the end.

Ciao.

Giuseppe

PickleRick
Ultra Champion

You might also have the hostname set in some other place. You probably don't if it's a fairly typical installation but in general - there is a possibility.

You can check it by calling your forwarder with

splunk btool inputs list --debug

This way you'll see if your hostname is overwritten somewhere and if so - in which file.

0 Karma
Get Updates on the Splunk Community!

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...