topic Re: Logs are not getting forwarded into splunk instance via splunk forwarder in Getting Data In

Why are the logs not getting forwarded into the splunk instance via splunk forwarder?

rahul2gupta — Mon, 14 Feb 2022 15:41:34 GMT

We have installed Splunk universal forwarder on a remote server but logs are not getting forwarded to Indexer.

I have tried to troubleshoot this issue but could not do so. Can you please help me to get rid of this issue.

Below are the steps I have tried so far.

Remote server is communicating with Indexer

root@host1:/opt/splunkforwarder/etc/system/local# telnet host2 9997
Trying 10.20.30.40...
Connected to host2
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Below is the content of outputs.conf

root@host1:/opt/splunkforwarder/etc/system/local# cat outputs.conf
[tcpout]
defaultGroup = splunk

[tcpout:splunk]
server = host2.ce.corp:9997

Below is the content of inputs.conf

root@host1:/opt/splunkforwarder/etc/system/local# cat inputs.conf
[default]
host = host1

[monitor:///var/log/messages]
disabled = false
sourcetype = web_haprx
index = webmethods_haprx

Ran ./splunk list forward-server

root@host1:/opt/splunkforwarder/bin# ./splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Password:
Active forwards:
host2:9997
Configured but inactive forwards:
None

port 9997 is enabled on receiver
Also I did check splunk.log to see any error but no luck.

Can you please help me to fix this issue?

Regards,

Rahul Gupta

Re: Logs are not getting forwarded into splunk instance via splunk forwarder

gcusello — Fri, 11 Feb 2022 07:29:20 GMT

Hi @rahul2gupta,

somethimes tyhere isn't a correct resolution of the hostname, so, please, try using IP address and than add a row to your outputs.conf:

[tcpout] defaultGroup = splunk [tcpout-server://ip_address_host2:9997] [tcpout:splunk] server = ip_address_host2:9997

Ciao.

Giuseppe

Re: Logs are not getting forwarded into splunk instance via splunk forwarder

PickleRick — Fri, 11 Feb 2022 07:32:42 GMT

What do you mean by "logs are not getting forwarded"? How do you know that?

Do you have any errors in your /opt/splunkforwarder/var/log/splunk/splunkd.log on your forwarder?

You can also check your _internal index for any logs from your forwarder host. If you have any logs from the forwarder, the forwarding as such is working properly so if you're not getting your events there's a problem in other part of your config.

Do a

| tstats count where index=_internal by host

for the last day or so and see whether you're getting data from that forwarder at all.

Re: Logs are not getting forwarded into splunk instance via splunk forwarder

rahul2gupta — Mon, 14 Feb 2022 11:39:08 GMT

Hi @PickleRick ,

Q:-What do you mean by "logs are not getting forwarded"? How do you know that?

It is because when am using network port UDP:5514, I can see logs into Splunk but when am trying to forward logs into Splunk. We are unable to do so. we are trying to send /var/log/messages

Q:-Do you have any errors in your /opt/splunkforwarder/var/log/splunk/splunkd.log on your forwarder?

No, we could not see any errors. It was there earlier but we fixed.

02-08-2022 15:39:15.907 +1100 ERROR TailingProcessor - Input stanza path, 'var/log/messages' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.

Q:- whether you're getting data from that forwarder at all?

Yes, we are getting data. Below is the sample.

Feb 14 22:35:27 host1 Container_ImageInventory[2911256]: Container image name () is improperly formed and could not be parsed in SetRepositoryImageTag

Regards,

Rahul Gupta

Re: Logs are not getting forwarded into splunk instance via splunk forwarder

PickleRick — Mon, 14 Feb 2022 12:58:27 GMT

OK, if you're sending data straight to udp input on your indexer it has nothing to do with the forwarder so it has no diagnostic value here.

About the log you showed - well, that's kinda interesting. If you only have an input defined for /var/log/messages - how are you getting the log about that Container_ImageInventory?

By default after installation and definition of output, the UF should only forward its own internal logs to _internal index.

Do a "splunk list monitor" on your forwarder. And "splunk btool inputs list --debug".

And see what inputs you have defined and running.