Why are the logs not getting forwarded into the splunk instance via splunk forwarder?

[rahul2gupta]

Hi  

We have installed Splunk universal forwarder on a remote server but logs are not getting forwarded to Indexer.

I have tried to troubleshoot this issue but could not do so. Can you please help me to get rid of this issue.

Below are the steps I have tried so far.

• Remote server is communicating with Indexer

root@host1:/opt/splunkforwarder/etc/system/local# telnet host2 9997
Trying 10.20.30.40...
Connected to host2
Escape character is '^]'.
^]
telnet> quit
Connection closed.

• Below is the content of outputs.conf

          root@host1:/opt/splunkforwarder/etc/system/local# cat outputs.conf
           [tcpout]
           defaultGroup = splunk

           [tcpout:splunk]
            server = host2.ce.corp:9997

• Below is the content of inputs.conf

         root@host1:/opt/splunkforwarder/etc/system/local# cat inputs.conf
         [default]
         host = host1

         [monitor:///var/log/messages]
         disabled = false
         sourcetype = web_haprx
         index = webmethods_haprx

• Ran ./splunk list forward-server

           root@host1:/opt/splunkforwarder/bin# ./splunk list forward-server
           Your session is invalid. Please login.
           Splunk username: admin
           Password:
           Active forwards:
           host2:9997
           Configured but inactive forwards:
            None

• port 9997 is enabled on receiver 
• Also I did check splunk.log to see any error but no luck.

Can you please help me to fix this issue?

Regards,

Rahul Gupta

[PickleRick]

What do you mean by "logs are not getting forwarded"? How do you know that?

Do you have any errors in your /opt/splunkforwarder/var/log/splunk/splunkd.log on your forwarder?

You can also check your _internal index for any logs from your forwarder host. If you have any logs from the forwarder, the forwarding as such is working properly so if you're not getting your events there's a problem in other part of your config.

Do a

| tstats count where index=_internal by host

 for the last day or so and see whether you're getting data from that forwarder at all.

[rahul2gupta]

Hi @PickleRick ,

Q:-What do you mean by "logs are not getting forwarded"? How do you know that?

It is because when am using network port UDP:5514, I can see logs into Splunk but when am trying to forward logs into Splunk. We are unable to do so. we are  trying to send /var/log/messages 

Q:-Do you have any errors in your /opt/splunkforwarder/var/log/splunk/splunkd.log on your forwarder?

No, we could not see any errors.  It was there earlier but we fixed.

02-08-2022 15:39:15.907 +1100 ERROR TailingProcessor - Input stanza path, 'var/log/messages' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.

Q:-  whether you're getting data from that forwarder at all?

Yes, we are getting data. Below is the sample.

Feb 14 22:35:27 host1 Container_ImageInventory[2911256]: Container image name () is improperly formed and could not be parsed in SetRepositoryImageTag

Regards,

Rahul Gupta

[PickleRick]

OK, if you're sending data straight to udp input on your indexer it has nothing to do with the forwarder so it has no diagnostic value here.

About the log you showed - well, that's kinda interesting. If you only have an input defined for /var/log/messages - how are you getting the log about that Container_ImageInventory?

By default after installation and definition of output, the UF should only forward its own internal logs to _internal index.

Do a "splunk list monitor" on your forwarder. And "splunk btool inputs list --debug".

And see what inputs you have defined and running.

[gcusello]

Hi @rahul2gupta,

somethimes tyhere isn't a correct resolution of the hostname, so, please, try using IP address and than add a row to your outputs.conf:

[tcpout]
defaultGroup = splunk

[tcpout-server://ip_address_host2:9997]

[tcpout:splunk]
server = ip_address_host2:9997

Ciao.

Giuseppe