We have installed Splunk universal forwarder on a remote server but logs are not getting forwarded to Indexer.
I have tried to troubleshoot this issue but could not do so. Can you please help me to get rid of this issue.
Below are the steps I have tried so far.
root@host1:/opt/splunkforwarder/etc/system/local# telnet host2 9997
Connected to host2
Escape character is '^]'.
root@host1:/opt/splunkforwarder/etc/system/local# cat outputs.conf
defaultGroup = splunk
server = host2.ce.corp:9997
root@host1:/opt/splunkforwarder/etc/system/local# cat inputs.conf
host = host1
disabled = false
sourcetype = web_haprx
index = webmethods_haprx
root@host1:/opt/splunkforwarder/bin# ./splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Configured but inactive forwards:
Can you please help me to fix this issue?
What do you mean by "logs are not getting forwarded"? How do you know that?
Do you have any errors in your /opt/splunkforwarder/var/log/splunk/splunkd.log on your forwarder?
You can also check your _internal index for any logs from your forwarder host. If you have any logs from the forwarder, the forwarding as such is working properly so if you're not getting your events there's a problem in other part of your config.
| tstats count where index=_internal by host
for the last day or so and see whether you're getting data from that forwarder at all.
Hi @PickleRick ,
Q:-What do you mean by "logs are not getting forwarded"? How do you know that?
It is because when am using network port UDP:5514, I can see logs into Splunk but when am trying to forward logs into Splunk. We are unable to do so. we are trying to send /var/log/messages
Q:-Do you have any errors in your /opt/splunkforwarder/var/log/splunk/splunkd.log on your forwarder?
No, we could not see any errors. It was there earlier but we fixed.
02-08-2022 15:39:15.907 +1100 ERROR TailingProcessor - Input stanza path, 'var/log/messages' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.
Q:- whether you're getting data from that forwarder at all?
Yes, we are getting data. Below is the sample.
Feb 14 22:35:27 host1 Container_ImageInventory: Container image name () is improperly formed and could not be parsed in SetRepositoryImageTag
OK, if you're sending data straight to udp input on your indexer it has nothing to do with the forwarder so it has no diagnostic value here.
About the log you showed - well, that's kinda interesting. If you only have an input defined for /var/log/messages - how are you getting the log about that Container_ImageInventory?
By default after installation and definition of output, the UF should only forward its own internal logs to _internal index.
Do a "splunk list monitor" on your forwarder. And "splunk btool inputs list --debug".
And see what inputs you have defined and running.
somethimes tyhere isn't a correct resolution of the hostname, so, please, try using IP address and than add a row to your outputs.conf:
[tcpout] defaultGroup = splunk [tcpout-server://ip_address_host2:9997] [tcpout:splunk] server = ip_address_host2:9997