Deployment Architecture

Unix Forwarder is not Sending Logs

scc00
Contributor

A newly installed Splunk Forwarder is not sending logs. I have configured the Receiver on Splunk Enterprise and on the forwarder: inputs.conf and outputs.conf. Please find the details of both below. We have tested the connectivity between the forwarder and splunk enterprise and they are connected. We have resolved all the errors found in the splunkd.log on the forwarder. Any thoughts as to what I may be missing?

Outputs.conf

[default]

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = *.*.*:6060
indexAndForward=false
[tcpout-server://IP:6060]

Inputs.conf

[default]
host = *.*.*
index = unix
sourcetype = unix
[monitor:/var/log/messages]
_TCP_ROUTING=*.*.*:6060
[monitor:/var/log/centrify_mapper_error.log]
_TCP_ROUTING=*.*.*:6060
Tags (1)
0 Karma

yong_ly
Path Finder

Here are three things you can check:

  1. Are you able to see the splunkd.logs from your server on the indexer? e.g if you search "index=_internal source=*splunkd.log host=.. - This should be visible if the indexer and forwarder are correctly configured to send/receive data.

  2. Check that the splunk user has access to those logs - if you can't read them as splunk user then it can't forward them.

  3. Do a search through "ALL TIME" for any logs from that host. I know it sounds stupid but I once had this issue where I thought it wasn't sending but it was actually using the wrong timestamps so events were being indexed on a date that was earlier than my search range by a few years.

If all else fails, you could turn the logging on the forwarder to DEBUG to get more information. Under $SPLUNK_HOME/etc/log.cfg.. Try setting the logging on the TailingProcessor and WatchedFile components to DEBUG and see what turns up.

0 Karma

scc00
Contributor

I am able to see the splunkd.log from the server on the indexer.
We made sure the splunk user has access to the logs.
No logs found when i searched through all time.

I'll have to turn on the debugging.

0 Karma

delink
Communicator

Your monitor stanzas are wrong in your inputs.conf file. They should read as:

[monitor:///var/log/messages]

Remember you can always check in $SPLUNK_HOME/var/log/splunk/splunkd.log to get an idea of what errors might exist during startup and operation.

I would also highly recommend you take a look at the nix app and its related TA for reading these kinds of logs. It will get everything into a format that other apps will be able to take advantage of. Plus, it means less work for you.

0 Karma

scc00
Contributor

I do not get any results

0 Karma

delink
Communicator

That is what you want to see. When you search on index=unix on your search head, do you get results?

0 Karma

scc00
Contributor

I updated my inputs.conf,restarted the forwarder.Restart was clear of errors and no errors from splunkd.log either. Just things like:

Parsing configuration stanza: monitor:///var/log/messages.
Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
Adding watch on path: /var/log/centrify_mapper_error.log.
Adding watch on path: /var/log/messages.
-0500 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
-0500 INFO TcpOutputProc - Connected to idx=...:6060

0 Karma

Ayn
Legend

Are other forwarders working in your setup? Also your monitor statements seem to be off. It should be "[monitor:///...]", not "[monitor:/...]".

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...