Unix Forwarder is not Sending Logs

[scc00]

A newly installed Splunk Forwarder is not sending logs. I have configured the Receiver on Splunk Enterprise and on the forwarder: inputs.conf and outputs.conf. Please find the details of both below. We have tested the connectivity between the forwarder and splunk enterprise and they are connected. We have resolved all the errors found in the splunkd.log on the forwarder. Any thoughts as to what I may be missing?

Outputs.conf

[default]

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = *.*.*:6060
indexAndForward=false
[tcpout-server://IP:6060]

Inputs.conf

[default]
host = *.*.*
index = unix
sourcetype = unix
[monitor:/var/log/messages]
_TCP_ROUTING=*.*.*:6060
[monitor:/var/log/centrify_mapper_error.log]
_TCP_ROUTING=*.*.*:6060

[yong_ly]

Here are three things you can check:

1. Are you able to see the splunkd.logs from your server on the indexer? e.g if you search "index=_internal source=*splunkd.log host=.. - This should be visible if the indexer and forwarder are correctly configured to send/receive data.

2. Check that the splunk user has access to those logs - if you can't read them as splunk user then it can't forward them.

3. Do a search through "ALL TIME" for any logs from that host. I know it sounds stupid but I once had this issue where I thought it wasn't sending but it was actually using the wrong timestamps so events were being indexed on a date that was earlier than my search range by a few years.

If all else fails, you could turn the logging on the forwarder to DEBUG to get more information. Under $SPLUNK_HOME/etc/log.cfg.. Try setting the logging on the TailingProcessor and WatchedFile components to DEBUG and see what turns up.

[scc00]

I am able to see the splunkd.log from the server on the indexer.
We made sure the splunk user has access to the logs.
No logs found when i searched through all time.

I'll have to turn on the debugging.

[delink]

Your monitor stanzas are wrong in your inputs.conf file. They should read as:

[monitor:///var/log/messages]

Remember you can always check in $SPLUNK_HOME/var/log/splunk/splunkd.log to get an idea of what errors might exist during startup and operation.

I would also highly recommend you take a look at the nix app and its related TA for reading these kinds of logs. It will get everything into a format that other apps will be able to take advantage of. Plus, it means less work for you.

[scc00]

I do not get any results

[delink]

That is what you want to see. When you search on index=unix on your search head, do you get results?

[scc00]

I updated my inputs.conf,restarted the forwarder.Restart was clear of errors and no errors from splunkd.log either. Just things like:

Parsing configuration stanza: monitor:///var/log/messages.
Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
Adding watch on path: /var/log/centrify_mapper_error.log.
Adding watch on path: /var/log/messages.
-0500 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
-0500 INFO TcpOutputProc - Connected to idx=...:6060

[Ayn]

Are other forwarders working in your setup? Also your monitor statements seem to be off. It should be "[monitor:///...]", not "[monitor:/...]".