Why is one Linux universal forwarder host and data not showing up in Splunk Light?

[rlorenzon]

I have Splunk Light v6.2.3 instance with the Add-on for Nix V5.1.2 running. I have two universal forwarders v6.2.1 identically configured on two different Red Hat servers running the same operating system. I want to send all the logs from `/var/log/` to splunk. One server works and one server doesn't. I have verified connectivity from both servers. The one that doesn't work is slated to be the production instance of the one that works.

etc/system/local/inputs.conf:

[default]
host = host1
index = syslog
disabled = false
[monitor:///var/log]

etc/system/local/outputs.conf:

[tcpout]
defaultGroup = linux-group
disabled = 0
[tcpout:linux-group]
server = ##.##.##.32:514
[tcpout-server://##.##.##.32:514]

Pretty basic config. I have a Windows forwarder which also works fine.

I think I've read every Splunk doc there is and run every diagnostic I could. I've seen posts of others with a similar issue here and have verified every one of those answers marked as correct. There are some mention of associating the host with an index, but the steps only apply to the enterprise version. I can see data coming from both hosts in the receiver logs, it just doesn't show up in the interface. I've tried using different ports, reinstalling, and tried various versions just in case. Any help would be greatly appreciated!

[rlorenzon · Accepted solution]

Followed this great step by step document again:.

http://answers.splunk.com/answers/50082/how-do-i-configure-a-splunk-forwarder-on-linux.html

Was failing on step 7 as others have had. I manually created the directory:
/opt/splunkforwarder/etc/apps/search/local

and the inputs.conf file in it with:
[monitor:///var/log]
disabled = false

and it worked!

[aljohnson_splun]

@rlorenzon glad to hear you figured it out. Go ahead and accept your own answer to close the question. Thanks!

[rlorenzon]

Also, the hostnames are different as well as their GUID's in etc/instance.cfg - They were clean installations.

[rlorenzon]

Also, just noticed this message in the splunk interface with the hostname of the one that I can't see:

received event for unconfigured/disabled/deleted index='syslog' with source='source::/var/log/dmesg.old' host='host::infoleaf' sourcetype='sourcetype::backup_file' (1 missing total)

[aljohnson_splun]

So it sounds like your syslog index is all begarbled - is there anyway you can delete the syslog index and create a new one ? That is just what I would do, not necessarily a solution 😛

[satishsdange]

I am suspecting problem with your inputs.conf [monitor:///var/log]
Shouldn't it be [monitor:///var/log/*] ?

[aljohnson_splun]

If you look at the example here, it should be valid without the *.