topic Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light? in All Apps and Add-ons

Why is one Linux universal forwarder host and data not showing up in Splunk Light?

rlorenzon — Thu, 16 Jul 2015 16:14:15 GMT

I have Splunk Light v6.2.3 instance with the Add-on for Nix V5.1.2 running. I have two universal forwarders v6.2.1 identically configured on two different Red Hat servers running the same operating system. I want to send all the logs from `/var/log/` to splunk. One server works and one server doesn't. I have verified connectivity from both servers. The one that doesn't work is slated to be the production instance of the one that works.

etc/system/local/inputs.conf:

[default]
host = host1
index = syslog
disabled = false
[monitor:///var/log]

etc/system/local/outputs.conf:

[tcpout]
defaultGroup = linux-group
disabled = 0
[tcpout:linux-group]
server = ##.##.##.32:514
[tcpout-server://##.##.##.32:514]

Pretty basic config. I have a Windows forwarder which also works fine.

I think I've read every Splunk doc there is and run every diagnostic I could. I've seen posts of others with a similar issue here and have verified every one of those answers marked as correct. There are some mention of associating the host with an index, but the steps only apply to the enterprise version. I can see data coming from both hosts in the receiver logs, it just doesn't show up in the interface. I've tried using different ports, reinstalling, and tried various versions just in case. Any help would be greatly appreciated!

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

rlorenzon — Thu, 16 Jul 2015 16:20:45 GMT

Also, the hostnames are different as well as their GUID's in etc/instance.cfg - They were clean installations.

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

rlorenzon — Thu, 16 Jul 2015 17:33:13 GMT

Also, just noticed this message in the splunk interface with the hostname of the one that I can't see:

received event for unconfigured/disabled/deleted index='syslog' with source='source::/var/log/dmesg.old' host='host::infoleaf' sourcetype='sourcetype::backup_file' (1 missing total)

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

aljohnson_splun — Thu, 16 Jul 2015 17:40:45 GMT

So it sounds like your syslog index is all begarbled - is there anyway you can delete the syslog index and create a new one ? That is just what I would do, not necessarily a solution 😛

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

satishsdange — Fri, 17 Jul 2015 07:22:44 GMT

I am suspecting problem with your inputs.conf [monitor:///var/log]
Shouldn't it be [monitor:///var/log/*] ?

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

rlorenzon — Fri, 17 Jul 2015 17:26:34 GMT

Followed this great step by step document again:.

http://answers.splunk.com/answers/50082/how-do-i-configure-a-splunk-forwarder-on-linux.html

Was failing on step 7 as others have had. I manually created the directory:
/opt/splunkforwarder/etc/apps/search/local

and the inputs.conf file in it with:
[monitor:///var/log]
disabled = false

and it worked!

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

aljohnson_splun — Fri, 17 Jul 2015 18:54:30 GMT

If you look at the example here, it should be valid without the *.

Re: Why is one Linux universal forwarder host and data not showing up in Splunk Light?

aljohnson_splun — Fri, 17 Jul 2015 18:55:09 GMT

@rlorenzon glad to hear you figured it out. Go ahead and accept your own answer to close the question. Thanks!