I have a universal forwarder installed in a few servers and I also have added the logs to be monitored for each. I'm not able to see the data in the indexer for some reason though. I've done the same steps before using the same versions and script, I'm not sure where else to look.
Splunk Universal Forwarder 6.2.1
Splunk 6.1.0 build - indexer
Are you indexing the data to an index that exists? Try to specify
inputs.conf on the Universal Forwarder.
first thing to do, run as admin the following search
index=* earliest=0 latest=now
If you still don't see your events, run this as admin
index=_internal sourcetype=splunkd metrics and check if your forwarders are sending anything
If you don't get anything from the forwarders, check any possible firewall blocking traffic or routing issues.
Last but not least login to the forwarder and check its config, like is it really configured to forward:
$SPLUNK_HOME/bin/ splunk list forward-server
or does it use the correct monitor stanza:
$SPLUNK_HOME/bin/ splunk cmd btool inputs list
Hope that helps ...
This is a new index which I've already added. I see result from
index=_internal sourcetype=splunkd metrics host="<server>"
so now I'm not sure why it's not working for that particular index. When I check in Indexes, there are no events for that index, however I'm sure I've added it right since I can see these from splunkd.log
06-30-2015 15:55:16.757 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/app_prod/jboss-as/domain/servers/server-one/log/server-one-*.log. 06-30-2015 15:55:16.757 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/app_prod/jboss-as/domain/servers/server-one/log/server.log. 06-30-2015 15:55:16.757 -0400 INFO TailingProcessor - Adding watch on path: /home/app_prod/jboss-as/domain/servers/server-one/log/server-one-*.log. 06-30-2015 15:55:16.757 -0400 INFO TailingProcessor - Adding watch on path: /home/app_prod/jboss-as/domain/servers/server-one/log/server.log.
and I see the file when I list monitor from the forwarder.
check for possible typos in the index option in
inputs.conf for this monitor
If it is a new index don't forget to set the rights correctly in de settings security for the admin rule. There you can set the index as one of the standard indexers to be able to search in.
issue fixed on its own, the log had to rotate before it got indexed.
In Splunk, go to "Settings" | "Forwarding and receiving"
In the Receive Data section, click 'Configure Receiving'
Add port 9997