topic Re: After installing and configuring universal forwarders, why can't I see the data in the indexer? in Getting Data In

After installing and configuring universal forwarders, why can't I see the data in the indexer?

lanilim16 — Tue, 30 Jun 2015 20:25:36 GMT

I have a universal forwarder installed in a few servers and I also have added the logs to be monitored for each. I'm not able to see the data in the indexer for some reason though. I've done the same steps before using the same versions and script, I'm not sure where else to look.

Splunk Universal Forwarder 6.2.1
Splunk 6.1.0 build - indexer

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

nce054 — Tue, 30 Jun 2015 20:36:29 GMT

Are you indexing the data to an index that exists? Try to specify index=main in inputs.conf on the Universal Forwarder.

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

MuS — Tue, 30 Jun 2015 20:38:41 GMT

Hi lanilim16,

first thing to do, run as admin the following search index=* earliest=0 latest=now
If you still don't see your events, run this as admin index=_internal sourcetype=splunkd metrics and check if your forwarders are sending anything
If you don't get anything from the forwarders, check any possible firewall blocking traffic or routing issues.
Last but not least login to the forwarder and check its config, like is it really configured to forward:
$SPLUNK_HOME/bin/ splunk list forward-server
or does it use the correct monitor stanza:
$SPLUNK_HOME/bin/ splunk cmd btool inputs list

Hope that helps ...

cheers, MuS

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

lanilim16 — Tue, 30 Jun 2015 21:37:13 GMT

This is a new index which I've already added. I see result from

 index=_internal sourcetype=splunkd metrics host="<server>"

so now I'm not sure why it's not working for that particular index. When I check in Indexes, there are no events for that index, however I'm sure I've added it right since I can see these from splunkd.log

06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///home/app_prod/jboss-as/domain/servers/server-one/log/server-one-*.log.
06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///home/app_prod/jboss-as/domain/servers/server-one/log/server.log.
06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Adding watch on path: /home/app_prod/jboss-as/domain/servers/server-one/log/server-one-*.log.
06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Adding watch on path: /home/app_prod/jboss-as/domain/servers/server-one/log/server.log.

and I see the file when I list monitor from the forwarder.

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

MuS — Tue, 30 Jun 2015 21:47:54 GMT

check for possible typos in the index option in inputs.conf for this monitor

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

frmaasdam — Wed, 01 Jul 2015 14:46:59 GMT

If it is a new index don't forget to set the rights correctly in de settings security for the admin rule. There you can set the index as one of the standard indexers to be able to search in.

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

lanilim16 — Thu, 02 Jul 2015 14:29:34 GMT

issue fixed on its own, the log had to rotate before it got indexed.

Re: After installing and configuring universal forwarders, why can't I see the data in the indexer?

mchapman5611 — Fri, 03 Jul 2015 18:13:12 GMT

In Splunk, go to "Settings" | "Forwarding and receiving"
In the Receive Data section, click 'Configure Receiving'
Click 'New'
Add port 9997