Getting Data In

How to troubleshoot the Universal Forwarder when it is not sending events to the indexer?

sarthakb
Explorer

We have a existing infrastructure of Splunk where events are passed from multiple Linux boxes to Splunk indexers.

We recently have installed Splunk forwarder in a Windows box. When we search in Splunk using that host name, we don't see the events.

We have checked the logs with the following observation
- It is picking up new monitor config.
- No error is reported in Splunkd.log

Can you please share the troubleshooting steps for the forwarder? Can forwarder log files help us pin point - if forwarder at all sending the events to Indexer?

0 Karma
1 Solution

sarthakb
Explorer

Interesting Observation - The forwarder is able to send data to indexer in each line does not starts with a date time.

e.g

12/13/2016 12:45:77.907 -0500 Some content 

The above line fails

12/13/2016 Some content 

Above line works

Seems like forwarder is trying to parse date time.

Is there a way to forcefully tell forwarder not to parse datetime?

View solution in original post

0 Karma

sarthakb
Explorer

Interesting Observation - The forwarder is able to send data to indexer in each line does not starts with a date time.

e.g

12/13/2016 12:45:77.907 -0500 Some content 

The above line fails

12/13/2016 Some content 

Above line works

Seems like forwarder is trying to parse date time.

Is there a way to forcefully tell forwarder not to parse datetime?

0 Karma

nekbote
Path Finder

@ sarthakb refer to below thread...you can refer to my answer there to get some tips and also others answers as well

https://answers.splunk.com/answers/5590/could-not-send-data-to-the-output-queue.html#answer-466859

As @chrishartsoc mentioned...your starting point is checking splunkd and metrics log on the forwarder.

0 Karma

ddrillic
Ultra Champion

Good place to start at I can't find my data!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

So the data is being forwarded correctly to your indexer(s) but you cannot find the host when searching?

Can you check the outputs.conf on the forwarder to verify you have the correct hostname there?

C:\SPLUNK_HOME\etc\system\local\outputs.conf

0 Karma

sarthakb
Explorer

Hi, Thanks for the quick reply.

I dont know if the data is being forwarded correctly. I am looking for guidance to confirm that.

regards,
Sarthak

0 Karma

chrishartsock
Path Finder

Along the lines of skoelpin's comment, it sounds like there may be an issue sending from the UF to the indexer. Can you see in the splunkd.log on the UF that the UF is successfully connecting to the indexer?

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...