Not able to forward udp messages from Universal fowarder. (v4.2.3)

[asingla]

I have installed a universal forwarder and configure it to listen on two ports tcp:55555 and udp:33333 and to forward messages to the main splunk indexer.

inputs.conf is

[tcp://55555]

[udp://33333]

output.conf looks like

[tcpout:group1]

server=XXX.XX.XX.XX:44444

I am able to see all the messages which are being sent over tcp:55555 in the search app but not seeing messages sent over udp:33333. Am I missing something to enable udp forwarding or something?

Please help.

[asingla · Accepted solution]

With the help of splunk support team, able to figure out the reason for this. It was the message format. I was trying with very small messages like "message:mymessage" but if my message looks like datetime message:mymessage, I start seeing these messages. By default splunk looks for certain pattern on indexer to treat the message as events and datetime is one of those. Splunk can be trained by editing the props.conf to treat messages without dateTime to treat as an event. Something like this.

**Excerpt of prpos.conf on YOUR SPLUNK INDEXER

[source::udp:33333]

DATETIME_CONFIG = CURRENT

MUST_BREAK_AFTER = /mymessage/g

[_d_]

Two other things that I would check for clues are the following log files: $SPLUNK_HOME/var/log/splunk/splunkd.log and metrics.log. See if there is any info regarding both of your input ports.

[asingla]

I see this log message in splunkd.log on main splunk indexer whenever I restart my uiniversal forwarder.

channel "source::udp:33333|host::XXX.XX.XX.XX|udp:33333|remoteport::38626" ended without a done-key

I search and find some similar post but the solution they proposed that use Heavy forwarder.

http://splunk-base.splunk.com/answers/7365/ended-without-a-done-key

Any body has any idea what is the solution for Universal forwarder?

[asingla]

Update: I am trying to figure out things by looking at the tcpdump. I see some packets containing data like sourcetype=udp:33333. It could be that forwarder is forwarding the messages but there is something wrong with the main splunk indexer that it is not showing these messages on the search app summary page.

[asingla]

I see something in metrics.log when I send the message. (I have masked the IPs before posting). I still need to learn to read this log file But here are the messages.

10-13-2011 18:11:19.716 +0000 INFO Metrics - group=per_sourcetype_thruput, series="udp:33333", kbps=0.001229, eps=0.096774, kb=0.038086, ev=3, avg_age=0.000000, max_age=0

I only see this message once I send the message else this lines does not appear in the log file. Looks like universal forwarder is receiving the udp messages.

I also see this

10-13-2011 18:12:52.795 +0000 INFO Metrics - group=tcpout_connections, group1:XXX.XX.XX.XX:44444:0, sourcePort=8089, destIp=XXX.XX.XX.XX, destPort=44444, _tcp_Bps=54.57, _tcp_KBps=0.05, _tcp_avg_thruput=0.06, _tcp_Kprocessed=470, _tcp_eps=0.13
10-13-2011 18:12:52.795 +0000 INFO Metrics - group=udpin_connections, 33333, sourcePort=33333, _udp_bps=0.00, _udp_kbps=0.00, _udp_avg_thruput=0.00, _udp_kprocessed=0.00, _udp_eps=0.00

Do you have any idea what udpin_connections group is?

Do I have to define something else to let forwarder know that messages received from udp port should be forwarded to tcp:44444?

[Drainy]

I'm slightly confused. Are you receiving UDP and TCP on the forwarder or do you mean on the indexer? I ask as you say you get 55555 on the indexer.

Firstly I believe the format should include a colon ":" before the port number, e.g. udp://:33333.
That might not make a massive difference however (especially if it looks like its working).
Also you need to ensure you have got the indexer setup to listen correctly for incoming data as per;
http://docs.splunk.com/Documentation/Splunk/4.2.3/Data/Configureyourinputs

If you edit the config directly bear in mind a restart will be required for the settings to take effect.
If it is Syslog data you want to forward from a server you could always either set the syslog server to forward itself or the UF (Universal Forwarder) to just monitor the log files, e.g. /var/log.... but I am making assuming a syslog link at the end there.

If this is no help then please feel free to update your question or comment with more detail 🙂

EDIT: Oh, just saw the other answer, good point - check your firewalls. Sometimes TCP is allowed as it is subject to packet analysis but UDP is blocked by default. (By check firewalls I mean clientside and any hardware firewalls)

[_d_]

The colon should not matter much, although it is a good practice to always have it.

[_d_]

Hi asingla, please run the commands below and check the output for any clues that might lead to a resolution:

$SPLUNK_HOME/bin/splunk list udp

$SPLUNK_HOME/bin/splunk list tcp

I'd also check any firewalls/ACLs.

Best,
d.

[asingla]

Thanks to all for your response. Actually I have already tried everything as you guys suggested before posting this message but it didn't work for me.

Just to be clear again, I setup a universal forwarder to receive messages on udp://33333 and tcp://55555. I also configure this universal forwarder to forward messages to main splunk indexer which is configured to receive messages on 44444. This setup is working for all the messages which are being sent to Universal forwarder on tcp://55555. I can see those messages in the search app on the main splunk indexer. But I don't see any messages which are sent to universal forwarder on udp://33333.

This is what I did to verify certain things as you all suggested.

1) I wrote my own small program to test if the UDP (on same port 33333) is working on the machine where I installed universal forwarder. I was able to receive messages.

2) $SPLUNK_HOME/bin/splunk list udp

Listening for input on the following UDP ports:
33333

$SPLUNK_HOME/bin/splunk list tcp

Splunk is listening for data on ports:
55555 for data from any host

3) I restart universal forwarder after any config changes.