topic Re: TCP:9514 logs doesn't show in Splunk in Getting Data In

TCP:9514 logs doesn't show in Splunk

krishnaar — Tue, 22 Nov 2016 00:21:02 GMT

Hi Team,

I have configured a Cisco router to send syslogs to Splunk over TCP port 9514. But that doesn't show up in the Splunk server.

Any log sent over UDP 514 is showing up in the Splunk. Yes, Splunk has a TCP 9514 listener.

I can telnet to splunk:9514 from router.

I don't know what else to check. Any help would be appreciated.

Krishna

Re: TCP:9514 logs doesn't show in Splunk

Richfez — Tue, 22 Nov 2016 00:50:37 GMT

Hi, krishnaar. I answered something similar just a bit ago, with a long, drawn out series of steps to find out where it's going wrong. That answer is here.

But you probably don't have to read all that, I expect it's going into a different index than you are expecting or the timestamps are off and you are just missing them in your search. So try this:

index=*

Run over All Time.

Once you start seeing them, you can stop the search and figure out a) what index they are in (probably one you can't search by default) or b) if the timestamps are all wrong and they all show up "last year."

Let us know, and Happy Splunking!
-Rich

Re: TCP:9514 logs doesn't show in Splunk

masonmorales — Tue, 22 Nov 2016 00:51:38 GMT

If you run tcpdump do you see data coming into TCP/9514?

/usr/sbin/tcpdump -i <interface> port 9514

Re: TCP:9514 logs doesn't show in Splunk

krishnaar — Tue, 22 Nov 2016 03:55:45 GMT

This makes me more confused.

I ran a real-time monitoring for "10.10.26.* AND index=" where 10.10.26. is the IP subnet of client which send the logs to Splunk. And I ran a tcpdump at the Splunk to capture the return packet.

[root@Server-Name bin]# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:29:47.475270 IP 10.10.24.14.9514 > 10.10.26.9.62640: . ack 2480355949 win 20416
19:29:48.475548 IP 10.10.24.14.9514 > 10.10.26.9.62640: . ack 226 win 20880
19:30:21.517224 IP 10.10.24.14.9514 > 10.10.26.9.46146: . ack 2104469992 win 5840
19:30:23.043887 IP 10.10.24.14.9514 > 10.10.26.9.62640: . ack 458 win 21344
19:31:07.721030 IP 10.10.24.14.9514 > 10.10.26.9.62640: . ack 690 win 21808
19:31:21.095833 IP 10.10.24.14.9514 > 10.10.26.9.46146: F 0:0(0) ack 2 win 5840

I see that the Splunk server(10.10.24.14) is replying back to the original client from TCP 9514.

But no logs show up in the Splunk portal.

Output from splunk list inputstatus

Raw:tcp :
514:Router Name
time opened = 2016-11-21T13:54:56-0800

    9514:10.10.26.9                                ----> This is the actual client.
            time opened = 2016-11-21T19:16:50-0800

tcp_raw:listenerports :
514

    9514

    9515

Krishna

Re: TCP:9514 logs doesn't show in Splunk

krishnaar — Tue, 22 Nov 2016 03:57:14 GMT

Looks like the Splunk is getting data on TCP 9514 but its not processing that data.

Re: TCP:9514 logs doesn't show in Splunk

krishnaar — Wed, 23 Nov 2016 05:45:05 GMT

Any help would be appreciated!

Re: TCP:9514 logs doesn't show in Splunk

Richfez — Wed, 23 Nov 2016 12:47:21 GMT

What does your inputs.conf look like for those stanzas?

Also run the search index=* 10.10.26.9 over all time (not all time real time, but just simply all time). If for some reason the timestamps are coming in as of a month ago, you won't see them in a current RT search because they'll show up in October.

Please be sure to use the "code" button in the formatting tool bar - it really helps posts to maintain special characters like asterisks.

Re: TCP:9514 logs doesn't show in Splunk

jkat54 — Wed, 23 Nov 2016 13:36:03 GMT

There are buffers for TCP & UDP inputs in Splunk. You'll need to send 0.75 to 1.5 megabytes of data over the port before Splunk will flush from buffer to the index. My guess is you havent sent enough data over the port yet.

Or you can put this on your TCP/UDP input stanza:

_rcvbuf = 0

Example:

[tcp://<remote server>:9514]
...
_rcvbuf = 0

https://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Monitornetworkports

I wouldnt recommend running without a buffer once you have more data coming in. It's there for a reason, and helps tremendously with high volume network inputs.