topic Re: How can I debug a TCP feed on a heavy forwarder? in Getting Data In

How can I debug a TCP feed on a heavy forwarder?

a212830 — Tue, 06 Oct 2015 20:48:22 GMT

Hi,

I need to debug a tcp feed from a load-balancer, on a server where I don't have root or sudo. Is there a props config that I can make to put it into debug and see exactly what it's processing? I'm using a heavy forwarder.

Re: How can I debug a TCP feed on a heavy forwarder?

grijhwani — Tue, 06 Oct 2015 21:31:51 GMT

You could ask the sysadmins to install tcpdump for you. It's not a definitive answer, but I don't know of any debug mode as such.

Re: How can I debug a TCP feed on a heavy forwarder?

grijhwani — Tue, 06 Oct 2015 21:32:40 GMT

I would, however, like to be corrected and shown wrong.

Re: How can I debug a TCP feed on a heavy forwarder?

a212830 — Tue, 06 Oct 2015 21:42:52 GMT

Yeah, well, that adds a whole new set of processing.... If necessary, I will, but hoping there's a config entry that provide this info.

Re: How can I debug a TCP feed on a heavy forwarder?

nnmiller — Tue, 06 Oct 2015 23:46:35 GMT

Tcpdump would also require sudo or root privileges, since it puts the network interface into promiscuous mode.

You could try strace on the incoming splunk process, since you can run that as the splunk user, but I am not sure it would give you enough information.

Another option would be to have the heavy forwarder write out the raw events it is receiving to a syslog port. To send just a subset of data, in props.conf, apply the send_to_syslog transform to the load balancer's hostname:

[host::LBhostname]
TRANSFORMS-problemLB = syslog_debug

In transforms.conf, configure the syslog_debug transform to specify _SYSLOG_ROUTING as the DEST_KEY and the my_syslog_group target group as the FORMAT:

[syslog_debug]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

In outputs.conf, define the my_syslog_group target group for the non-Splunk server:

[syslog:my_syslog_group]
server = sysloghost.example.com:514

The syslog host here could be localhost, for example. You would also need to configure your syslog server process to accept incoming connections and write the log out to a file. Since Splunk already has access to the data stream you could run a syslog process on a non-privileged port as the Splunk user so the sysadmins don't have to be involved.

Reference: Route & Filter Data Specifically, look at "Replicate a subset of data to a third-party system".

Re: How can I debug a TCP feed on a heavy forwarder?

sloshburch — Wed, 07 Oct 2015 00:01:49 GMT

Splunk for Stream FTW! lol

Re: How can I debug a TCP feed on a heavy forwarder?

a212830 — Wed, 07 Oct 2015 02:40:47 GMT

Oohhh, this is interesting. Thanks.

Re: How can I debug a TCP feed on a heavy forwarder?

nnmiller — Wed, 14 Oct 2015 14:11:18 GMT

How is the debugging going?