How can I debug a TCP feed on a heavy forwarder?

[a212830]

Hi,

I need to debug a tcp feed from a load-balancer, on a server where I don't have root or sudo. Is there a props config that I can make to put it into debug and see exactly what it's processing? I'm using a heavy forwarder.

[sloshburch]

Splunk for Stream FTW! lol

[nnmiller]

Tcpdump would also require sudo or root privileges, since it puts the network interface into promiscuous mode.

You could try strace on the incoming splunk process, since you can run that as the splunk user, but I am not sure it would give you enough information.

Another option would be to have the heavy forwarder write out the raw events it is receiving to a syslog port. To send just a subset of data, in props.conf, apply the send_to_syslog transform to the load balancer's hostname:

[host::LBhostname]
TRANSFORMS-problemLB = syslog_debug

In transforms.conf, configure the syslog_debug transform to specify _SYSLOG_ROUTING as the DEST_KEY and the my_syslog_group target group as the FORMAT:

[syslog_debug]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

In outputs.conf, define the my_syslog_group target group for the non-Splunk server:

[syslog:my_syslog_group]
server = sysloghost.example.com:514

The syslog host here could be localhost, for example. You would also need to configure your syslog server process to accept incoming connections and write the log out to a file. Since Splunk already has access to the data stream you could run a syslog process on a non-privileged port as the Splunk user so the sysadmins don't have to be involved.

Reference: Route & Filter Data Specifically, look at "Replicate a subset of data to a third-party system".

[a212830]

Oohhh, this is interesting. Thanks.

[nnmiller]

How is the debugging going?

[grijhwani]

You could ask the sysadmins to install tcpdump for you. It's not a definitive answer, but I don't know of any debug mode as such.

[grijhwani]

I would, however, like to be corrected and shown wrong.

[a212830]

Yeah, well, that adds a whole new set of processing.... If necessary, I will, but hoping there's a config entry that provide this info.