topic Graylog sending to SPLUNK over 9997 in Getting Data In

Graylog sending to SPLUNK over 9997

pfabrizi — Tue, 19 Jun 2018 11:16:52 GMT

I have Graylog forwarding to a UF over port 9997 and I see events streaming in but not being picked up by SPLUNK. I have a inputs.conf set to [splunktcp:9997] . I tried to setup a syslog.conf to tream to a file but realized that is port 514 and 9997. Can any one provide some debugging hints?
How can I see if the events are getting picked up by the UF and just not forwarding?
I looked in the Metrics.log but see nothing, is that the correct place to look?

Thanks!

Re: Graylog sending to SPLUNK over 9997

pfabrizi — Tue, 19 Jun 2018 11:20:48 GMT

Sorry, I wasn't seeing this yesterday but I am today.

06-19-2018 07:16:00.830 -0400 ERROR TcpInputProc - Message rejected. Received unexpected message of size=842019128 bytes from src=10.00.0.7:52640 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

I am going to go back to our Graylog folks and see if they can decrease the payload, is this correct?

Thanks!

Re: Graylog sending to SPLUNK over 9997

s2_splunk — Tue, 19 Jun 2018 21:48:07 GMT

You are trying to send to a port that expects the (proprietary) Splunk-2-Splunk protocol; the message indicates that: "Possible invalid source sending data to splunktcp port". It will not understand the wire format Graylog is using.
You may be more successful by creating a network input for a different port and use that as your Graylog destination.

Re: Graylog sending to SPLUNK over 9997

pfabrizi — Tue, 19 Jun 2018 22:10:23 GMT

Hi SSIEVENT,
I was just thinking that and in the process to tell the Graylog folks to send under another port. I am using a UF, so I have no UI, but i should be able to setup a inputs.conf with a tcp listener.

Thank You!

Re: Graylog sending to SPLUNK over 9997

yannK — Tue, 19 Jun 2018 22:50:30 GMT

Usually the port 9997 is used for the splunk protocol splunktcp. (only used by splunk forwarders)
If your "graylog" software is sending logs, it is probably not using this protocol.

looking at the internet, it seems that some people created code to have graylog send data over TCP to splunk :
https://github.com/graylog-labs/graylog-plugin-splunk

If it is sending data as syslog , please setup splunk to listen to UDP or TCP on a different port, and try send the data to it to see. (you may have to create a sourcetype to get proper event parsing)

If the graylog is able to send data to a splunk HEC "http event collector" API, try to setup such an input on splunk, grab the token, and use it to configure the graylog sender.

Re: Graylog sending to SPLUNK over 9997

s2_splunk — Tue, 19 Jun 2018 23:12:20 GMT

The link I sent also includes instructions on how to do it using inputs.conf (or the CLI).

Re: Graylog sending to SPLUNK over 9997

pfabrizi — Tue, 29 Sep 2020 20:06:25 GMT

OK, I changed the port to 9996. I see it listening on that port. I no longer get the error messages but I am not seeing any data flow to indexer.
am I missing anything? This is running on a Linux UF.

This is my inputs.conf
[tcp://9996]
index=wineventlog

I see these messages in the metrics.log but don't know what they mean.

06-20-2018 10:35:13.269 -0400 INFO Metrics - group=tcpin_connections, 10.xx.xx.4:51718:9996, connectionType=raw, sourcePort=51718, sourceHost=server.doamin.net, sourceIp=10.xx.xx.4, destPort=9996, kb=0.00, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.01, _tcp_Kprocessed=2.14, _tcp_eps=0.00, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.00, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00

Re: Graylog sending to SPLUNK over 9997

yannK — Wed, 20 Jun 2018 16:14:20 GMT

So we see some connections coming in. (do the sum of kb over some time to see the volume).

now you need to get a proper parsing.
- define a sourcetype on your input on the UF
- on the indexers, define the sourcetype in props.conf with the proper rules to : break the events, find the timestamp, consider multiline events, define the timezone etc..

If you are not sure, try with sourcetype=syslog (on the UF) and see what it does.

Re: Graylog sending to SPLUNK over 9997

pfabrizi — Thu, 21 Jun 2018 09:52:17 GMT

Thank You!