topic Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method in Splunk Search

Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Tue, 29 Sep 2020 15:48:03 GMT

Hi Everyone,

I am using splunk stream. Packet stream to capture data from source and destination content fields.
For a persistent TCP connection i just cannot seem to break/split in separate events or lines.
Is there no way to so this?
Other advice appreciated.
I am willing to check other alternatives, willing to truncate the data, etc etc etc

Sample event Single TCP Conn Open/Close
{"endtime":"2017-09-17T15:30:47.271015Z","timestamp":"2017-09-17T15:30:36.440073Z","ack_packets_in":4,"ack_packets_out":5,"app":"tcp","bytes":645,"bytes_in":353,"bytes_out":292,"client_rtt":16,"client_rtt_packets":1,"client_rtt_sum":16,"connection":"192.168.100.3:65534","data_packets_in":1,"data_packets_out":0,"dest_ip":"192.168.100.3","dest_port":65534,"duplicate_packets_in":0,"duplicate_packets_out":0,"missing_packets_in":0,"missing_packets_out":0,"network_interface":"lo0","packets_in":6,"packets_out":5,"protocol_stack":"ip:tcp:unknown","server_rtt":40,"server_rtt_packets":2,"server_rtt_sum":81,"src_ip":"192.168.100.3","src_port":51448,"tcp_status":0,"time_taken":10830958,"SRCCNT":"68656c6c6f"}

Sample Event TCP persistent Stream
{"endtime":"2017-09-17T15:32:06.278243Z","timestamp":"2017-09-17T15:30:57.342570Z","ack_packets_in":3,"ack_packets_out":158,"app":"tcp","bytes":18484,"bytes_in":9624,"bytes_out":8860,"client_rtt":14,"client_rtt_packets":1,"client_rtt_sum":14,"connection":"192.168.100.3:65534","data_packets_in":153,"data_packets_out":0,"dest_ip":"192.168.100.3","dest_port":65534,"duplicate_packets_in":0,"duplicate_packets_out":0,"missing_packets_in":0,"missing_packets_out":0,"network_interface":"lo0","packets_in":157,"packets_out":158,"protocol_stack":"ip:tcp:unknown","server_rtt":33,"server_rtt_packets":154,"server_rtt_sum":5226,"src_ip":"192.168.100.3","src_port":51475,"tcp_status":0,"time_taken":68935687,"SRCCNT":"68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f"}

I need the above stream to be broken up into separate events. Sorta like a wireshark view

Thank You & appreciate any and all ideas/assistance.
Pinaki

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Sun, 17 Sep 2017 18:40:46 GMT

If the above is not at all possible, kindly let me know. I will choose alternate methods. Thanks

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

Sukisen1981 — Mon, 18 Sep 2017 09:42:07 GMT

not clear what you want...I need the above stream to be broken up into separate events. Sorta like a wireshark view??
Can you give a sample of how you would like the events to be broken up into?

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Tue, 29 Sep 2020 15:48:40 GMT

So let me clarify my objective - I am concerned only with the data i.e. src & dest content (extracted data field SRCCNT) with couple of other fields like RTT , dest/src ip/port timestamp etc.
In splunk stream - TCP persistent connection all the data values club together and the SRCCNT becomes a huge bulk field (due to TCPreassembly at the splunk) and I cannot work with the data. I ultimately need the data to be shown as below - Each as a Single Event

{"endtime":"2017-09-17T15:30:47.271015Z","timestamp":"2017-09-17T15:30:36.440073Z","ack_packets_in":4,"ack_packets_out":5,"app":"tcp","bytes":645,"bytes_in":353,"bytes_out":292,"client_rtt":16,"client_rtt_packets":1,"client_rtt_sum":16,"connection":"192.168.100.3:65534","data_packets_in":1,"data_packets_out":0,"dest_ip":"192.168.100.3","dest_port":65534,"duplicate_packets_in":0,"duplicate_packets_out":0,"missing_packets_in":0,"missing_packets_out":0,"network_interface":"lo0","packets_in":6,"packets_out":5,"protocol_stack":"ip:tcp:unknown","server_rtt":40,"server_rtt_packets":2,"server_rtt_sum":81,"src_ip":"192.168.100.3","src_port":51448,"tcp_status":0,"time_taken":10830958,"SRCCNT":"68656c6c6f"}

Thanx
Pinaki

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

Sukisen1981 — Mon, 18 Sep 2017 17:57:07 GMT

I am sorry , is till can not figure this out. You have 49391 in your sample 'would be' event but I can not see that number anywhere in your raw event, where is this number coming from? Also you have 61 , 56 etc, in your screen shot , I can not see them anywhere in your raw event...

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Tue, 19 Sep 2017 16:58:13 GMT

Hi Suki,

The above are just samples. 49391 is the port. 61/56 are the bytes. This is Wireshark.
This is TCP PERSISTENT Stream. TCP PERSISTENT Stream is a TCP connection is always in connected state till Closure by either client/server.

You can see this value in the pic? - 68656c6c6f. In a single TCP OPEN/Close this comes as 1 event in splunk.

However in a Persistent stream this become "68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f" in splunk. I want the similar line event view in splunk as shown in the wireshark image. This is the value I want in single line for TCP PERSISTENT Stream.

So hence "68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f68656c6c6f" should be broken in separate lines with supporting data
e.g. events
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f
68656c6c6f

hope the above clarifies your question. I just need to know if possible or not. If possible how. Thank you for your help.
Pinaki

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

lfedak_splunk — Tue, 19 Sep 2017 17:04:13 GMT

Hey @sukisen1981, thank you so much for contributing to Splunk Answers! 🙂 Please remember to press "add comment" instead of "add answer" when it's part of discussion. (I just converted this answer post to a comment.)

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

DalJeanis — Fri, 29 Sep 2017 14:50:50 GMT

@pinakicybermak - are all the SRCCNT values identical in length, or is there another way to distinguish where one ends and the next begins?

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Tue, 29 Sep 2020 16:04:44 GMT

Dear Dal,

SRCCNT values - Not identical.
SRCCNT values extracted from either dest_content/src_content in hex
no distinguishing start/end
for example 1 hex full packet maybe
"00ac00ef384eeee000000100ac00ef384eeee0000020000000ac00ef384eeee<128Bytes of data><128bytes>"
2nd packet may be
"00ac00ef2343234000000100ac00ef23432340000020000000ac00ef2343234<128Bytes of data><128bytes>"
3rd packet maybe
"00ab00ef1234000000100ab00ef12340000020000000ab00ef1234<128Bytes of data><128bytes>"
I am actually trying to parse ISO8583 msg which comes in hex. I am successful if the data is coming with 1 tcp open/close. When it comes in stream it looks like
SRCCNT = 00ac00ef384eeee000000100ac00ef384eeee0000020000000ac00ef384eeee<128Bytes of data><128bytes>00ac00ef2343234000000100ac00ef23432340000020000000ac00ef2343234<128Bytes of data><128bytes>00ab00ef1234000000100ab00ef12340000020000000ab00ef1234<128Bytes of data><128bytes>

Thank You
Pinaki

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

Richfez — Tue, 03 Oct 2017 11:15:16 GMT

Pinaki,

If I am reading this right, your problem is that Splunk/Stream reassembled the packets in a TCP flow, but you want to see the raw individual packets that made up that flow?

Have you tried faking it out by setting maxTcpReassemblyPacketCount in streamfwd.conf to 1 instead of the default 500000?

I do NOT know if that will work - well, I know it should work but I don't know if it will work usefully. It could very well just make a mess of things. 😞

Give that a try, though, and report back. If that does the trick I would be happy to convert this to an answer so you can accept it. If it does not, well, it may be a useful comment/trick for the next person who stumbles across this question.

Happy Splunking!
-Rich

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Thu, 05 Oct 2017 18:51:24 GMT

Dear Rich,

Thank you for your reply. Yes i have already tried that. It does work, but again not a perfect solution. If i could configure that to "ms" instead of "sec" the result would be better.

Thank You
Pinaki Banerjee

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

Richfez — Thu, 05 Oct 2017 19:03:14 GMT

When you say if you could configure that to "ms" instead of seconds, are you talking about the timestamp that Splunk is using isn't using the full precision available? That might be fixable.

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Sat, 07 Oct 2017 06:53:57 GMT

Dear Rich,

Clarification and apologies. I used the tcpConnectionTimeout (lowest value 1 sec) to almost solve the problem but I faced scenarios where I got multiple transactions with 1 sec.
I also used maxTcpReassemblyPacketCount = 1 but i was getting 2 records always.

If I have understood your comment correctly - "ms", "full precision" - This is an idea i havent tried. If this is doable then maybe it will break with full precision timestamp and i may get all the records including the breakdown handshakes etc. I want to try. Can you give me how to do it?

Thank You
Pinaki

Re: Splunk Stream split separate events for persistent tcp stream or Disable TCPREASSEMBLY or some other method

pinakicybermak — Sun, 05 Nov 2017 20:18:04 GMT

Hi everybody

After extensive research & self learning about the capabilities & limitations splunk, this is not feasible.
I have opted to use Protocol Data Input app.
Using Module Inputs I have managed to get the results.

Again thank you everyone for your support and answers.

Pinaki