topic Using TCP data input to receive from switch sending data in raw format protocol buffers in Getting Data In

Using TCP data input to receive from switch sending data in raw format protocol buffers

sawgata12345 — Tue, 29 Sep 2020 20:18:55 GMT

Hi,
I am trying to receive data in splunk using TCP Data input from switch at port 20010. The data is in raw format(send via grpc using protocol buffers). Splunk is receiving the data and adding to event but it is like encrypted format in which it is being transferred.
Sample below:

Blockquote

\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0\x00\x00\xFF\xFF\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00%\x00\x00\x00:methodPOST@:path&/mdt_dialout.gRPCMdtDialout/MdtDialout@
:authority:57010@
grpc-encodingidentity@grpc-accept-encodingidentity,deflate,@content-typeapplication/grpc@
user-agent+grpc-c++/1.0.0 grpc-c/1.0.0 (linux; )@grpc-timeout30S\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFB
wIObw*9728zrtIOb*3170304zwtIOb*3170304zrtIOint*13785891zwtIOint*562518*

Blockquote

Is it possible to run any python or any other code to get this data while it is being received and convert to json/anyother format then send to the splunk index.

I didn't find any place to include any script file which modifies the TCP received input before sending to index.
Is there any way to do that?

Alternative I have done is a seperate python script to receive the data from switch and parse it and send to splunk and placed the python script inside an AddOn. But I want to use the direct method that splunk receives from switch directly via TCP and parse it via some script.

Re: Using TCP data input to receive from switch sending data in raw format protocol buffers

renjith_nair — Tue, 03 Jul 2018 11:50:03 GMT

Hi ,

You could use https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/Monitornetworkports and then use props and transforms to modify data before indexing

Re: Using TCP data input to receive from switch sending data in raw format protocol buffers

sawgata12345 — Tue, 29 Sep 2020 20:19:59 GMT

Hi Renjith,
I have used the same link to create the TCP input but my issue was to parse the output.
I had done it in python and created a AddOn by converting the raw input into dictionary and then parsing and taking out the required fields and forming specific json and send to Splunk via sdk.
I checked the prof.conf and transform.conf from the links where only option is writing regex to parse the inputs.
Here is How the input comes from switch:

Blockquote
node_id_str: "switch1"
encoding_path: "query:10050"
collection_id: 1100
msg_timestamp: 1515492081100
data_gpbkv {
fields {
name: "keys"
fields {
name: "query:10050"
string_value: "query:10050"
}
}
fields {
name: "content"
fields {
fields {
name: "values"
fields {
fields {
name: "1"
fields {
fields {
name: "port"
string_value: "fc2/4"
}
fields {
name: "scsi_target_count"
string_value: "0"
}
}
}
fields {
name: "2"
fields {
fields {
name: "port"
string_value: "fc2/3"
}
fields {
name: "scsi_target_count"
string_value: "1"
}
}
}
fields {
name: '3"
fields {
fields {
name: "port"
string_value: "fc16/3"
}
fields {
name: "scsi_target_count"
string_value: "3"
}
}
}
//similarly there are details for "name":"4", "name":"5" ... so on till "name":"40"
}
}
}
}
}
Blockquote

Here i need to go through a loop and take out the values like below:
from the output from above code:
name: "scsi_target_count"
string_value: "3"
we need to make like "scsi_target_count":"3" . like key value pair and form json as shown below to send to splunk. This i have done in python for the AddOn. I thought to call python script from the TCP input and get the json formated output and then send to indexer.

{"name":"1","node_id_str": "switch1","msg_timestamp": "1515492081100","port": "fc2/4","scsi_target_count":"0"}
{name: "2","node_id_str": "switch1","msg_timestamp": "1515492081100","port": "fc2/3","scsi_target_count":"1"}

similarly more ports are there so multiple json objects with the common part for all is
"node_id_str": "switch1","msg_timestamp": "1515492081100" and the varying part is "port": "xx/x","scsi_target_count":"count" .

Is there any way in regex in transform.conf to get similar output?

Re: Using TCP data input to receive from switch sending data in raw format protocol buffers

renjith_nair — Thu, 05 Jul 2018 10:00:00 GMT

If you have a lot of data processing, have a look at this blog also : https://www.splunk.com/blog/2014/11/11/protocol-data-inputs.html