topic Re: Streaming app - netflow - data not parsed in Getting Data In

Streaming app - netflow - data not parsed

heskez — Tue, 29 Sep 2020 18:40:20 GMT

Hi There, I've netflow forwarding configured and streaming app installed.
I'm receiving the netflow data:

sourcetype="stream.netflow"

3/24/18
1:04:26.190 PM
{ [-]
app:
bytes: 0
count: 1
dest_ip: 8.8.8.8
dest_port: 53
drop_packet_count: 0
endtime: 2018-03-24T12:04:26.190502Z
packets: 0
packets_in: 1
packets_out: 1
src_ip: 192.168.1.2
src_mac:
src_port: 54808
sum(bytes_in): 73
sum(bytes_out): 73
timestamp: 2018-03-24T12:04:26.190502Z
}
Show as raw text

However, when i click the Stream application nothing is shown in the dashboard, analytics overview or flow visualization. -> No results found. Looks like the data isn't parsed by Splunk Streaming app.

Re: Streaming app - netflow - data not parsed

Azeemering — Tue, 29 Sep 2020 18:40:27 GMT

I think you need to look at the SPL behind the dashboards and see if it matches the sourcetype that you have.
For example that specific dashboard you mention (Flow Visualization) has the following spl:

index=* source=stream:Splunk_IP | rex field=src_ip "(?.).(?.).(?.).(?.)" | where src1 NOT null | rex field=dest_ip "(?.).(?.).(?.).(?.)" | where dest1 NOT null | eval source_ip=round(src1+exact(src2*.001), 3) | eval destination_ip=round(dest1+exact(dest2*.001), 3) | eventstats sum(sum(bytes)) as bytes by source_ip, destination_ip | stats latest(source_ip), latest(destination_ip), sum(count) by bytes | rename latest(source_ip) as "Source IP", latest(destination_ip) as "Destination IP", sum(count) as "Flows", bytes as "Bytes", sourcetype as "Sourcetype"

So take off all spl from that and search only index=* source=stream:Splunk_IP. Do you get any data in verbose mode?

Re: Streaming app - netflow - data not parsed

heskez — Mon, 26 Mar 2018 11:44:29 GMT

When I run the search like that "index=* source=stream:Splunk_IP" I don't get any results..

Re: Streaming app - netflow - data not parsed

heskez — Tue, 29 Sep 2020 18:45:03 GMT

I do get data running this query btw:
index=* source="stream:*" splunk_server="host_ip"

Re: Streaming app - netflow - data not parsed

heskez — Sat, 31 Mar 2018 09:47:13 GMT

This query doesn't return data (This is extracted from the SPL):
index=* source="stream:Splunk_Tcp"

Re: Streaming app - netflow - data not parsed

heskez — Sat, 31 Mar 2018 09:50:12 GMT

This query doesn't return data either (Extracted from flow visualisation)
index=* source=stream:Splunk_IP

Re: Streaming app - netflow - data not parsed

heskez — Tue, 29 Sep 2020 18:45:06 GMT

So next I've created a new stream "fortistream" protocol netflow.
Then I've edited the search from the VF dashboard like this:

index=* source="stream:fortistream" | rex field=src_ip "(?.).(?.).(?.).(?.)" | where src1 NOT null | rex field=dest_ip "(?.).(?.).(?.).(?.)" | where dest1 NOT null | eval source_ip=round(src1+exact(src2*.001), 3) | eval destination_ip=round(dest1+exact(dest2*.001), 3) | eventstats sum(sum(bytes)) as bytes by source_ip, destination_ip | stats latest(source_ip), latest(destination_ip), sum(count) by bytes | rename latest(source_ip) as "Source IP", latest(destination_ip) as "Destination IP", sum(count) as "Flows", bytes as "Bytes", sourcetype as "Sourcetype"

Put it in verbose mode.

Now I do get data but ONLY the flow data. And here is the funny thing:
The source ip address is cut in half.. (192.168)
The destination ip address too (8.008)

Very weird..

Re: Streaming app - netflow - data not parsed

heskez — Tue, 29 Sep 2020 19:58:54 GMT

So next I've created a new stream "fortistream" protocol netflow.
Then I've edited the search from the VF dashboard like this:

index= source="stream:fortistream" | rex field=src_ip "(?.).(?.).(?.).(?.)" | where src1 NOT null | rex field=dest_ip "(?.).(?.).(?.).(?.)" | where dest1 NOT null | eval source_ip=round(src1+exact(src2.001), 3) | eval destination_ip=round(dest1+exact(dest2*.001), 3) | eventstats sum(sum(bytes)) as bytes by source_ip, destination_ip | stats latest(source_ip), latest(destination_ip), sum(count) by bytes | rename latest(source_ip) as "Source IP", latest(destination_ip) as "Destination IP", sum(count) as "Flows", bytes as "Bytes", sourcetype as "Sourcetype"

Put it in verbose mode.

Now I do get data but ONLY the flow data. And here is the funny thing:
The source ip address is cut in half.. (192.168)
The destination ip address too (8.008)

Very weird..

Re: Streaming app - netflow - data not parsed

heskez — Wed, 01 Aug 2018 20:08:58 GMT

This mystery is still not solved.. Yes, I get data now with this query index= source="stream:fortistream" in verbose mode:

But still ip addresses are only showing the first two octets..

Hmm, I suspect the regex: rex field=src_ip "(?<src1>.*)\.(?<src2>.*)\.(?<src3>.*)\.(?<src4>.*)"

basically the field ip is calculated as: (?:(?<!\d)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(?!\d))

So where is the syntax error...????

Anyone?