topic Re: How to join multiple sources to build a network path in Dashboards & Visualizations

How to join multiple sources to build a network path

PaoloR84 — Wed, 30 Sep 2020 02:31:38 GMT

Hi all,
I have the following dataset:

Source A: "DEVICE INFO"
Source B: "SOURCE" (maps to SourceA DEVICE),"SOURCE_PORTS",DESTINATION, DESTINATION_PORTS
Source C: "SOURCE" (which is the DESTINATION of Source B) etc..

Basically I'm trying to dynamically build a network path between multiple devices (and from multiple sources), the ultimate goal will be a network topology (probably with sankey but doesn't matter right now)

As example:

SourceA

    | makeresults | eval sourcetype = "A" | eval Device = "Device_XYZ" | eval Model = "Vendor"

SourceB

    | append [| makeresults | eval sourcetype = "B" | eval Source = "Device_XYZ" | eval SourcePorts = "123456" | eval Destination = "Device_QWE" | eval DestinationPorts = "AAABBBB"] 
    | append [| makeresults | eval sourcetype = "B" | eval Source = "Device_XYZ" | eval SourcePorts = "789000" | eval Destination = "Device_QWE" | eval DestinationPorts = "CCCDDDD"]

SourceC

| append [| makeresults | eval sourcetype = "C" | eval Source = "Device_QWE" | eval SourcePorts = "AAABBBB" | eval Destination = "Device_MNB" | eval DestinationPorts = "QQQWWW"] 
| append [| makeresults | eval sourcetype = "C" | eval Source = "Device_QWE" | eval SourcePorts = "CCCDDDD" | eval Destination = "Device_MNB" | eval DestinationPorts = "QQQWWW"]

Any idea on how to approach is welcome, ty guys for your time

PaoloR

Re: How to join multiple sources to build a network path

urana — Wed, 30 Sep 2020 02:31:44 GMT

You could try multisearch, something like this

|multisearch

[ search Source A
| search search query
| fields all fields you want from that search]

[ search Source B
| search search query
| fields all fields you want from that search]

[ search Source C
| search search query
| fields all fields you want from that search]

| eval Source A=if(like(field A),"field B",field C)

For example I use it for Potential Malicious User agents:

| multisearch

[ search (index=proxy) "script"
| search http_user_agent="script"
| fields _time, http_user_agent, src_ip, url]

[ search (index=proxy OR sourcetype=f5*) "Iceweasel"
| search http_user_agent="Iceweasel"
| fields _time, http_user_agent, src_ip, url]

[ search (index=proxy OR sourcetype=f5*) "Meterpreter/Windows"
| search http_user_agent="*Meterpreter/Windows"
| fields _time, http_user_agent, src_ip, url]

[ search (index=proxy OR sourcetype=f5*) "Mozilla/5.00 (Nikto/"
| search http_user_agent="Mozilla/5.00 (Nikto/*"
| fields _time, http_user_agent, src_ip, url]

[ search (index=proxy OR sourcetype=f5*) "dirb"
| search http_user_agent="dirb"
| fields _time, http_user_agent, src_ip, url]

[ search (index=proxy OR sourcetype=f5*) "WinHttp.WinHttpRequest"
| search http_user_agent="Win32; WinHttp.WinHttpRequest"
| fields _time, http_user_agent, src_ip, url]

| eval suspect_issue=if(like(http_user_agent,"%script%"),"Cross Site Scripting",suspect_issue)
| eval suspect_issue=if(like(http_user_agent,"%Iceweasel%"),"Kali",suspect_issue)
| eval suspect_issue=if(like(http_user_agent,"%Meterpreter%"),"Meterpreter",suspect_issue)
| eval suspect_issue=if(like(http_user_agent,"%(Nikto/%"),"Nikto Scanning",suspect_issue)
| eval suspect_issue=if(like(http_user_agent,"%dirb%"),"DirbScanning",suspect_issue)
| eval suspect_issue=if(like(http_user_agent,"%WinHttp.WinHttpRequest%"),"WScript",suspect_issue)
| stats latest(_time) AS Latest, values(url) as url by http_user_agent, suspect_issue, src_ip

Re: How to join multiple sources to build a network path

woodcock — Sun, 13 Oct 2019 06:21:50 GMT

You really need to look at Business Flow:
https://www.splunk.com/en_us/software/business-analytics-and-process-mining.html

You might also check out some mod-viz on Splunkbase:
Force Directed App: https://splunkbase.splunk.com/app/3767/
Graph Viz: https://splunkbase.splunk.com/app/4346/
AfterGlow: https://splunkbase.splunk.com/app/277/

Re: How to join multiple sources to build a network path

PaoloR84 — Sun, 13 Oct 2019 09:34:07 GMT

@urana
Thank you but that approach doesn't work (or I wasn't able to make it works); I've ended doing a map command from A & B sources and the a join with the C source. I try to avoid join as much as possible but the devices aren't billions and the performances are more than acceptable.

@woodcock
I was playing with Business flow a few weeks ago in the Splunk Oxygen, may worths another look, ty.
The topology apps are all great and I already use them; the issue with this use case is the tons of variables to be handled

Basically I have (just as example):
- Switch 1 connected to Switch 2 with 4 ports; each link has is own metrics/info
- Switch 2 connected to Switch 3 with 8 ports; again, each link with is own info

I tried with multivalues and succesfully build a single line, multivalue topology across all devices/link. Right now I'm stuck on splitting multivalues fields because of they have uneven elements; the "standard" mvjoin/split/rex works well when you have same number of events in each multivalue but that's not my case 😞

Anyway, ty all for your time
PaoloR