All Apps and Add-ons
Highlighted

Why does Splunk sometimes not recognize Scapy generated packets in Web Page Monitor?

Explorer

Before the question, a bit of background.
I have a setup in which I have two machines.
The first collects data from various devices and sends it directly to the second over UDP, targeting port 5005.
The second is running Splunk and has been configured to listen on port 5005 for UDP messages, and record them on a "sandbox" index.
These machines are isolated from the internet and are connected by Ethernet cables to a hub switch right next to them.
For the sake of brevity, let's call them M1 and S, short for Machine 1 and Splunk Machine.
S has been assigned IP 192.168.0.5 while M1 has 192.168.0.6.

There are three ways I can transmit packets from M1 to S.
The first is by running from the terminal on S1:

echo -n "{'Message':'hello'}" > /dev/udp/192.168.0.5/5005

This message is successfully sent from M1 to S and shows up in the sandbox index.

The second is by running some scripts that emulate our desired behavior and form packets using Scapy, again on M1 targeting S.
This is Scapy, not Scipy.
This process also successfully completes the full loop and shows up in the sandbox index.

The final method, and the one this question centers on, is to open scapy on M1 and generate packets and send them that way. Assuming we want to emulate sending packets from a docker container on M1 with an an IP of 10.10.12.9, the command used to generate these packets is as follows, with some slight editing, namely substituting text for the actual mac addresses and placing each field on its own line:

sendp(Ether(dst="<mac address of container here>",src="<mac address of S here>")
            /IP(src="10.10.12.9",dst=192.168.0.5")
            /UDP(dport=5005,sport=33017)
            /Raw(load="{'Message':'Hello.'}"), iface="veth201")

If I execute this command in Scapy, I'm told that it sends the packet.
If I run a tcpdump on S, I can see that the packet generated by the command does in fact travel from M1 to S and is received.
However, this packet is entirely ignored by Splunk.
When I say "entirely ignored" I mean that, if I open the Splunk Web Search in a browser on S, and start a real time search with a 5 minute window for all events on the "sandbox" or "main" indexes, sending the scapy packet as described above does not cause an event, while the other two methods trigger events on the "sandbox" index as expected.
How do I make Splunk recognize this packet?

EDIT 1: I've found that if I run

send(IP(src="192.168.0.6",dst=192.168.0.5")
          /UDP(dport=5005,sport=33017)
          /Raw(load="{'Message':'Hello.'}"), iface="veth201")

Then the packet will be received successfully in TCP dump and is recognized by Splunk.
If I then change the command to

send(IP(src="10.10.12.9",dst=192.168.0.5")
          /UDP(dport=5005,sport=33017)
          /Raw(load="{'Message':'Hello.'}"), iface="veth201")

Then it will fail once again to be recognized by Splunk, but the packet is successfully delivered according to tcpdump.

0 Karma
Highlighted

Re: Why does Splunk sometimes not recognize Scapy generated packets in Web Page Monitor?

Explorer

Oh, and I forgot to mention. Changing the mac and src IP in the scapy packet to those of M1 itself instead of the container that is running on M1 makes no difference; Splunk still does not recognize the packet.

0 Karma
Highlighted

Re: Why does Splunk sometimes not recognize Scapy generated packets in Web Page Monitor?

Can you share your inputs.conf configuration for that port? If you've defined a <remote server> as per the docs, then Splunk will only pick up data from a particular remote host and ignore data from other hosts.
https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Monitornetworkports#Configure_a_UDP_input

0 Karma
Highlighted

Re: Why does Splunk sometimes not recognize Scapy generated packets in Web Page Monitor?

Explorer

I did all of my configuration through Splunk Web, and did not restrict anything when I defined the input. I went to /opt/splujnk/etc/system/local to look at inputs.conf anyway, and it is only two lines long. The file is copied below, verbatim.

[default]
host = docker-swarm-mgr

docker-swarm-mgr is the name of the host running Splunk Enterprise.

0 Karma
Highlighted

Re: Why does Splunk sometimes not recognize Scapy generated packets in Web Page Monitor?

Explorer

I figured out the answer.
If you send a packet with a given source address, Splunk must be able to actually reach that address. If it is unable to reach the address, even if a packet is received, Splunk will not consider it a valid event.

I discovered this by trying to ping 10.10.12.9 from S and found it was unreachable. I added a route on S then checked by pinging that the 10.10.12 address was now reachable. After doing so, I re-sent the Scapy-created packet, and it showed up in the sandbox index as a valid event.

View solution in original post