No port_scan data

[richgalloway]

We're running Splunk for Asset Discovery 6.0 under Splunk 6 on an Ubuntu system. The app has been running for a week, but we have no data in the asset_discovery index. There are several input scripts defined and enabled, including '$SPLUNK_HOME/etc/apps/asset_discovery/bin/nmap.sh -A -O 192.168.100.0/24'. If I run this command manually, I see data for all of the hosts in that subnet. However, a search of 'index=asset_discovery' returns no events. nmap is owned by root. I assume it is running as root also since all of Splunk does so.

I see nothing in splunkd.log other than "INFO ExecProcessor - New scheduled exec process:
/opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -A -O 192.168.100.0/24."

Where is my port_scan data going?

[AnilPujar]

03-10-2019 13:23:14.272 +0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\etc\apps\asset_discovery\bin\nmap.cmd" -A -O" The system cannot find the file C:\Program Files\Splunk\etc\apps\asset_discovery\bin\nmap.path.

[AnilPujar]

installed nmap, set env variables and cmd result also

C:\Windows\System32>nmap -v -A scanme.nmap.org
Starting Nmap 7.70 ( https://nmap.org ) at 2019-03-10 13:26 Arabian Standard Time
NSE: Loaded 148 scripts for scanning.
NSE: Script Pre-scanning.
Initiating NSE at 13:26
Completed NSE at 13:26, 0.00s elapsed
Initiating NSE at 13:26
Completed NSE at 13:26, 0.00s elapsed
Initiating Ping Scan at 13:26
Scanning scanme.nmap.org (45.33.32.156) [4 ports]
Completed Ping Scan at 13:26, 0.95s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 13:26
Completed Parallel DNS resolution of 1 host. at 13:26, 0.00s elapsed
Initiating SYN Stealth Scan at 13:26
Scanning scanme.nmap.org (45.33.32.156) [1000 ports]
Discovered open port 80/tcp on 45.33.32.156
Discovered open port 22/tcp on 45.33.32.156
Completed SYN Stealth Scan at 13:27, 14.48s elapsed (1000 total ports)
Initiating Service scan at 13:27
Scanning 2 services on scanme.nmap.org (45.33.32.156)
Completed Service scan at 13:27, 6.56s elapsed (2 services on 1 host)
Initiating OS detection (try #1) against scanme.nmap.org (45.33.32.156)
Retrying OS detection (try #2) against scanme.nmap.org (45.33.32.156)
Initiating Traceroute at 13:27
Completed Traceroute at 13:27, 6.04s elapsed
Initiating Parallel DNS resolution of 6 hosts. at 13:27
Completed Parallel DNS resolution of 6 hosts. at 13:27, 0.00s elapsed
NSE: Script scanning 45.33.32.156.
Initiating NSE at 13:27
Completed NSE at 13:27, 8.52s elapsed
Initiating NSE at 13:27
Completed NSE at 13:27, 0.00s elapsed
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.27s latency).
Not shown: 993 filtered ports
PORT STATE SERVICE VERSION
21/tcp closed ftp
22/tcp open ssh OpenSSH 6.6.1p1 Ubuntu 2ubuntu2.11 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey:
| 1024 ac:00:a0:1a:82:ff:cc:55:99:dc:67:2b:34:97:6b:75 (DSA)
| 2048 20:3d:2d:44:62:2a:b0:5a:9d:b5:b3:05:14:c2:a6:b2 (RSA)
| 256 96:02:bb:5e:57:54:1c:4e:45:2f:56:4c:4a:24:b2:57 (ECDSA)
|_ 256 33:fa:91:0f:e0:e1:7b:1f:6d:05:a2:b0:f1:54:41:56 (ED25519)
23/tcp closed telnet
80/tcp open http Apache httpd 2.4.7 ((Ubuntu))
|http-favicon: Unknown favicon MD5: 156515DA3C0F7DC6B2493BD5CE43F795
| http-methods:
| Supported Methods: POST OPTIONS GET HEAD
|_http-server-header: Apache/2.4.7 (Ubuntu)
|_http-title: Go ahead and ScanMe!
443/tcp closed https
5061/tcp closed sip-tls
8080/tcp closed http-proxy
Device type: general purpose
Running (JUST GUESSING): Linux 4.X|3.X|2.6.X (99%)
OS CPE: cpe:/o:linux:linux_kernel:4.9 cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:2.6.32
Aggressive OS guesses: Linux 4.9 (99%), Linux 3.10 - 3.16 (96%), Linux 3.10 (93%), Linux 2.6.32 (93%), Linux 3.10 - 3.12 (93%), Linux 4.4 (93%), Linux 2.6.39 (90%)
No exact OS matches for host (test conditions non-ideal).
Uptime guess: 14.541 days (since Sun Feb 24 00:28:30 2019)
Network Distance: 19 hops
TCP Sequence Prediction: Difficulty=261 (Good luck!)
IP ID Sequence Generation: All zeros
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

TRACEROUTE (using port 23/tcp)
HOP RTT ADDRESS
1 1.00 ms ******************
2 1.00 ms 172.30.7.1
3 1.00 ms 172.30.6.78
4 1.00 ms 172.30.3.45
5 1.00 ms 10.192.116.66
6 ... 18
19 260.00 ms scanme.nmap.org (45.33.32.156)

NSE: Script Post-scanning.
Initiating NSE at 13:27
Completed NSE at 13:27, 0.00s elapsed
Initiating NSE at 13:27
Completed NSE at 13:27, 0.00s elapsed
Read data files from: C:\Program Files (x86)\Nmap
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 59.31 seconds
Raw packets sent: 2124 (97.044KB) | Rcvd: 151 (11.090KB)

C:\Windows\System32>

[mario_traf]

Hi,
I also have a similar problem. I can see data within a splunk search "index=asset_discovery sourcetype=port_scan", but the eventtype port_scan (index=asset_discovery sourcetype=port_scan "Host:" "Ports:" "Ignored State:" ) doesn't produce anything as my script isn't generating any "Ignored State:"

I am running the following script:
/opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -A -O -t 172.20.32.0/24 --max-retries 1 --osscan-guess --system-dns
and I have added "unset LD_LIBRARY_PATH" to the nmap.sh script as well as ensuring that nmap is chmod'ed so the splunk user can use it.
Have a missed something and argument when calling the script?
Mario

[mario_traf]

I wasn't sure if the ignored state was needed or not.
anyway, I have done as suggested.
turns out that the version of nmap I am using doesn't generate the "Ignored State:" text anymore

[mw]

You can edit the eventtype to remove that portion.

[richgalloway · Accepted solution]

Redirecting the nmap.sh output to a file showed nmap was failing because of a missing OpenSSL library.

nmap: /opt/splunk/lib/libcrypto.so.1.0.0: version `OPENSSL_1.0.0' not found (required by nmap)
nmap: /opt/splunk/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by nmap)

Adding unset LD_LIBRARY_PATH to nmap.sh fixed the problem.

Thanks to Splunk tech support for their help with this.

[tednaleid]

Thanks! I hit this when running curl in my shell command to post an alert to a slack channel. Doing ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4 in my script and piping that to a file showed that when splunk executed the script it was adding a prefix to the location of libssl as you saw. Unsetting the LD_LIBRARY_PATH it fixed it.

[mw]

Rich, I'm not sure what's up here. Could you shoot me an email when you have a chance and we can try doing some debugging? I'm curious about what's happening here as well, particularly since I really haven't changed the scanning stuff in the latest version. mwilson at splunk dot com.

[mikaelbje]

I figured it out. I had to chmod +s the nmap binary. I also had to chsnge ifconfig in nmap.sh to /sbin/ifconfig. This was in Ubuntu

Restarting Splunk shouldn't be necessary AFAIK as you are only modifying Linux permissions for a binary that's called. However I have Splunk running as the "splunk" user, so if you're running as root it should absolutely work. My permissions for reference:

-rwsr-s--- 1 root adm 756464 Dec 14  2011 /usr/bin/nmap

groups splunk
splunk : splunk adm

Running this from command line works fine, also as a scripted input in Splunk:

/opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -A -O -t 172.24.201.0/24

Since this is Ubuntu sh is a symbolic link to dash, not bash, but it should work in bash too.

asset_discovery/local/inputs.conf

[script://./bin/nmap.sh -A -O -t 172.24.201.0/24]

disabled = false

index = asset_discovery

interval = 3600

sourcetype = port_scan

source = nmap

[jrodman]

The reason setting the executable setUID works (which I don't recommend!) is that Linux sanitizes the environment when jumping through a setuid gate, specifically dropping LD_LIBRARY_PATH and other linker controls, so that a user cannot execute arbitrary code as root trivially.

Thus this indirectly requests the action the app should have taken in the first place, to strip the library path when running a system binary.

[JoeIII]

For NMAP to do OS detection it requires root privileges. This is why setUID works if Splunk is not running as root already.

I recommend configuring sudoers to allow splunk to run nmap without a password, then modifying the script to "sudo nmap", in addition to adding unset LD_LIBRARY_PATH in the event of that issue.

my /etc/sudoers.d/splunknmap (use visudo so you don't botch the file rights (my splunk instance runs as "splunk":

splunk ALL = (root) NOPASSWD: /usr/bin/nmap

Still not an ideal solution, but the best available, in my opinion

[dmartinez_splun]

I'm running this on my Mac for testing and I had to:

[as root]
sh-3.2# type nmap
nmap is /usr/local/bin/nmap
sh-3.2# cd /usr/local/bin/
sh-3.2# chmod 6711 nmap
sh-3.2# ls -lrt
-rws--s--x 1 root wheel 6059116 23 Sep 01:17 nmap

[as daniel]
192-168-1-5:bin Daniel$ ./nmap.sh -A -O

Nmap 6.49BETA5 scan initiated Fri Oct 16 12:17:08 2015 as: nmap -oG - -A -O 192.168.1.5/24

[it works]... data coming into Splunk

[richgalloway]

[script://./bin/nmap.sh]
disabled = 0

[script://./bin/nmap.sh -A -O]
disabled = 0

[script:///opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -p14147 -t 172.16.42.
64 172.16.42.220 172.16.42.230]
disabled = false
index = asset_discovery
interval = 60
source = nmap
sourcetype = port_scan

[mikaelbje]

Can you paste your inputs.conf?

[richgalloway]

I restarted Splunk and still am getting no port_scan data. In fact, my asset_discovery index contains nothing at all.

[mikaelbje]

I updated my answer. Not sure if it's any help. You might try restarting Splunk just in case.

[richgalloway]

I made the same changes and still no data. Did you have to restart Splunk?

[mikaelbje]

I'm having the same problem. Running nmap through nmap.sh for a port scan works in the bash shell, even as the splunk user, but nothing is added for port_scan to Splunk. Tried running Splunk as root and splunk

[richgalloway]

-rwxr-xr-x 1 root root 1972032 Jan 4 2013 /usr/bin/nmap*

Splunk is running as root so wouldn't nmap also run as root?

[lukejadamec]

Is the owner:group of the nmap binary root:root

And the permissions set to 4755?
It could be a suid bit problem that you're not seeing when you run it yourself as root.