Network Toolkit - Lookup whois - How to use

[Niffchen]

Hello,

I am transferring all my network and some firewall data to splunk.
I try to analyse part of my firewall traffic of an IoT network. Therefore I am trying to transfer the source IPs which are communicating with my IoT devices to user friendly data.

This is why I tried to use the „Network Toolkit“ with its lookup called „whois“. But I don‘t get it working.
Combining my data with the source IPs - called src_ip - with „whois“ is not producing any data. I only get empty values.

search request like:

... | lookup whois host as src_ip OUTPUT ...

Is this usage correct? I cannot produce any different output than empty output.
Do you have any suggestions?

Regards,

Jens

[ownion]

Same problem for me, I need to populate some alert with the info generated by the whois command, but considering that the command " | whois xxx.xxx.xxx.xxx" must be inserted as the first command, I opted for the lookup whois, but when using:

... | lookup whois host as <my_field_containing_ip_addr> ...

 it show only blank columns, this makes me thinking that the "whois" lookup is empty, so with

| inputlookup whois

the result is empty...

Hoping someone will help with this.

[ericnewman]

We were having the same problem and discovered that we was getting the below errors in the search.log (Job-->Inpect Job->Search job properties - search.log) even though there was no indication on an issue.  We are running Splunk Enterprise version 8.1.2 which defaults to python3.  We were able to get the lookups working by setting them to run as python2. 

We added a custom /opt/splunk/etc/apps/network_tools/local/transforms.conf.

[whois]
python.version = python2

[nslookup]
python.version = python2

[traceroute]
python.version = python2

[ping]
python.version = python2

[portscan]
python.version = python2

Example Errors:

05-25-2021 15:14:39.784 INFO  PreviewExecutor - Preview Enforcing initialization done
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':  Exception in thread ping_lookup:
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':  Traceback (most recent call last):
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/lib/python3.7/threading.py", line 926, in _bootstrap_inner
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      self.run()
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/lib/python3.7/threading.py", line 870, in run
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      self._target(*self._args, **self._kwargs)
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/etc/apps/network_tools/bin/network_tools_app/custom_lookup.py", line 253, in do_lookup
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      self.execute_lookup(result, w, fieldnames)
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/etc/apps/network_tools/bin/network_tools_app/custom_lookup.py", line 210, in execute_lookup
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      output = self.do_lookup(**keyword_arguments)
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/etc/apps/network_tools/bin/whois_lookup.py", line 55, in do_lookup
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      index = get_default_index()
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/etc/apps/network_tools/bin/network_tools_app/__init__.py", line 133, in get_default_index
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      app_config = get_app_config(session_key)
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':    File "/opt/splunk/etc/apps/network_tools/bin/network_tools_app/__init__.py", line 106, in get_app_config
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':      conf = ConfigParser.SafeConfigParser()
05-25-2021 15:14:40.222 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/network_tools/bin/whois_lookup.py host':  AttributeError: type object 'ConfigParser' has no attribute 'SafeConfigParser'
05-25-2021 15:14:40.270 INFO  DispatchExecutor - END OPEN: Processor=noop

[ownion]

Thank you @ericnewman, switching from Python3 to Python2 solved our problem, now we can see and use the lookup command correctly.

[ericnewman]

After upgrading to Splunk 9.x the fix of reverting back to python 2 was no longer applicable so we dug into the code and made a few modifications to get the whois_lookup working with Python version 3.7.  A big disclaimer on this as the code isn't thoroughly tested and my Python development skills are limited.

network_tools/bin/network_tools_app/__init__.py

# import configparser instead of ConfigParser for Python 3 compatibility
#try:
#    import ConfigParser
#except ModuleNotFoundError:
#    from configparser import ConfigParser
import configparser
...
# use configparser.ConfigParser() instead of ConfigParser.SafeConfigParser() for Python 3 Compatibility
#        conf = ConfigParser.SafeConfigParser()
        conf = configparser.ConfigParser() 
...

network_toolkit/networktools_app/custom_lookup.py

...
from logging import handlers
import threading

# Added for Python 3 compatibility
from builtins import str

from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
...
# Use str instead of basestring for Python 3 compatibility
#                if isinstance(value, (list, tuple)) and not isinstance(value, basestring):
                if isinstance(value, (list, tuple)) and not isinstance(value, str):
... 

We also made some modification to network_toolkit/bin/whois_lookup.py to parse more fields for IP address lookups and populate the raw field.