All Apps and Add-ons

Extracting nested JSON from TA-WebTools curl response

bcrypt
Explorer

I'm trying to extract all the CVEs and associated their CVSS scores from Shodan's API (JSON response). The response is typically in the format where the number after data depends on the number of services detected, example data:

  • data :[
    • 0 22/tcp/OpenSSH :{  },
    • 1 80/tcp/Apache httpd :{
      • vulns :{
        • "CVE-2013-6501" :{
          • cvss :4.6,
          • references :[  ],
          • summary :"The default soap.wsdl_cache_dir setting in (1) php.ini-production and (2) php.ini-development in PHP through 5.6.7 specifies the /tmp directory, which makes it easier for local users to conduct WSDL injection attacks by creating a file under /tmp with a predictable filename that is used by the get_sdl function in ext/soap/php_sdl.c.",

Current search:

| curl method=get uri=https://api.shodan.io/shodan/host/"IP"?key=APIKEY
| spath input=curl_message path="data{0}.vulns" output=test_array
| mvexpand test_array
| spath input=test_array
| table CVE*.cvss

When using curl from WebTools, spath doesn't appear to be extracting all the fields (e.g. only 4 of the 15 CVEs are displayed in the table), likely because of the 5000 character limit for spath. Is there another method that would be able to keep data like the CVE, cvss and summary linked while splitting the data? Delim via comma seems like it wouldn't be possible since the summaries also include commas.

Labels (1)
Tags (3)
0 Karma
1 Solution

bowesmana
SplunkTrust
SplunkTrust

I believe the app supports urifield as well as uri=, so just make 

| eval uri="https://api.shodan.io/shodan/host/".clientIP."?key=APIKEY"
| curl urifield=uri...

View solution in original post

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Have you tried any of the eval json functions

https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/JSONFunctions

json_extract or json_array_to_mv?

0 Karma

bcrypt
Explorer

Thanks! Managed to get it parsing JSON correctly using those two functions. However, do you know if it's possible to have a field present in the URI requirement for curl?

The parent search has a field named clientIP which stores IP addresses depending on the client that made a request. Doesn't seem like it's possible to submit it as a concatenated string since it always needs to begin with https://. e.g. neither of the below work:

index=api_test uri_path=exampleRequest
| curl method=get uri=https://api.shodan.io/shodan/host/clientIP?key=APIKEY

OR

index=api_test uri_path=exampleRequest
| curl method=get uri="https://api.shodan.io/shodan/host/" + clientIP +"?key=APIKEY"

0 Karma

bowesmana
SplunkTrust
SplunkTrust

I believe the app supports urifield as well as uri=, so just make 

| eval uri="https://api.shodan.io/shodan/host/".clientIP."?key=APIKEY"
| curl urifield=uri...
0 Karma

bcrypt
Explorer

Thank you! Search is returning some results, but hangs indefinitely. 

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...