Extracting nested JSON from TA-WebTools curl response

[bcrypt]

I'm trying to extract all the CVEs and associated their CVSS scores from Shodan's API (JSON response). The response is typically in the format where the number after data depends on the number of services detected, example data:

• data :[
  • 0 22/tcp/OpenSSH :{ … },
  • 1 80/tcp/Apache httpd :{
    • vulns :{
      • "CVE-2013-6501" :{
        • cvss :4.6,
        • references :[ … ],
        • summary :"The default soap.wsdl_cache_dir setting in (1) php.ini-production and (2) php.ini-development in PHP through 5.6.7 specifies the /tmp directory, which makes it easier for local users to conduct WSDL injection attacks by creating a file under /tmp with a predictable filename that is used by the get_sdl function in ext/soap/php_sdl.c.",

Current search:

| curl method=get uri=https://api.shodan.io/shodan/host/"IP"?key=APIKEY
| spath input=curl_message path="data{0}.vulns" output=test_array
| mvexpand test_array
| spath input=test_array
| table CVE*.cvss

When using curl from WebTools, spath doesn't appear to be extracting all the fields (e.g. only 4 of the 15 CVEs are displayed in the table), likely because of the 5000 character limit for spath. Is there another method that would be able to keep data like the CVE, cvss and summary linked while splitting the data? Delim via comma seems like it wouldn't be possible since the summaries also include commas.

[bowesmana]

Have you tried any of the eval json functions

https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/JSONFunctions

json_extract or json_array_to_mv?

[bcrypt]

Thanks! Managed to get it parsing JSON correctly using those two functions. However, do you know if it's possible to have a field present in the URI requirement for curl?

The parent search has a field named clientIP which stores IP addresses depending on the client that made a request. Doesn't seem like it's possible to submit it as a concatenated string since it always needs to begin with https://. e.g. neither of the below work:

index=api_test uri_path=exampleRequest
| curl method=get uri=https://api.shodan.io/shodan/host/clientIP?key=APIKEY

OR

index=api_test uri_path=exampleRequest
| curl method=get uri="https://api.shodan.io/shodan/host/" + clientIP +"?key=APIKEY"

[bowesmana · Accepted solution]

I believe the app supports urifield as well as uri=, so just make 

| eval uri="https://api.shodan.io/shodan/host/".clientIP."?key=APIKEY"
| curl urifield=uri...

[bcrypt]

Thank you! Search is returning some results, but hangs indefinitely.