I'm trying to extract all the CVEs and associated their CVSS scores from Shodan's API (JSON response). The response is typically in the format where the number after data depends on the number of services detected, example data:
Current search:
| curl method=get uri=https://api.shodan.io/shodan/host/"IP"?key=APIKEY
| spath input=curl_message path="data{0}.vulns" output=test_array
| mvexpand test_array
| spath input=test_array
| table CVE*.cvss
When using curl from WebTools, spath doesn't appear to be extracting all the fields (e.g. only 4 of the 15 CVEs are displayed in the table), likely because of the 5000 character limit for spath. Is there another method that would be able to keep data like the CVE, cvss and summary linked while splitting the data? Delim via comma seems like it wouldn't be possible since the summaries also include commas.
I believe the app supports urifield as well as uri=, so just make
| eval uri="https://api.shodan.io/shodan/host/".clientIP."?key=APIKEY"
| curl urifield=uri...
Have you tried any of the eval json functions
https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/JSONFunctions
json_extract or json_array_to_mv?
Thanks! Managed to get it parsing JSON correctly using those two functions. However, do you know if it's possible to have a field present in the URI requirement for curl?
The parent search has a field named clientIP which stores IP addresses depending on the client that made a request. Doesn't seem like it's possible to submit it as a concatenated string since it always needs to begin with https://. e.g. neither of the below work:
index=api_test uri_path=exampleRequest
| curl method=get uri=https://api.shodan.io/shodan/host/clientIP?key=APIKEY
OR
index=api_test uri_path=exampleRequest
| curl method=get uri="https://api.shodan.io/shodan/host/" + clientIP +"?key=APIKEY"
I believe the app supports urifield as well as uri=, so just make
| eval uri="https://api.shodan.io/shodan/host/".clientIP."?key=APIKEY"
| curl urifield=uri...
Thank you! Search is returning some results, but hangs indefinitely.