Splunk REST API: Issue wtih flattening JSON-formatted results

[dominiquevocat]

So I call the Splunk REST API and collect results in JSON format and that is kind of okay.
Then I would like to pass it to splunk.Intersplunk.outputResults()
Intersplunk fails to flatten this kind of complex object, so a workaround would be to just get the subset data["entry"] and one leven nesting gets flattened nicely.
However, there are nested elements in that as well like ACL, links, etc.
The nested elements show up as python object in string representation, which you can not then (as a lazy way out) use spath on...
Has anyone found a good way to do this transformation?

example:

{
  "links": {

  },
  "generator": {
    "build": "debde650d26e",
    "version": "6.4.1"
  },
  "updated": "2017-08-04T15:19:21+02:00",
  "origin": "https://xxx:8089/servicesNS/nobody/-/",
  "entry": [
    {
      "links": {
        "alternate": "/servicesNS/nobody/SplunkUniversalForwarder/"
      },
      "name": "SplunkUniversalForwarder",
      "updated": "2017-08-04T15:19:21+02:00",
      "id": "https://xxx:8089/servicesNS/nobody/SplunkUniversalForwarder/"
    },
    {
      "links": {
        "alternate": "/servicesNS/nobody/alert_logevent/"
      },
      "name": "alert_logevent",
      "updated": "2017-08-04T15:19:21+02:00",
      "id": "https://xxx:8089/servicesNS/nobody/alert_logevent/"
    },
    {
      "links": {
        "alternate": "/servicesNS/nobody/introspection_generator_addon/"
      },
      "name": "introspection_generator_addon",
      "updated": "2017-08-04T15:19:21+02:00",
      "id": "https://xxx:8089/servicesNS/nobody/introspection_generator_addon/"
    }
  ]
}

[dominiquevocat · Accepted solution]

Seems to be mostly good enough:

        if pagehandle.status_code==200:
            data = json.loads(pagehandle.text)
            if "entry" in data:
                results = data["entry"]
                for row in results:
                    #decorate each list item with fields from the returned data
                    if "origin" in data: row["origin"]=data["origin"]
                    if "generator" in data: 
                        row["version"]=data["generator"]["version"]
                        row["build"]=data["generator"]["build"]
                    if "paging" in data: 
                        row["paging:total"]=data["paging"]["total"]
                        row["paging:offset"]=data["paging"]["offset"]
                        row["paging:perPage"]=data["paging"]["perPage"]

                    #flatten notable data
                    if "acl" in row:
                        for acl in row["acl"]:
                            aclitem = str(acl)
                            row[aclitem] = row["acl"][acl]
                        row.pop("acl", None)
                        #alternatively return it as json                        
                        #row["acl"] = json.dumps(row["acl"])
                        #row["acl"].replace("\"", "'")
                    if "perms" in row:
                        if row["perms"] is not None: # there might be "None" values
                            for perm in row["perms"]:
                                permitem = str(perm)
                                row[permitem] = row["perms"][perm]
                            row.pop("perms", None)
                            #alternatively return it as json                        
                            #row["acl"] = json.dumps(row["acl"])
                            #row["acl"].replace("\"", "'")
                    if "content" in row:
                        for content in row["content"]:
                            contentitem = str(content)
                            row[contentitem] = row["content"][content]
                        row.pop("content", None)
                        #alternatively return it as json
                        #row["content"] = json.dumps(row["content"])
                        #row["content"].replace("\"", "'")
                    if "links" in row:
                        for link in row["links"]:
                            linkitem = str(link)
                            row[linkitem] = row["links"][link]
                        row.pop("links", None)
                        #alternatively return it as json
                        #row["links"] = json.dumps(row["links"])
                        #row["links"].replace("\"", "'")
                splunk.Intersplunk.outputResults(results)
            else:
                splunk.Intersplunk.generateErrorResults("nothing to show")

[jkat54]

In that case just use the Python json library, and iterate through dumping strings into results (example of creating results provided in my comments above).

https://docs.python.org/2/library/json.html

[jkat54]

Why are you pulling data from the API in a SPL command?

The results in the pipeline are readily available to you.

Are you taking the results of one search and running another based on them?

[dominiquevocat]

for the usecase see https://splunkbase.splunk.com/app/2775/ 🙂

[jkat54]

import splunk.Intersplunk

# get the keywords suplied to the curl command
keywords, options = splunk.Intersplunk.getKeywordsAndOptions()

# get the previous search results
results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

if len(results) > 0:
  for result in results:
    #result["fieldName"] is the field value in one row of your results
    result["fieldName"]="NEW VALUE"

else:
  #no data in pipe
  row={}
  results=[]
  row["foo"] = "bar"
  row["foo2"] = "bar2"
  results.append(row)

splunk.Intersplunk.outputResults(results)