topic Re: Splunk REST API: Issue wtih flattening JSON-formatted results in Getting Data In

Splunk REST API: Issue wtih flattening JSON-formatted results

dominiquevocat — Fri, 04 Aug 2017 15:38:20 GMT

So I call the Splunk REST API and collect results in JSON format and that is kind of okay.
Then I would like to pass it to splunk.Intersplunk.outputResults()
Intersplunk fails to flatten this kind of complex object, so a workaround would be to just get the subset data["entry"] and one leven nesting gets flattened nicely.
However, there are nested elements in that as well like ACL, links, etc.
The nested elements show up as python object in string representation, which you can not then (as a lazy way out) use spath on...
Has anyone found a good way to do this transformation?

example:

{
  "links": {

  },
  "generator": {
    "build": "debde650d26e",
    "version": "6.4.1"
  },
  "updated": "2017-08-04T15:19:21+02:00",
  "origin": "https://xxx:8089/servicesNS/nobody/-/",
  "entry": [
    {
      "links": {
        "alternate": "/servicesNS/nobody/SplunkUniversalForwarder/"
      },
      "name": "SplunkUniversalForwarder",
      "updated": "2017-08-04T15:19:21+02:00",
      "id": "https://xxx:8089/servicesNS/nobody/SplunkUniversalForwarder/"
    },
    {
      "links": {
        "alternate": "/servicesNS/nobody/alert_logevent/"
      },
      "name": "alert_logevent",
      "updated": "2017-08-04T15:19:21+02:00",
      "id": "https://xxx:8089/servicesNS/nobody/alert_logevent/"
    },
    {
      "links": {
        "alternate": "/servicesNS/nobody/introspection_generator_addon/"
      },
      "name": "introspection_generator_addon",
      "updated": "2017-08-04T15:19:21+02:00",
      "id": "https://xxx:8089/servicesNS/nobody/introspection_generator_addon/"
    }
  ]
}

Re: Splunk REST API: Issue wtih flattening JSON-formatted results

jkat54 — Fri, 04 Aug 2017 18:21:40 GMT

Why are you pulling data from the API in a SPL command?

The results in the pipeline are readily available to you.

Are you taking the results of one search and running another based on them?

Re: Splunk REST API: Issue wtih flattening JSON-formatted results

jkat54 — Fri, 04 Aug 2017 18:30:17 GMT

import splunk.Intersplunk

# get the keywords suplied to the curl command
keywords, options = splunk.Intersplunk.getKeywordsAndOptions()

# get the previous search results
results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

if len(results) > 0:
  for result in results:
    #result["fieldName"] is the field value in one row of your results
    result["fieldName"]="NEW VALUE"

else:
  #no data in pipe
  row={}
  results=[]
  row["foo"] = "bar"
  row["foo2"] = "bar2"
  results.append(row)

splunk.Intersplunk.outputResults(results)

Re: Splunk REST API: Issue wtih flattening JSON-formatted results

dominiquevocat — Mon, 07 Aug 2017 09:02:38 GMT

for the usecase see https://splunkbase.splunk.com/app/2775/ 🙂

Re: Splunk REST API: Issue wtih flattening JSON-formatted results

jkat54 — Mon, 07 Aug 2017 11:50:01 GMT

In that case just use the Python json library, and iterate through dumping strings into results (example of creating results provided in my comments above).

https://docs.python.org/2/library/json.html

Re: Splunk REST API: Issue wtih flattening JSON-formatted results

dominiquevocat — Tue, 08 Aug 2017 11:06:00 GMT

Seems to be mostly good enough:

        if pagehandle.status_code==200:
            data = json.loads(pagehandle.text)
            if "entry" in data:
                results = data["entry"]
                for row in results:
                    #decorate each list item with fields from the returned data
                    if "origin" in data: row["origin"]=data["origin"]
                    if "generator" in data: 
                        row["version"]=data["generator"]["version"]
                        row["build"]=data["generator"]["build"]
                    if "paging" in data: 
                        row["paging:total"]=data["paging"]["total"]
                        row["paging:offset"]=data["paging"]["offset"]
                        row["paging:perPage"]=data["paging"]["perPage"]

                    #flatten notable data
                    if "acl" in row:
                        for acl in row["acl"]:
                            aclitem = str(acl)
                            row[aclitem] = row["acl"][acl]
                        row.pop("acl", None)
                        #alternatively return it as json                        
                        #row["acl"] = json.dumps(row["acl"])
                        #row["acl"].replace("\"", "'")
                    if "perms" in row:
                        if row["perms"] is not None: # there might be "None" values
                            for perm in row["perms"]:
                                permitem = str(perm)
                                row[permitem] = row["perms"][perm]
                            row.pop("perms", None)
                            #alternatively return it as json                        
                            #row["acl"] = json.dumps(row["acl"])
                            #row["acl"].replace("\"", "'")
                    if "content" in row:
                        for content in row["content"]:
                            contentitem = str(content)
                            row[contentitem] = row["content"][content]
                        row.pop("content", None)
                        #alternatively return it as json
                        #row["content"] = json.dumps(row["content"])
                        #row["content"].replace("\"", "'")
                    if "links" in row:
                        for link in row["links"]:
                            linkitem = str(link)
                            row[linkitem] = row["links"][link]
                        row.pop("links", None)
                        #alternatively return it as json
                        #row["links"] = json.dumps(row["links"])
                        #row["links"].replace("\"", "'")
                splunk.Intersplunk.outputResults(results)
            else:
                splunk.Intersplunk.generateErrorResults("nothing to show")