How to index a SOAP response using REST API Modular Input?

[guimilare]

Hello Splunkers.

I have a WebService that I need to get data from.
I have to do the following steps:

1) Send a SOAP request to login to the WebService and get a SID (Session ID);
2) Use this SID to do a select on the WebService;
3) Index the SOAPresponse in Splunk;
3) Logout from the WebService.

How can I achieve this?
My first thought was using REST API.

I'm a bit lost on how to implement this.
Is REST API the best way? Or is using a script better?

Thanks in advance!
Best regards,
GMA

[mrgibbon]

I have been looking for a way of doing this with peoplesoft, please let me know if you work out how to do it!
Thanks

[jkat54]

What's wrong with my scripted input response? We integrate Splunk with APIs all the time. Write a script.

[mrgibbon]

I'd love to have the skills and time to sit and write a script for this, and I need to. I just haven't dealt with SOAP or REST API via scripting before.
I'm looking for how to write a custom splunk command, so take an ARG from a search result and run it against the SOAP interface on an external system. Returning the information back into Splunk.

[jkat54]

If you want to take data from within Splunk and post it to an api then use the results in the search stream, then you can do that with the curl command in jkats toolkit found on splunkbase.

[mrgibbon]

Thanks! I'll take a look 🙂

[jkat54]

Just start a thread and tag the app if you have any questions.

[mrgibbon]

Will do, i'll be wanting to index the returned results for a start, not just have them displayed.
As all this will be possibly done via overnight summary indexing.

[jkat54 · Accepted solution]

You need to create a scripted input that does the ETL for you. If you only need the data available in a search, you can use the curl command in the app I created called jkats toolkit.

check out this python code for getting a session from splunk using admin/password as the username/pass:

import sys
import re
import json
import requests
import splunk.Intersplunk 
import splunk.mining.dcutils as dcu

logger = dcu.getLogger()

###cant make help context work... dont know why
help = """------------------------------------------------------------------------------------
motd title="title" message="message" severity="{warn|info|error}"
------------------------------------------------------------------------------------"""

contexthelp = """------------------------------------------------------------------------------------
motd creates a bulletin message
------------------------------------------------------------------------------------"""
def getSession(username,password):
 uri = "https://localhost:8089/services/auth/login"
 r = requests.get(uri, data={'username':username,'password':password}, verify=False)
 sessionkey = re.sub('"',"",json.dumps(re.sub('<response>\n\s+<sessionKey>|<\/sessionKey>\n<\/response>\n',"",r.text)))
 return sessionkey

def motd(results,sessionKey, title="default title",message="default message",severity="info"):
 try:
  uri = "https://localhost:8089/services/messages/new"
  #headers = {'Authorization':'Splunk '}
  headers = {'Authorization':''}
  headers['Authorization'] = 'Splunk ' + sessionKey
  data = {'name':title,'value':message,'severity':severity}
  logger.info(data)
  r = requests.post(uri, headers=headers, data=data, verify=False)
  if r.status_code<300:
   logger.info("Status Code: " + str(r.status_code))
   for result in results:
    result["motd"] = "true"
   return results
  else:
   logger.error("Status Code: " + str(r.status_code))
   for result in results:
    result["motd"] = str(r.status_code) 
   return results
 except Exception, e:
  logger.exception(e)
  logger.exception("sessionKey: " + sessionKey)
  for result in results:
   result["motd"] = e
  return results

#get the arguments
(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
for a in sys.argv[1:]:
 if a.startswith("title="):
  title = re.sub("^.*=","",a)
  logger.info("Title: " + title)
 if a.startswith("message="):
  message = re.sub("^.*=","",a)
  logger.info("Message: " + message)
 if a.startswith("severity=warn") or a.startswith("severity=error") or a.startswith("severity=info"):
  severity = re.sub("^.*=","",a)
  logger.info("Severity: " + severity)
 elif not a.startswith("severity=warn") or not a.startswith("severity=error") or not a.startswith("severity=info"):
  severity = "info"
  logger.warn("Severity not provided, defaulting to " + severity)
 elif isgetinfo:
   splunk.Intersplunk.parseError("Invalid argument '%s'" % a)

# get the previous search results
results,dummy,settings = splunk.Intersplunk.getOrganizedResults()
logger.info(json.dumps(settings))

#get a session key
sessionKey = getSession("admin","password")
logger.info(sessionKey)

#set the message of the day using the arguments, all of them are optional
motd = motd(results,sessionKey,title,message,severity)

# return the previous search results
splunk.Intersplunk.outputResults(motd)

It gets the session in the getSession() function.

It uses the session id in the motd() function to post an alert message in splunk.

Maybe you can hack the code to do what you need to do.

[guimilare]

Hi jkat54, i fact I'll have to write a script.
Thanks for the code, however I'll have to write it on SH, since I don't have teh expertise in python.
I'll let you know how things go...

[guimilare]

Hi all.
I worte a SH script that worked as expected.
I'll try now to write it on Python.
Thanks for the help.

[jameswatts]

Can you post it for all to see?
Thanks.

[mrgibbon]

Yes! Please do 🙂

[mrgibbon]

Hi guimilare, Please let me know too, I might even be able to help out, although I dont have any python knowledge either! 🙂