All Apps and Add-ons

How to index a SOAP response using REST API Modular Input?

guimilare
Communicator

Hello Splunkers.

I have a WebService that I need to get data from.
I have to do the following steps:

1) Send a SOAP request to login to the WebService and get a SID (Session ID);
2) Use this SID to do a select on the WebService;
3) Index the SOAPresponse in Splunk;
3) Logout from the WebService.

How can I achieve this?
My first thought was using REST API.

I'm a bit lost on how to implement this.
Is REST API the best way? Or is using a script better?

Thanks in advance!
Best regards,
GMA

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

You need to create a scripted input that does the ETL for you. If you only need the data available in a search, you can use the curl command in the app I created called jkats toolkit.

check out this python code for getting a session from splunk using admin/password as the username/pass:

import sys
import re
import json
import requests
import splunk.Intersplunk 
import splunk.mining.dcutils as dcu

logger = dcu.getLogger()

###cant make help context work... dont know why
help = """------------------------------------------------------------------------------------
motd title="title" message="message" severity="{warn|info|error}"
------------------------------------------------------------------------------------"""

contexthelp = """------------------------------------------------------------------------------------
motd creates a bulletin message
------------------------------------------------------------------------------------"""
def getSession(username,password):
 uri = "https://localhost:8089/services/auth/login"
 r = requests.get(uri, data={'username':username,'password':password}, verify=False)
 sessionkey = re.sub('"',"",json.dumps(re.sub('<response>\n\s+<sessionKey>|<\/sessionKey>\n<\/response>\n',"",r.text)))
 return sessionkey

def motd(results,sessionKey, title="default title",message="default message",severity="info"):
 try:
  uri = "https://localhost:8089/services/messages/new"
  #headers = {'Authorization':'Splunk '}
  headers = {'Authorization':''}
  headers['Authorization'] = 'Splunk ' + sessionKey
  data = {'name':title,'value':message,'severity':severity}
  logger.info(data)
  r = requests.post(uri, headers=headers, data=data, verify=False)
  if r.status_code<300:
   logger.info("Status Code: " + str(r.status_code))
   for result in results:
    result["motd"] = "true"
   return results
  else:
   logger.error("Status Code: " + str(r.status_code))
   for result in results:
    result["motd"] = str(r.status_code) 
   return results
 except Exception, e:
  logger.exception(e)
  logger.exception("sessionKey: " + sessionKey)
  for result in results:
   result["motd"] = e
  return results

#get the arguments
(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
for a in sys.argv[1:]:
 if a.startswith("title="):
  title = re.sub("^.*=","",a)
  logger.info("Title: " + title)
 if a.startswith("message="):
  message = re.sub("^.*=","",a)
  logger.info("Message: " + message)
 if a.startswith("severity=warn") or a.startswith("severity=error") or a.startswith("severity=info"):
  severity = re.sub("^.*=","",a)
  logger.info("Severity: " + severity)
 elif not a.startswith("severity=warn") or not a.startswith("severity=error") or not a.startswith("severity=info"):
  severity = "info"
  logger.warn("Severity not provided, defaulting to " + severity)
 elif isgetinfo:
   splunk.Intersplunk.parseError("Invalid argument '%s'" % a)

# get the previous search results
results,dummy,settings = splunk.Intersplunk.getOrganizedResults()
logger.info(json.dumps(settings))

#get a session key
sessionKey = getSession("admin","password")
logger.info(sessionKey)

#set the message of the day using the arguments, all of them are optional
motd = motd(results,sessionKey,title,message,severity)

# return the previous search results
splunk.Intersplunk.outputResults(motd)

It gets the session in the getSession() function.

It uses the session id in the motd() function to post an alert message in splunk.

Maybe you can hack the code to do what you need to do.

View solution in original post

mrgibbon
Contributor

I have been looking for a way of doing this with peoplesoft, please let me know if you work out how to do it!
Thanks

0 Karma

jkat54
SplunkTrust
SplunkTrust

What's wrong with my scripted input response? We integrate Splunk with APIs all the time. Write a script.

0 Karma

mrgibbon
Contributor

I'd love to have the skills and time to sit and write a script for this, and I need to. I just haven't dealt with SOAP or REST API via scripting before.
I'm looking for how to write a custom splunk command, so take an ARG from a search result and run it against the SOAP interface on an external system. Returning the information back into Splunk.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If you want to take data from within Splunk and post it to an api then use the results in the search stream, then you can do that with the curl command in jkats toolkit found on splunkbase.

0 Karma

mrgibbon
Contributor

Thanks! I'll take a look 🙂

0 Karma

jkat54
SplunkTrust
SplunkTrust

Just start a thread and tag the app if you have any questions.

0 Karma

mrgibbon
Contributor

Will do, i'll be wanting to index the returned results for a start, not just have them displayed.
As all this will be possibly done via overnight summary indexing.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You need to create a scripted input that does the ETL for you. If you only need the data available in a search, you can use the curl command in the app I created called jkats toolkit.

check out this python code for getting a session from splunk using admin/password as the username/pass:

import sys
import re
import json
import requests
import splunk.Intersplunk 
import splunk.mining.dcutils as dcu

logger = dcu.getLogger()

###cant make help context work... dont know why
help = """------------------------------------------------------------------------------------
motd title="title" message="message" severity="{warn|info|error}"
------------------------------------------------------------------------------------"""

contexthelp = """------------------------------------------------------------------------------------
motd creates a bulletin message
------------------------------------------------------------------------------------"""
def getSession(username,password):
 uri = "https://localhost:8089/services/auth/login"
 r = requests.get(uri, data={'username':username,'password':password}, verify=False)
 sessionkey = re.sub('"',"",json.dumps(re.sub('<response>\n\s+<sessionKey>|<\/sessionKey>\n<\/response>\n',"",r.text)))
 return sessionkey

def motd(results,sessionKey, title="default title",message="default message",severity="info"):
 try:
  uri = "https://localhost:8089/services/messages/new"
  #headers = {'Authorization':'Splunk '}
  headers = {'Authorization':''}
  headers['Authorization'] = 'Splunk ' + sessionKey
  data = {'name':title,'value':message,'severity':severity}
  logger.info(data)
  r = requests.post(uri, headers=headers, data=data, verify=False)
  if r.status_code<300:
   logger.info("Status Code: " + str(r.status_code))
   for result in results:
    result["motd"] = "true"
   return results
  else:
   logger.error("Status Code: " + str(r.status_code))
   for result in results:
    result["motd"] = str(r.status_code) 
   return results
 except Exception, e:
  logger.exception(e)
  logger.exception("sessionKey: " + sessionKey)
  for result in results:
   result["motd"] = e
  return results

#get the arguments
(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
for a in sys.argv[1:]:
 if a.startswith("title="):
  title = re.sub("^.*=","",a)
  logger.info("Title: " + title)
 if a.startswith("message="):
  message = re.sub("^.*=","",a)
  logger.info("Message: " + message)
 if a.startswith("severity=warn") or a.startswith("severity=error") or a.startswith("severity=info"):
  severity = re.sub("^.*=","",a)
  logger.info("Severity: " + severity)
 elif not a.startswith("severity=warn") or not a.startswith("severity=error") or not a.startswith("severity=info"):
  severity = "info"
  logger.warn("Severity not provided, defaulting to " + severity)
 elif isgetinfo:
   splunk.Intersplunk.parseError("Invalid argument '%s'" % a)

# get the previous search results
results,dummy,settings = splunk.Intersplunk.getOrganizedResults()
logger.info(json.dumps(settings))

#get a session key
sessionKey = getSession("admin","password")
logger.info(sessionKey)

#set the message of the day using the arguments, all of them are optional
motd = motd(results,sessionKey,title,message,severity)

# return the previous search results
splunk.Intersplunk.outputResults(motd)

It gets the session in the getSession() function.

It uses the session id in the motd() function to post an alert message in splunk.

Maybe you can hack the code to do what you need to do.

guimilare
Communicator

Hi jkat54, i fact I'll have to write a script.
Thanks for the code, however I'll have to write it on SH, since I don't have teh expertise in python.
I'll let you know how things go...

0 Karma

guimilare
Communicator

Hi all.
I worte a SH script that worked as expected.
I'll try now to write it on Python.
Thanks for the help.

jameswatts
Explorer

Can you post it for all to see?
Thanks.

0 Karma

mrgibbon
Contributor

Yes! Please do 🙂

0 Karma

mrgibbon
Contributor

Hi guimilare, Please let me know too, I might even be able to help out, although I dont have any python knowledge either! 🙂

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...