All Apps and Add-ons

REST API Modular Input is not working

nitishgku
Loves-to-Learn

REST API Modular Input v1.5.6

I'm trying to pull Defender ATP alerts but getting this error.

112-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: No JSON object could be decoded
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" raise ValueError("No JSON object could be decoded")
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/json/decoder.py", line 382, in raw_decode
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" obj, end = self.raw_decode(s, idx=w(s, 0).end())
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/json/decoder.py", line 364, in decode
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" return _default_decoder.decode(s)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/json/
init.py", line 339, in loads
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" output = json.loads(raw_response_output)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 440, in __call
_
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" RESPONSE_HANDLER_INSTANCE(response,output,type,req_args,endpoint)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 657, in handle_output
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" handle_output(r,r.text,response_type,req_args,endpoint)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 564, in do_run
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" self.target(*self.args, **self.__kwargs)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/threading.py", line 754, in run
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" self.run()
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/threading.py", line 801, in __bootstrap_inner
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Traceback (most recent call last):
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception in thread Thread-1:

Need your help to fix this issue.

Thanks!

0 Karma

woodcock
Esteemed Legend

I just got this working TODAY!

What most people do not understand (and indeed, neither did I) is that this TA probably won't work for you right out of the box.  It generally requires that a custom event handler be written to process the data payload that is returned by your REST endpiont.  That is the case here.  The default handler cannot handle this payload.  Neither can the default JSON handler because it is expecting pure JSON but that is not what the Windows Defender API returns.  Its payload is newline-delineated serial JSON lines.  Therefore we need a custom handler that can process that kind of payload.  Thankfully you can get excellent community support for this app by joining the BaboonBones slack for free here:
https://www.baboonbones.com/

Those are the guys that helped me get this all cleared up.

First, you need to add this to the existing responsehandler.py file:

class WindowsDefenderATPJSONArrayHandler:
    def __init__(self,**args):
        pass
    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint,oauth2):
        if response_type == "json":
            output = json.loads(raw_response_output)
            for alert in output:
                print_xml_stream(json.dumps(alert))
            # use/set checkpoint value based on time
            if not "params" in req_args:
                req_args["params"] = {}
            # Increment the checkpoint time for next run to be "now"
            # This will get automagically persisted
            date_from = datetime.now()
            req_args["params"]["sinceTimeUtc"] = date_from

        else:
            print_xml_stream(raw_response_output)

Last, your inputs.conf file should look like this:

[rest://Windows Defender ATP]
activation_key = <short distinct redacted>
auth_type = oauth2
client_key_path =
endpoint = https://wdatp-alertexporter-us.windows.com/api/alerts
http_method = GET
index_error_response_codes = 0
oauth2_access_token = <very long distinct redacted>
oauth2_client_id = <short distinct redacted>
oauth2_client_secret = <short distinct redacted>
oauth2_refresh_token = <medium distinct redacted>
oauth2_refresh_url = https://login.windows.net/[short distinct redacted]/oauth2/token
polling_interval = 60
response_handler = WindowsDefenderATPJSONArrayHandler
response_type = json
sequential_mode = 0
sourcetype = wdatp:alerts
streaming_request = 0
disabled = 0
expires_in = 3600
index = foo

 

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@nitishgku

Your configured Defender ATP alerts API not returning valid JSON response. It is probably with below reason.

  • Something going wrong during API call might be authentication isuse OR server response issue. ( check it by doing manual call from splunk instance, like curl or any other way)
  • If there is no issue with call then check type of response text. That should be JSON.

shiboo
Loves-to-Learn Lots

Any Solution for this. It is intermittent for me. Sometimes work. Getting below error.

 

/opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: No JSON object could be decoded
event_message = message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: No JSON object could be decodedhost = *.global.locsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd



08-21-2020 08:31:42.850 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized
event_message = message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorizedhost = *.global.locsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

Tags (1)
0 Karma

hkubavat_splunk
Splunk Employee
Splunk Employee

You need to put the response in the log. It seems there is an error or something that is being return.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!