REST API Modular Input v1.5.6
I'm trying to pull Defender ATP alerts but getting this error.
112-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: No JSON object could be decoded
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" raise ValueError("No JSON object could be decoded")
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/json/decoder.py", line 382, in raw_decode
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" obj, end = self.raw_decode(s, idx=w(s, 0).end())
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/json/decoder.py", line 364, in decode
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" return _default_decoder.decode(s)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/json/init.py", line 339, in loads
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" output = json.loads(raw_response_output)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 440, in __call_
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" RESPONSE_HANDLER_INSTANCE(response,output,type,req_args,endpoint)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 657, in handle_output
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" handle_output(r,r.text,response_type,req_args,endpoint)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 564, in do_run
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" self.target(*self.args, **self.__kwargs)
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/threading.py", line 754, in run
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" self.run()
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" File "/opt/splunk/lib/python2.7/threading.py", line 801, in __bootstrap_inner
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Traceback (most recent call last):
12-22-2019 05:18:50.905 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception in thread Thread-1:
Need your help to fix this issue.
Thanks!
I just got this working TODAY!
What most people do not understand (and indeed, neither did I) is that this TA probably won't work for you right out of the box. It generally requires that a custom event handler be written to process the data payload that is returned by your REST endpiont. That is the case here. The default handler cannot handle this payload. Neither can the default JSON handler because it is expecting pure JSON but that is not what the Windows Defender API returns. Its payload is newline-delineated serial JSON lines. Therefore we need a custom handler that can process that kind of payload. Thankfully you can get excellent community support for this app by joining the BaboonBones slack for free here:
https://www.baboonbones.com/
Those are the guys that helped me get this all cleared up.
First, you need to add this to the existing responsehandler.py file:
class WindowsDefenderATPJSONArrayHandler:
def __init__(self,**args):
pass
def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint,oauth2):
if response_type == "json":
output = json.loads(raw_response_output)
for alert in output:
print_xml_stream(json.dumps(alert))
# use/set checkpoint value based on time
if not "params" in req_args:
req_args["params"] = {}
# Increment the checkpoint time for next run to be "now"
# This will get automagically persisted
date_from = datetime.now()
req_args["params"]["sinceTimeUtc"] = date_from
else:
print_xml_stream(raw_response_output)
Last, your inputs.conf file should look like this:
[rest://Windows Defender ATP]
activation_key = <short distinct redacted>
auth_type = oauth2
client_key_path =
endpoint = https://wdatp-alertexporter-us.windows.com/api/alerts
http_method = GET
index_error_response_codes = 0
oauth2_access_token = <very long distinct redacted>
oauth2_client_id = <short distinct redacted>
oauth2_client_secret = <short distinct redacted>
oauth2_refresh_token = <medium distinct redacted>
oauth2_refresh_url = https://login.windows.net/[short distinct redacted]/oauth2/token
polling_interval = 60
response_handler = WindowsDefenderATPJSONArrayHandler
response_type = json
sequential_mode = 0
sourcetype = wdatp:alerts
streaming_request = 0
disabled = 0
expires_in = 3600
index = foo
@nitishgku
Your configured Defender ATP alerts
API not returning valid JSON response. It is probably with below reason.
Any Solution for this. It is intermittent for me. Sometimes work. Getting below error.
/opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: No JSON object could be decoded
event_message = message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: No JSON object could be decodedhost = *.global.locsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
08-21-2020 08:31:42.850 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized
event_message = message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorizedhost = *.global.locsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
You need to put the response in the log. It seems there is an error or something that is being return.