Why am I unable to configure Microsoft OMS Modular Inputs TA?

[dpanych]

I'm trying to pull WAF and Server logs from our Azure OMS. The first option we're piloting to accomplish this using the OMS TA. I downloaded Microsoft OMS Modular Inputs TA and am having trouble configuring/getting it to work. I'm seeing the following errors in splunkd.log. I believe Azure permissions are proper, but that's something that been configured by the Azure Admins. Anyone know what the errors are?

@jkat54 - Any insight?

05-04-2018 10:18:40.266 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" ERRORGet Token request returned http error: 400 and server response: {"error":"unauthorized_client","error_description":"AADSTS70001: Application with identifier 'AAAAAAAAAAAAAAAAAA' was not found in the directory BBBBBBBBBBBBBBBBBBBBBB\r\nTrace ID: CCCCCCCCCCCCCCCCCCCCCC\r\nCorrelation ID: DDDDDDDDDDDDDDDDDDDDDD\r\nTimestamp: 2018-05-04 17:18:39Z","error_codes":[70001],"timestamp":"2018-05-04 17:18:39Z","trace_id":"CCCCCCCCCCCCCCCCCCCCCC","correlation_id":"DDDDDDDDDDDDDDDDDDDDDD"}

05-04-2018 10:20:34.145 -0700 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/TA-OMS_Inputs/TA_oms_inputs_settings/proxy: Winsock error 10054

05-04-2018 10:20:28.897 -0700 INFO ExecProcessor - Removing status item ""C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py" (oms_inputs://csl_pws_oms) (isModInput=yes)

Many errors such as:

05-04-2018 10:18:39.819 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" return client.get_token(oauth_parameters)

05-04-2018 10:18:39.819 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\adal\oauth2_client.py", line 281, in get_token

[thambisetty]

could you please provide sample queries for OMS which can be executed through this TA?

I tested few queries in python script and got result below:

Request : search_params="{'query': 'Event | summarize count(Computer)'}'

Response : {"tables":[{"name":"PrimaryResult","columns":[{"name":"count_Computer","type":"long"}],"rows":[[23572]]}]}

When I tried testing same query in TA. getting an error like below,

2018-07-11 14:24:50,693 ERROR pid=24842 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-OMS_Inputs/bin/ta_oms_inputs/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-OMS_Inputs/bin/oms_inputs.py", line 96, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-OMS_Inputs/bin/input_module_oms_inputs.py", line 95, in collect_events
    search_id = data["id"].split("/")
KeyError: 'id'

as per the python scirpt in TA :

data = response.json()
            search_id = data["id"].split("/")
            id = search_id[len(search_id)-1]
            status = data["__metadata"]["Status"]

its expecting id from the response. if there is no id found in response then its throwing an error. if this is the behaviour we cant execute dynamic queries using this TA.

Please let me know how to parse this using this TA, otherwise I need to write parser for this. then there is no use of this TA.

[ips_mandar]

@thambisetty , are you able to fix OMS issue and able to get data in splunk ?

[thambisetty]

Yes. But you have to follow the document on creating azure app which will have access to your subscription and get the key of that app and secret these will be used in the script.

I have tested the connection using app key and secret its working fine.

I am thinking on what to collect. Full logs or the alerts for the configured saved searches in log analytics.

[ips_mandar]

@thambisetty once I configure azure app, from which files I can get key and secret ...can you guide me to receive oms data as currently I am getting error in OMS add-on.

[dpanych]

See my new post. We got the app working with the new API.

[thambisetty]

Please go through the conversation here. Its saying that the REST API which is used in this TA has been depricated by Microsoft and they have been using different queries all together.

Contact me on t.balaji2k12@gmail.com for further details on this.

[dpanych]

Here's a response I got from the author of the app:

Try using this article to convert the query to the “legacy”
Style that the api version this app uses will support.

https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-log-search-transition

The current TA uses the legacy Microsoft API to query OMS, so you must convert the query.

[jkat54]

I’ve been working on a version that lets us specify which version of the API to use. Microsoft had their own idea about that and they’re deprecating the API this app uses soon. So I will be releasing a completely new app and deprecating this app as soon as I can find the time.

[jkat54 · Accepted solution]

we found that @dpanych was using a newer query language that isnt supported by the API version that this uses.

We also found we had to add the following to the top of /bin/modular_input_oms_inputs.py

  from  splunklib.modularinput import *

The query we were able to get working was "Type=Alert".

[dpanych]

Additionally, we found that in the app GUI, one of the input fields is "Workspace ID", it should be Workspace Name

Again, thank you @jkat54 for helping developing such a wonderful app, and for helping troubleshoot the issue.

[jkat54]

APi.loganalytics.io is called the directAPI.

You access it a bit differently from the Azure API this app uses.

See this documentation for getting the necessary details:

https://dev.loganalytics.io/documentation/1-Tutorials/ARM-API

[dpanych]

@jkat54 Are the error message I mentioned before relevant to the link you posted? Errors seem to be referencing Python initialization problems.

[jkat54]

Yes. If you don’t provide the correct access the app fails to pull data and you’ll see this error message.

[jkat54]

Look at it this way api.loganalytics.io = endpoints you use if your app resides inside of an azure region.

the endpoints this app uses are for accessing OMS from outside of azure. A different token and api is required as such.

[dpanych]

@jkat54 I tried using the ARM API with Postman and it worked. I noticed the ARM API doesn't require workspace ID, but it does require workspace name. I tried putting the name into the Splunk Workspace ID field, that didn't work either. Any other suggestions? Access from both APIs seem to work as I'm able to successfully return data with Postman.. I don't have direct access to Azure so I've been working with the Admins (which is a PITA).

[jkat54]

So have you been able to provide everything my app asks for to the app?

Resource Group, Workspace ID, Subscription ID, Tenant ID, Application ID, and Application Key.

[dpanych]

@jkat54 Yes. If you have time, I can open up a Skype/Webex session (if you want to look).

[jkat54]

Says you’re unauthorized so permissions aren’t right.

Send this link to the admins:
https://dev.loganalytics.io/documentation/1-Tutorials/ARM-API

[dpanych]

Thank you @jkat54, this worked! Well at least partially... I was able to query OMS using the APIs in Postman, but in Splunk, I'm seeing the following errors.

05-18-2018 11:37:33.736 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" ERRORlocal variable 'data' referenced before assignment
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" UnboundLocalError: local variable 'data' referenced before assignment
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""     for data_value in data["value"]:
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""   File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\input_module_oms_inputs.py", line 106, in collect_events
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""     input_module.collect_events(self, ew)
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""   File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py", line 96, in collect_events
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""     self.collect_events(ew)
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""   File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\modinput_wrapper\base_modinput.py", line 127, in stream_events
05-18-2018 11:37:33.221 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" Traceback (most recent call last):
05-18-2018 11:34:33.802 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" ERRORlocal variable 'data' referenced before assignment
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" UnboundLocalError: local variable 'data' referenced before assignment
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""     for data_value in data["value"]:
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""   File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\input_module_oms_inputs.py", line 106, in collect_events
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""     input_module.collect_events(self, ew)
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""   File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py", line 96, in collect_events
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""     self.collect_events(ew)
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py""   File "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\modinput_wrapper\base_modinput.py", line 127, in stream_events
05-18-2018 11:34:33.301 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" Traceback (most recent call last):
05-18-2018 11:33:34.349 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" ERRORlocal variable 'data' referenced before assignment
05-18-2018 11:33:34.349 -0700 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-OMS_Inputs\bin\oms_inputs.py"" UnboundLocalError: local variable 'data' referenced before assignment

Using Postman to call the Log Analytics API, it required tenant_id, client_id(app id), redirect_uri(http://localhost:3000/login), resource(https://api.loganalytics.io), client_secret(app key), and workspace_id. ** Your app is asking for **Resource Group, Workspace ID, Subscription ID, Tenant ID, Application ID, and Application Key. Could this be the difference, or is the errors mentioned above a separate issue?

@jkat54