Large amount of erorrs in Microsft Azure-Add on for Splunk

[ajiwanand]

I'll start out by saying the collection of logs from eventhub via this add-on works fine. I am seeing events in the azure index and they seem to be coming in just fine, however there is a significant amount of errors in splunkd.log around the TA-MS-AAD app.

Errors:

07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/async_ops/client_async.py", line 835, in _client_run_async
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     await self._connection.work_async()
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/async_ops/connection_async.py", line 139, in work_async
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     self._conn.do_work()
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/receiver.py", line 239, in _message_received
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     delivery_no=message_number)
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/etc/apps/TA-MS-AAD/bin/uamqp/message.py", line 99, in __init__
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     self._parse_message_body(message)
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Message: 'Deallocating %r'
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Arguments: ('ArrayValue',)
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" --- Logging error ---
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Traceback (most recent call last):
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/lib/python3.7/logging/handlers.py", line 69, in emit
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     if self.shouldRollover(record):
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/lib/python3.7/logging/handlers.py", line 186, in shouldRollover
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" RuntimeError: reentrant call inside <_io.BufferedWriter name='/opt/splunk/var/log/splunk/ta_ms_aad_azure_event_hub.log'>
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Call stack:
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"   File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py", line 4, in <module>
07-30-2020 20:22:39.836 +0000 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py"     import azure_event_hub_core

I can't really pin-point what the errors may be, but this is driving up the amount of logs we're ingesting significantly, up to the point where its actually affecting out licenses.

Any idea?

[richgalloway]

I don't have a solution, but splunkd.log does not count against your license.

[ajiwanand · Accepted solution]

really, interesting then i guess i need to do more investigation into why we're having licensing errors.

Also, after digging more into this i believe the issue was simply having the logs set to DEBUG for the Microsoft Azure Add-on for Splunk. Simply changing this to WARNING, removed these errors. I guess this App implements the splunk logging module incorrectly, but it doesnt seem to affect ingested events.