Azure Monitor Add-on For Splunk: Why are Inputs configured correctly but no data is being indexed?

[bcootes]

I added the Input via the GUI and have triple checked the details. Still, I get the errors in splunkd.log below:

======

  10-22-2018 06:21:36.977 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" Modular input azure_diagnostic_logs://AMDL No connection on hub: insights-logs-alerts. Is there a network route to the endpoint?
    10-22-2018 06:22:14.105 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"   File "mask_secret.py", line 31, in 
    10-22-2018 06:22:14.105 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     import splunklib.client as client
    10-22-2018 06:22:14.105 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ImportError: No module named splunklib.client

=========================

Is it a network or an application issue? The Heavy Forwarder that this TA is installed on 'should' have the necessary outbound permissions. I presume you don't have to modify any inbound ACLs on the Eventhub?

[jeremiahhainly]

Installing and validating permissions for splunklib will resolve "No module named splunklib.client"

Opening TCP/5671 to Azure on the firewall will resolve "Modular input azure_diagnostic_logs://AMDL No connection on hub: insights-logs-alerts. Is there a network route to the endpoint?"

[lmjoin]

need to open port 5671 on azure side , so we can telnet port from heavy forwarder

[bcootes]

I was able to fix the issue you're experiencing, anywhere99, by ensuring the TA could make outbound connection on TCP 5671 to Azure

[anywhere99]

Hi , I still facing the “ No onnection on hub: insights-operational-logs. Is there a network route to the endpoint?”

Something related in the azure side?

[jeremiahhainly]

Make sure TCP/5671 to Azure Event Hub is open on your firewall

[bcootes]

Thank you for the advice. That seems to have fixed the splunklib error. Still left with what reads like a communication issue

10-22-2018 23:08:08.524 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" Modular input azure_activity_log://AMDL No connection on hub: insights-operational-logs. Is there a network route to the endpoint?

[jconger · Accepted solution]

This sounds like the splunklib directory is not in the app's bin folder. Try copying (not moving) the splunklib folder from the app's bin/app folder to the app's bin folder.

[njytrde]

Hi jconger,

I went to copy the splunklib as you directed, but all the files in the splunklib are already in the bin directory. Yes, I had my network team open ports 5671 and 5672 but I am only receiving the activity logs, not the diagnostic logs.

-rw-rw-r-- 1 splunk splunk 57301 May 29 13:09 binding.py
-rw------- 1 splunk splunk 55281 May 29 13:09 binding.pyc
-rw-rw-r-- 1 splunk splunk 142847 May 29 13:09 client.py
-rw------- 1 splunk splunk 148687 May 29 13:09 client.pyc
-rw-rw-r-- 1 splunk splunk 8528 May 29 13:09 data.py
-rw------- 1 splunk splunk 9235 May 29 13:09 data.pyc
-rw-rw-r-- 1 splunk splunk 771 May 29 13:09 init.py
-rw------- 1 splunk splunk 451 May 29 13:09 init.pyc
drwxrwxr-x 2 splunk splunk 4096 May 29 13:09 modularinput
-rw-rw-r-- 1 splunk splunk 4223 May 29 13:09 ordereddict.py
-rw-rw-r-- 1 splunk splunk 4692 May 29 13:09 ordereddict.pyc
-rw-rw-r-- 1 splunk splunk 10820 May 29 13:09 results.py
-rw-rw-r-- 1 splunk splunk 9437 May 29 13:09 results.pyc
drwxrwxr-x 2 splunk splunk 4096 May 29 13:09 searchcommands
-rw-rw-r-- 1 splunk splunk 30098 May 29 13:09 six.py
-rw------- 1 splunk splunk 31599 May 29 13:09 six.pyc
[root@xxxxxxxxxx splunklib]# pwd
/opt/splunk/etc/apps/TA-Azure_Monitor/bin/splunklib

Any ideas?

[bcootes]

I have resolved the issues with a restart of splunk

To summarise issues i had:

• splunklib not in app folder (in TA-Azure-Monitor) even though I followed documentation instructions for install
• heavy forwarder where the app was installed couldn't make an outbound connection on TCP 5671 to the Azure EventHub
• python script(s) didn't work properly until I restarted the heavy forwarder - recycling the app via disable/enable wasn't sufficient

[bcootes]

Thank you - I have resolved that splunklib issues and the underlying network issue - Looks like there's still script errors. I don't get any of the documented initialization errors in Splunk by the way and installed all the dependencies as per the doco. Appreciate any insights into the errors below:

10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:367
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" var operationNameRaw = data.operationName.toUpperCase() || '';
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ^
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" TypeError: Cannot read property 'toUpperCase' of undefined
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at messageHandler (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:367:54)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:576:17
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Array.forEach (native)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ehMessageHandler (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:575:21)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at emitTwo (events.js:106:13)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.emit (events.js:191:7)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink._messageReceived (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/receiver_link.js:203:8)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Session._processTransferFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/session.js:419:45)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Session._processFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/session.js:352:63)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection._processFrameEH (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/session.js:224:49)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at emitOne (events.js:101:20)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.emit (events.js:188:7)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection._receiveAny (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:427:12)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection._receiveData (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:358:8)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at TlsTransport. (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:516:38)