All Apps and Add-ons

Azure Monitor Add-on For Splunk: Why are Inputs configured correctly but no data is being indexed?

bcootes
Explorer

I added the Input via the GUI and have triple checked the details. Still, I get the errors in splunkd.log below:

======

  10-22-2018 06:21:36.977 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" Modular input azure_diagnostic_logs://AMDL No connection on hub: insights-logs-alerts. Is there a network route to the endpoint?
    10-22-2018 06:22:14.105 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"   File "mask_secret.py", line 31, in 
    10-22-2018 06:22:14.105 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     import splunklib.client as client
    10-22-2018 06:22:14.105 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ImportError: No module named splunklib.client

=========================

Is it a network or an application issue? The Heavy Forwarder that this TA is installed on 'should' have the necessary outbound permissions. I presume you don't have to modify any inbound ACLs on the Eventhub?

1 Solution

jconger
Splunk Employee
Splunk Employee

This sounds like the splunklib directory is not in the app's bin folder. Try copying (not moving) the splunklib folder from the app's bin/app folder to the app's bin folder.

View solution in original post

0 Karma

jeremiahhainly
Explorer

Installing and validating permissions for splunklib will resolve "No module named splunklib.client"

Opening TCP/5671 to Azure on the firewall will resolve "Modular input azure_diagnostic_logs://AMDL No connection on hub: insights-logs-alerts. Is there a network route to the endpoint?"

0 Karma

lmjoin
Explorer

need to open port 5671 on azure side , so we can telnet port from heavy forwarder

0 Karma

bcootes
Explorer

I was able to fix the issue you're experiencing, anywhere99, by ensuring the TA could make outbound connection on TCP 5671 to Azure

anywhere99
Explorer

Hi , I still facing the “ No onnection on hub: insights-operational-logs. Is there a network route to the endpoint?”

Something related in the azure side?

0 Karma

jeremiahhainly
Explorer

Make sure TCP/5671 to Azure Event Hub is open on your firewall

0 Karma

bcootes
Explorer

Thank you for the advice. That seems to have fixed the splunklib error. Still left with what reads like a communication issue

10-22-2018 23:08:08.524 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" Modular input azure_activity_log://AMDL No connection on hub: insights-operational-logs. Is there a network route to the endpoint?

0 Karma

jconger
Splunk Employee
Splunk Employee

This sounds like the splunklib directory is not in the app's bin folder. Try copying (not moving) the splunklib folder from the app's bin/app folder to the app's bin folder.

0 Karma

njytrde
Explorer

Hi jconger,

I went to copy the splunklib as you directed, but all the files in the splunklib are already in the bin directory. Yes, I had my network team open ports 5671 and 5672 but I am only receiving the activity logs, not the diagnostic logs.

-rw-rw-r-- 1 splunk splunk 57301 May 29 13:09 binding.py
-rw------- 1 splunk splunk 55281 May 29 13:09 binding.pyc
-rw-rw-r-- 1 splunk splunk 142847 May 29 13:09 client.py
-rw------- 1 splunk splunk 148687 May 29 13:09 client.pyc
-rw-rw-r-- 1 splunk splunk 8528 May 29 13:09 data.py
-rw------- 1 splunk splunk 9235 May 29 13:09 data.pyc
-rw-rw-r-- 1 splunk splunk 771 May 29 13:09 init.py
-rw------- 1 splunk splunk 451 May 29 13:09 init.pyc
drwxrwxr-x 2 splunk splunk 4096 May 29 13:09 modularinput
-rw-rw-r-- 1 splunk splunk 4223 May 29 13:09 ordereddict.py
-rw-rw-r-- 1 splunk splunk 4692 May 29 13:09 ordereddict.pyc
-rw-rw-r-- 1 splunk splunk 10820 May 29 13:09 results.py
-rw-rw-r-- 1 splunk splunk 9437 May 29 13:09 results.pyc
drwxrwxr-x 2 splunk splunk 4096 May 29 13:09 searchcommands
-rw-rw-r-- 1 splunk splunk 30098 May 29 13:09 six.py
-rw------- 1 splunk splunk 31599 May 29 13:09 six.pyc
[root@xxxxxxxxxx splunklib]# pwd
/opt/splunk/etc/apps/TA-Azure_Monitor/bin/splunklib

Any ideas?

0 Karma

bcootes
Explorer

I have resolved the issues with a restart of splunk

To summarise issues i had:

  • splunklib not in app folder (in TA-Azure-Monitor) even though I followed documentation instructions for install
  • heavy forwarder where the app was installed couldn't make an outbound connection on TCP 5671 to the Azure EventHub
  • python script(s) didn't work properly until I restarted the heavy forwarder - recycling the app via disable/enable wasn't sufficient
0 Karma

bcootes
Explorer

Thank you - I have resolved that splunklib issues and the underlying network issue - Looks like there's still script errors. I don't get any of the documented initialization errors in Splunk by the way and installed all the dependencies as per the doco. Appreciate any insights into the errors below:

10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:367
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" var operationNameRaw = data.operationName.toUpperCase() || '';
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ^
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" TypeError: Cannot read property 'toUpperCase' of undefined
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at messageHandler (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:367:54)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:576:17
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Array.forEach (native)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ehMessageHandler (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/azure_monitor_logs.js:575:21)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at emitTwo (events.js:106:13)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.emit (events.js:191:7)
10-23-2018 00:06:08.322 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink._messageReceived (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/receiver_link.js:203:8)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Session._processTransferFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/session.js:419:45)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Session._processFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/session.js:352:63)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection._processFrameEH (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/session.js:224:49)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at emitOne (events.js:101:20)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.emit (events.js:188:7)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection._receiveAny (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:427:12)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection._receiveData (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:358:8)
10-23-2018 00:06:08.323 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at TlsTransport. (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:516:38)

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...