NetApp logs not showing on the Splunk App for NetApp

[cemiam]

Hi,

I have installed Splunk App for NetApp and complete required configurations. Connection to Data Collection Node which is installed as Heavy Forwarder seems ok. Also the "ONTAP Collection Configuration" looks valid. Environment has a search head and two indexers. I have set one of the indexers as forward server for the Heavy Forwarder. After the installations dashboards are still empty and I cannot find any event on the search head and Heavy Forwarder.

I have checked the logs with the help of the "index=_internal (source=netapp OR source=hydra)" command and noticed too many of the logs below.

2017-06-08 16:04:53,581 INFO [ta_ontap_collection_worker://theta:6831] [getJob] could not find a job to do, sleeping before retry

2017-06-08 16:04:53,577 INFO GET /hydra/job/pop '404 Not Found' - - - 2500ms

I am trying to get what is wrong with the configurations. Is this error might be related with NetApp?

Best Regards,
Cem

[archspangler]

The subject states "NetApp logs not showing on the Splunk App for NetApp". The Perf data and syslog event data is collected differently.

Have you configured splunk to listen for syslog messages from your filers?
Have you configured your c-mode\7-mode filers to send syslog events to your splunk syslog listener?

If you run splunkd as non-root user; splunk cannot create the listener port under 1024. You will need to use a port above 1024 or use iptables to NAT the port..

https://answers.splunk.com/answers/1653/cant-add-udp-input-because-of-error-udp-port-514-is-not-avai...

[cemiam]

UDP port 514 is succesfully added to the data inputs after installing the NetApp Add-on. We are using NetApp on 7-mode and I have run the command for sending syslog messages to a specific IP. I have also checked the configuration file. All seems fine. I have set a stand alone Splunk instance's IP address instead Data Collection Node's and I can successfully get the syslog messages.

[cemiam]

I have seen your comment on the below discussion. I am getting the same errors on the 7-Mode installation. Do you have chance to update me if you can get some information to resolve this problem?

https://answers.splunk.com/answers/536342/splunk-add-on-for-netapp-data-ontap-after-install.html

Best Regards,

[cemiam]

I have also noticed below error message and re-enabled the sslv3. I have also changed URL with the IP address of the NetApp server to make sure it could cause trouble. However I am still getting error messages.

2017-06-09 14:50:29,438 ERROR [ta_ontap_collection_worker://eta:9689] [loginToTarget] Failed to create OntapClient for target=https://10.19.2.235:443 user=comparex: [OntapClient] Client Side Code Error 13001: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:676)
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap_collection_worker.py", line 148, in loginToTarget
major, minor = oc.getApiVersion(cached=False)
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 251, in getApiVersion
results = self.queryApi('system-get-ontapi-version',OntapClient.convertElementToXML)
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 443, in queryApi
raise ClientSideError(errno, reason)
ClientSideError: [OntapClient] Client Side Code Error 13001: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:676)

2017-06-09 14:50:29,439 ERROR [ta_ontap_collection_worker://eta:9689] Could not login to target=https://10.19.2.235:443 with username=comparex after num_retries=3, setting OntapClient as None and marking this target as bad in the worker

2017-06-09 14:50:29,440 ERROR [ta_ontap_collection_worker://eta:9689] Failed to complete job="job_d7fb1e544d0911e79c56005056b6295e" of task="megainv" against target="https://10.19.2.235:443"

Best Regards,
Cem

[cemiam]

Hi Lukas,

Thanks for your response. I have checked the heavy forwarder logs with the help of the query you have provided. I have found below article when I googled the error messages.

https://answers.splunk.com/answers/405915/splunk-app-for-netapp-data-ontap-after-upgrade-fro.html

I have enabled TLS as mentioned on the article however I am still getting the error messages below.

2017-06-09 14:18:44,143 ERROR [ta_ontap_collection_worker://beta:7436] [loginToTarget] Failed to create OntapClient for target=https://netappsim.dataserv.local:443 user=comparex: [OntapClient] Client Side Code Error 13001: [Errno -2] Name or service not known
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap_collection_worker.py", line 148, in loginToTarget
major, minor = oc.getApiVersion(cached=False)
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 251, in getApiVersion
results = self.queryApi('system-get-ontapi-version',OntapClient.convertElementToXML)
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 443, in queryApi
raise ClientSideError(errno, reason)
ClientSideError: [OntapClient] Client Side Code Error 13001: [Errno -2] Name or service not known

2017-06-09 14:18:44,144 ERROR [ta_ontap_collection_worker://beta:7436] Could not login to target=https://netappsim.dataserv.local:443 with username=comparex after num_retries=3, setting OntapClient as None and marking this target as bad in the worker

2017-06-09 14:18:44,145 ERROR [ta_ontap_collection_worker://beta:7436] Failed to complete job="job_655f139c4d0511e78f28005056b6295e" of task="megainv" against target="https://netappsim.dataserv.local:443"

Best Regards,
Cem

[lukas_loder]

Hello Cem,

I'm working with the NetApp App as well. And I have all the logs I need to fill the Graphs in the NetApp App.
But I also have both of your messages you have (sleeping and 404).
If you search for

index=_internal level=ERROR host=YourHeavyforwarder

you find some errors?

[cemiam]

Hi Lukas,

Thanks for the response. I have get the Forwarder as required. You can find them below. I have added additional details on the posts below.

2017-06-09 14:18:44,143 ERROR [ta_ontap_collection_worker://beta:7436] [loginToTarget] Failed to create OntapClient for target=https://netappsim.dataserv.local:443 user=comparex: [OntapClient] Client Side Code Error 13001: [Errno -2] Name or service not known
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap_collection_worker.py", line 148, in loginToTarget
major, minor = oc.getApiVersion(cached=False)
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 251, in getApiVersion
results = self.queryApi('system-get-ontapi-version',OntapClient.convertElementToXML)
File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 443, in queryApi
raise ClientSideError(errno, reason)
ClientSideError: [OntapClient] Client Side Code Error 13001: [Errno -2] Name or service not known

Best Regards,
Cem