Hi - I have a few dashboards that use expressions like eval var=ifnull(x,"true","false") ...which assigns "true" or "false" to var depending on x being NULL Those dashboards still work, but I notice that ifnull() does not show up in any of the current documentation, and it seems the current way to get the same result would be eval var=if(isnull(x),"true","false") Did I miss some kind of deprecation of that syntax ages ago (must have been before 6.3.0), and it just happens to still be parsed? ... View more

I've been trying to whip up a quick proof-of-conecpt installation of the NetApp app on our existing Splunk enterprise instance (running on Debian/wheezy)... Unfortunately, the data collector doesn't actually seem to want to connect to our cDOT systems - I do see connections from the search head as I add ONTAP collection targets in the app settings, though - presumably checking credentials. (I have the Splunk App for NetApp Data ONTAP 2.0.2 on Splunk 6.2.1) I set up an additional Splunk heavy forwarder on a CentOS box for the data collector. The scheduler still runs on our Debian search head. Right now, on the CentOS data collector, there is a Socket error visible in splunkd.log: 12-20-2014 14:46:21.238 +0100 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap_collection_worker.py 12-20-2014 14:46:21.238 +0100 INFO ExecProcessor - interval: run once 12-20-2014 14:46:21.239 +0100 INFO ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage 12-20-2014 14:46:21.239 +0100 INFO ExecProcessor - interval: 0 ms 12-20-2014 14:46:21.925 +0100 WARN HttpListener - Socket error from 127.0.0.1 while accessing /services/hydra/hydra_gatekeeper/hydra_gateway: Broken pipe I can't find out where that orginates from, though. At the same time, I see the following messages in hydra_scheduler_ta_ontap_collection_scheduler_nidhogg.log on the scheduler: 2014-12-20 14:46:21,286 INFO [ta_ontap_collection_scheduler://nidhogg] [HydraWorkerNode] New meta data is distributed: Owner: admin, Namespace: Splunk_TA_ontap, Name: metadata, Id: /servicesNS/nobody/Splunk_TA_ontap/configs/conf-hydra_metadata/metadata. 2014-12-20 14:46:21,286 DEBUG [ta_ontap_collection_scheduler://nidhogg] [HydraWorkerNodeManifest] checking the status of all nodes 2014-12-20 14:46:21,293 DEBUG [ta_ontap_collection_scheduler://nidhogg] [HydraWorkerNodeManifest] checking health of node=https://172.16.123.12:8089 2014-12-20 14:46:21,331 DEBUG [ta_ontap_collection_scheduler://nidhogg] [HydraWorkerNode] no heads regrown after they cried for help on node=https://172.16.123.12:8089 2014-12-20 14:46:21,331 DEBUG [ta_ontap_collection_scheduler://nidhogg] Updated status of active nodes 2014-12-20 14:46:21,331 DEBUG [ta_ontap_collection_scheduler://nidhogg] Checked status of dead nodes 2014-12-20 14:46:21,331 DEBUG [ta_ontap_collection_scheduler://nidhogg] [HydraWorkerNodeManifest] checking the status of all nodes 2014-12-20 14:47:16,119 ERROR [ta_ontap_collection_scheduler://nidhogg] [HydraWorkerNode] node=https://172.16.123.12:8089 is likely dead, could not get info on current job count, msg : <urlopen error Tunnel connection failed: 502 cannotconnect> Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 933, in getActiveJobInfo job_info = self.gateway_adapter.get_job_info() File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_common.py", line 199, in get_job_info resp = self.opener.open(req) File "/opt/splunk/lib/python2.7/urllib2.py", line 404, in open response = self._open(req, data) File "/opt/splunk/lib/python2.7/urllib2.py", line 422, in _open '_open', req) File "/opt/splunk/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/opt/splunk/lib/python2.7/urllib2.py", line 1222, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/opt/splunk/lib/python2.7/urllib2.py", line 1184, in do_open raise URLError(err) URLError: <urlopen error Tunnel connection failed: 502 cannotconnect> Any hints where I can look to further debug this error? Possibly unrelated, I have noticed that there still are two ta_ontap_collection_scheduler.py - processes lingering around on the scheduler even after I stop Splunk. Does the scheduler maybe have problems on a Debian host, too (the install docs just note that the data collector has to run on RHEL or CentOS)? ... View more

I have noticed that the eventtype cisco_ios-diag in TA-cisco_ios matches on some of the log file entries generated by our Juniper switches (primarily log lines matching facility=KERN). As far as I have traced this through TA-cisco_ios, force_sourcetype_for_cisco_ios in the TA's transforms.conf matches on some JunOS log entries (and overwrites our custom source type). I don't have a good idea on how to exclude these log lines yet, though - especially in a way that could be included into the TA so we don't have to apply a local fix after every update... Did anyone run into this yet and has adapted the TA-cisco_ios ruleset? Example Juniper log entry: Nov 26 14:09:48 jp45-xxx /kernel: %KERN-5-KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/33 - CD state - ready to carry traffic ... View more

In earlier versions of Splunk, it was possible to CTRL-click on statistics results, and have a search with the appropriate field opened in a new browser window. In Splunk 6, the new search is always opened in the same window, and I have to rely on the browser back button to go back to the statistics view. That's a major hassle when trying to quickly go through event details from statistics - especially as going back will always end up on the first page of the statistics, not the one you were coming from. Simple example: I have a search in the line of host=server | top limit=100 id. When I click on one of the results in the id row of the statistics page, Splunk 6 will start a new search with host=server id=result in the same browser window, losing the statistics overview. Is there any way to get the old behaviour back? ... View more