All Apps and Add-ons

Splunk for Cisco Identity Services (ISE) dashboards show "no results found" after upgrading the Splunk Add-on for Cisco ISE

Path Finder

I have been trying to figure out why the Splunk App for Cisco ISE quit working after I updated the Splunk Add-on for Cisco ISE, but I am not having any luck. I have verified that Cisco ISE is still sending syslog data to our syslog-ng server, and that the syslog-ng server is still processing it as it always has. I looked at: /opt/splunkforwarder/etc/system/local/inputs.conf on the syslog-ng server, and it still has the following entry:

[monitor:///var/log/network/avn/ise/...]
sourcetype = cisco:ise:syslog
index = network
blacklist = \.(gz|gz2)$

According to the Splunk Add-on for Cisco ISE release notes,

The Splunk Add-on for Cisco ISE automatically sets the source type for Cisco ISE records as cisco:ise:syslog, provided that all of the following are true:
Your Splunk platform is consuming syslog data through a syslog aggregator, or directly
You have configured your Cisco ISE devices to send logs via syslog to your aggregator, or directly to your Splunk platform instance
The Cisco ISE records include sourcetype=syslog

Thus, I went back to inputs.conf and changed the sourcetype to sourcetype=syslog and then restarted Splunk on the forwarder/syslog-ng box.

If I do a search against the network index for "ise", I am seeing lots of traffic for:

source = /var/log/network/avn/ise/2016.06.30
sourcetype = cisco:ise:syslog

Thus, I can tell that the ISE syslog data is being ingested into the indexer and the Splunk Add-on for Cisco ISE is recognizing it as ISE traffic as it is changing the sourctype from "syslog" to "cisco:ise:syslog". However, the Cisco ISE app itself shows "No results found" on all of its dashboards. Any idea what might be going on here? It looks like it should work from what the docs say.

1 Solution

Path Finder

I think I found the issue as all the dashboards have this error: Eventtype 'cisco-ise' does not exist or is disabled. I see the following eventtypes:
nix-all-logs
cisco-ise-system-statistics
cisco-ise-authentication
cisco-ise-guest-authentication-failed
nix_errors

cisco-ise-failed-authentication

No 'cisco-ise' eventtypes at all. I did this search " index=network eventtype=cisco-ise" and got zero results. It looks like the TA isn't tagging the ise traffic with the correct eventtype, or the ISE app itself isn't looking for the correct eventtype.

View solution in original post

New Member

Within the Splunk_CiscoISE App, Go to Settings -> Event Types
Create a new Event Type named "cisco-ise"
and its definition should be - sourcetype=cisco:ise:syslog eventtype=cisco-ise-*
Change the permissions to **Global* and RW privileges as needed
Now all the graphs should have data populated

0 Karma

Explorer

So, any updates? Issue is still present.

0 Karma

New Member

Within the Splunk_CiscoISE App, Go to Settings -> Event Types
Create a new Event Type named "cisco-ise"
and its definition should be - sourcetype=cisco:ise:syslog eventtype=cisco-ise-*
Change the permissions to **Global* and RW privileges as needed
Now all the graphs should have data populated

0 Karma

Path Finder

I think I found the issue as all the dashboards have this error: Eventtype 'cisco-ise' does not exist or is disabled. I see the following eventtypes:
nix-all-logs
cisco-ise-system-statistics
cisco-ise-authentication
cisco-ise-guest-authentication-failed
nix_errors

cisco-ise-failed-authentication

No 'cisco-ise' eventtypes at all. I did this search " index=network eventtype=cisco-ise" and got zero results. It looks like the TA isn't tagging the ise traffic with the correct eventtype, or the ISE app itself isn't looking for the correct eventtype.

View solution in original post

Path Finder

It appears I have been negligent in updating this post. The solution provided did fix the issue, but I had to do it on the search-head and not the indexer.

0 Karma

Splunk Employee
Splunk Employee

Hi jon.d.irish.ctr,

Thanks for reporting the issue, i worked on the update and will investigate your issues and findings and report back.

0 Karma

Splunk Employee
Splunk Employee

You're correct jon.d.irish.ctr the following code was removed from eventtypes.conf.

[cisco-ise] 
search = sourcetype=cisco:ise:syslog

if you haven't already done so i recommend adding this to an eventtypes.conf file in the local directory of the ISE Add-On directory.

Apologies for the inconvenience. We will update this error.

Don

Explorer

Do you know if this fix was ever applied to the application package?

0 Karma

Path Finder

Sorry about the delay in getting back to you. We lost the main mount point on the Splunk server and it has taken us awhile to recover everything. I went to my indexer as that is where I have the ISE TA installed. I verified that there was no eventtypes.conf file in the /opt/splunk/etc/apps/splunkTAcisco-ise/local directory. So I copied the one from /opt/splunk/etc/apps/splunkTAcisco-ise/default, changed the owner, group, and permissions to match the other files in the local directory, and then added the following to the beginning of the file:

[cisco-ise]
search = sourcetype=cisco:ise:syslog

I saved the file and restarted the splunk service. Next, I went back to the ISE app, and I am still getting "No results found." errors. The pan_logs index shows data is being ingested, so I am not entirly sure where the issue is.

0 Karma