All Apps and Add-ons

Splunk for Cisco Identity Services (ISE) dashboards show "no results found" after upgrading the Splunk Add-on for Cisco ISE

jon_d_irish_ctr
Path Finder

I have been trying to figure out why the Splunk App for Cisco ISE quit working after I updated the Splunk Add-on for Cisco ISE, but I am not having any luck. I have verified that Cisco ISE is still sending syslog data to our syslog-ng server, and that the syslog-ng server is still processing it as it always has. I looked at: /opt/splunkforwarder/etc/system/local/inputs.conf on the syslog-ng server, and it still has the following entry:

[monitor:///var/log/network/avn/ise/...]
sourcetype = cisco:ise:syslog
index = network
blacklist = \.(gz|gz2)$

According to the Splunk Add-on for Cisco ISE release notes,

The Splunk Add-on for Cisco ISE automatically sets the source type for Cisco ISE records as cisco:ise:syslog, provided that all of the following are true:
Your Splunk platform is consuming syslog data through a syslog aggregator, or directly
You have configured your Cisco ISE devices to send logs via syslog to your aggregator, or directly to your Splunk platform instance
The Cisco ISE records include sourcetype=syslog

Thus, I went back to inputs.conf and changed the sourcetype to sourcetype=syslog and then restarted Splunk on the forwarder/syslog-ng box.

If I do a search against the network index for "ise", I am seeing lots of traffic for:

source = /var/log/network/avn/ise/2016.06.30
sourcetype = cisco:ise:syslog

Thus, I can tell that the ISE syslog data is being ingested into the indexer and the Splunk Add-on for Cisco ISE is recognizing it as ISE traffic as it is changing the sourctype from "syslog" to "cisco:ise:syslog". However, the Cisco ISE app itself shows "No results found" on all of its dashboards. Any idea what might be going on here? It looks like it should work from what the docs say.

1 Solution

jon_d_irish_ctr
Path Finder

I think I found the issue as all the dashboards have this error: Eventtype 'cisco-ise' does not exist or is disabled. I see the following eventtypes:
nix-all-logs
cisco-ise-system-statistics
cisco-ise-authentication
cisco-ise-guest-authentication-failed
nix_errors

cisco-ise-failed-authentication

No 'cisco-ise' eventtypes at all. I did this search " index=network eventtype=cisco-ise" and got zero results. It looks like the TA isn't tagging the ise traffic with the correct eventtype, or the ISE app itself isn't looking for the correct eventtype.

View solution in original post

mangoadmin
New Member

Within the Splunk_CiscoISE App, Go to Settings -> Event Types
Create a new Event Type named "cisco-ise"
and its definition should be - sourcetype=cisco:ise:syslog eventtype=cisco-ise-*
Change the permissions to **Global* and RW privileges as needed
Now all the graphs should have data populated

0 Karma

m0ps
Explorer

So, any updates? Issue is still present.

0 Karma

mangoadmin
New Member

Within the Splunk_CiscoISE App, Go to Settings -> Event Types
Create a new Event Type named "cisco-ise"
and its definition should be - sourcetype=cisco:ise:syslog eventtype=cisco-ise-*
Change the permissions to **Global* and RW privileges as needed
Now all the graphs should have data populated

0 Karma

jon_d_irish_ctr
Path Finder

I think I found the issue as all the dashboards have this error: Eventtype 'cisco-ise' does not exist or is disabled. I see the following eventtypes:
nix-all-logs
cisco-ise-system-statistics
cisco-ise-authentication
cisco-ise-guest-authentication-failed
nix_errors

cisco-ise-failed-authentication

No 'cisco-ise' eventtypes at all. I did this search " index=network eventtype=cisco-ise" and got zero results. It looks like the TA isn't tagging the ise traffic with the correct eventtype, or the ISE app itself isn't looking for the correct eventtype.

jon_d_irish_ctr
Path Finder

It appears I have been negligent in updating this post. The solution provided did fix the issue, but I had to do it on the search-head and not the indexer.

0 Karma

dtregonning_spl
Splunk Employee
Splunk Employee

Hi jon.d.irish.ctr,

Thanks for reporting the issue, i worked on the update and will investigate your issues and findings and report back.

0 Karma

dtregonning_spl
Splunk Employee
Splunk Employee

You're correct jon.d.irish.ctr the following code was removed from eventtypes.conf.

[cisco-ise] 
search = sourcetype=cisco:ise:syslog

if you haven't already done so i recommend adding this to an eventtypes.conf file in the local directory of the ISE Add-On directory.

Apologies for the inconvenience. We will update this error.

Don

lacrosse1991
Explorer

Do you know if this fix was ever applied to the application package?

0 Karma

jon_d_irish_ctr
Path Finder

Sorry about the delay in getting back to you. We lost the main mount point on the Splunk server and it has taken us awhile to recover everything. I went to my indexer as that is where I have the ISE TA installed. I verified that there was no eventtypes.conf file in the /opt/splunk/etc/apps/splunk_TA_cisco-ise/local directory. So I copied the one from /opt/splunk/etc/apps/splunk_TA_cisco-ise/default, changed the owner, group, and permissions to match the other files in the local directory, and then added the following to the beginning of the file:

[cisco-ise]
search = sourcetype=cisco:ise:syslog

I saved the file and restarted the splunk service. Next, I went back to the ISE app, and I am still getting "No results found." errors. The pan_logs index shows data is being ingested, so I am not entirly sure where the issue is.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...