Issue is that none of the dashboards (except Overview) is populating with data. I have reviewed all of the help articles I could find as well as the official Troubleshooting page. What I have done:
NOTE: we are using Splunk cloud
Ensured correct versions - Add-on is version 3.7.1 and App is 5.3.1 (these are compatible)
Checked the data model - I rebuilt the 2 data models that we have which are "Palo Alto Networks Firewall Logs" and "Palo Alto Networks WildFire Malware Reports". They are at 100% and I have used the "pivot" feature and I get search results.
The search macro pan_logs populates events and uses the eventtype="pan".
All logs appear to be parsed correctly.
These searches also return results:
| tstats summariesonly=t count FROM datamodel="pan_firewall"
| tstats summariesonly=t count FROM datamodel="pan_firewall" GROUPBY nodename
I do not know what the problem could be. I did not originally setup the app and I don't think it ever worked properly. Since the 'Overview' dashboard does work, I originally thought it was a data model problem but those seem to be working fine.
Most of the dashboards work fine now but having an issue with URL filtering (setting Palo Alto inputs.conf to TZ=UTC resolved most of the issues). Going to ask an additional question since much has changed since I originally asked this one.
I will add that I am currently waiting on Splunk support to change the props.conf file on the indexers so that the time zones properly match. The issue is that our firewall logs are in UTC and Splunk is seeing them as EDT. This makes it so the only way to search for current logs is to force a search 4 hours into the future (we are UTC -4).
Would this be the issue as to why our dashboards are not populating? I figured that they would still populate, but simply have data that is behind by 4 hours.
Changes to the props.conf file did fix our time issue but the dashboards still are not populating.
I have also tried to open some of the searches to see the SPL, but clicking on the magnifying glass does nothing. I get no error and no additional window with the search.
However, when I inspect the search, I get an error stating "Unknown sid." I have let the screen run for as long as 20 minutes and still nothing populates (I throw in a source IP which shows up in the "Threat" logs and should show up in the dashboard). I have already tried rebuilding the data models multiple times and those appear to be working correctly (we have the Endpoint Logs, Firewall Logs, and WildFire and all are at 100%).