Palo Alto Networks App and Add-on for Splunk: Why have dashboards stopped working?

[BrendanCO]

Hello. I've had some correspondence with your team but have not found a solution yet for my problem. Was wondering if we could try again.
Basically, my Splunk instance with this add-on is seeing all the PAN traffic coming in. Everything seems to be set up properly. I've got the data models accelerated and they appear to be normal:
Palo Alto Networks Firewall Logs
This data model represents all the syslogs produced by Palo Alto Networks Next-generation Firewalls and Panorama. The data model is structured such that all logs are part of the first root event object so the entire data model is accelerated.
MODEL
Datasets
17 Events Edit
Permissions
Shared in App. Owned by nobody. Edit
ACCELERATION
Rebuild Update Edit
Status
100.00% Completed
Access Count
110. Last Access: 4/17/17 11:21:47.000 AM
Size on Disk
1046.07MB
Summary Range
2678400 second(s)
Buckets
49
Updated
4/17/17 11:32:10.000 AM

I've rebuilt the data models, per prior instruction.

I'm at a point where one by one, all the dashboards just stop working. They worked perfectly when I first installed the add-on. I'm at a loss here. Could someone please help me? Splunk support will not help me with this and yes, I have the Enterprise version.

Thank you in advance.

Brendan

[adonio]

you are missing a tic (the button above the tab in keyboard) in both sides of pan_logs
$$$$ pan_logs sourcetype="pan:" log_subtype="" | timechart span=10s count by log_subtype $$$$

[BrendanCO]

I'm sorry that I'm missing something here. This is what I'm typing and this is what the error is:
http://imgur.com/npGdNCb

[adonio]

now you have 2 tics on the left and 1 on the right of pan_logs
remove one tic from the left. examples of macros here. this app (PAN relies on about 15 macros) want to check they work
http://docs.splunk.com/Documentation/Splunk/6.5.2/Knowledge/Definesearchmacros

[BrendanCO]

I believe I have it typed correctly now, but it returned zero results:

http://imgur.com/2fTKhRR

[BrendanCO]

I changed all the permissions, restarted splunk with user splunk and then went back in. I'm still getting a blank dashboard

[adonio]

ok,
yes seems like you got tome permission issues there, some parts run as root and some run as splunk. this is probably has to do with insatlling splunk as root or restarting as root and can be the source of the issue. it does seems like permission issue since you have the data and sourcetypes are extracted
try and chown -R splunk:splunk /opt/splunk (that will change all permissions to splunk user) then su to splunk and restart splunk as splunk user
after that you will probably want to enable boot-start as user splunk
described here: https://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/ConfigureSplunktostartatboottime
issue the command /opt/splunk/bin/splunk enable boot-start -user splunk

[BrendanCO]

Question. I have the $SPLUNK_HOME/var/run/dispatch linked to forward to another drive. It looks like this:
http://imgur.com/w0vW1hJ

The dispatch folder is where I believe searches are saved. Could this be causing an issue? Also, I see it's owner is root. Is that correct?

[BrendanCO]

Looks good:
http://imgur.com/5yQyhVy

[adonio]

ok, try now this one to check if eventtypes works:

eventtype="pan" sourcetype="pan:*" log_subtype="*" | timechart span=10s count by log_subtype

[BrendanCO]

Gotcha. I think I'm putting it in right?
http://imgur.com/AgWpwdJ

[adonio]

please add the tics "" so "" pan_logs"`"
had to add double quotes around the tics to demonstrate

[BrendanCO]

That doesn't seem to be getting any results:
http://imgur.com/3AFLyDy

[adonio]

lets go back to the macro attempt,
i see what was missing, it was the star between "" in log_subtype = "*"
lets try again with the macro search to make sure macros and eventtyppes work
pan_logs sourcetype="pan:" log_subtype="*" | timechart span=10s count by log_subtype
dont forget the tics around pan_logs

[BrendanCO]

OK, that one worked.
http://imgur.com/qlPZYzk

[adonio]

sourcetype=pan_* OR sourcetype=pan:* sourcetype="pan:*" log_subtype="*" | timechart span=10s count by log_subtype

[BrendanCO]

I think I have it right this time, but still no results.
http://imgur.com/dP7GIdL

Weird how copying from this site doesn't always paste correctly.

[BrendanCO]

You'll be happy to know that there are no more results in the palo logs when i restart Splunk.

index = _internal log_level = ERROR OR log_level = WARN paloalto

No new results! Just ones that triggered earlier

[adonio]

you need the star there:
sourcetype=pan_* OR sourcetype=pan:* sourcetype="pan:" log_subtype="" | timechart span=10s count by log_subtype


[BrendanCO]

Nothing for that search either:
http://imgur.com/LP9LDVC

[adonio]

how about this:
sourcetype=pan_* OR sourcetype=pan:* sourcetype="pan:" log_subtype="" | timechart span=10s count by log_subtype

[BrendanCO]

Hmm. Also nothin...