All Apps and Add-ons

Palo Alto Networks App and Add-on for Splunk: Why have dashboards stopped working?

BrendanCO
Path Finder

Hello. I've had some correspondence with your team but have not found a solution yet for my problem. Was wondering if we could try again.
Basically, my Splunk instance with this add-on is seeing all the PAN traffic coming in. Everything seems to be set up properly. I've got the data models accelerated and they appear to be normal:
Palo Alto Networks Firewall Logs
This data model represents all the syslogs produced by Palo Alto Networks Next-generation Firewalls and Panorama. The data model is structured such that all logs are part of the first root event object so the entire data model is accelerated.
MODEL
Datasets
17 Events Edit
Permissions
Shared in App. Owned by nobody. Edit
ACCELERATION
Rebuild Update Edit
Status
100.00% Completed
Access Count
110. Last Access: 4/17/17 11:21:47.000 AM
Size on Disk
1046.07MB
Summary Range
2678400 second(s)
Buckets
49
Updated
4/17/17 11:32:10.000 AM

I've rebuilt the data models, per prior instruction.

I'm at a point where one by one, all the dashboards just stop working. They worked perfectly when I first installed the add-on. I'm at a loss here. Could someone please help me? Splunk support will not help me with this and yes, I have the Enterprise version.

Thank you in advance.

Brendan

0 Karma

BrendanCO
Path Finder

It appears to be the same results:
http://imgur.com/MAW5nv2

Still nothing on the dashboards...

0 Karma

BrendanCO
Path Finder

That pan:newapps showed up on the main search towards the end, so they are in fact identical.

0 Karma

adonio
Ultra Champion

try debug refresh
on url http://localhost:8000/en-US/debug/refresh
then click on refresh button

0 Karma

BrendanCO
Path Finder

Looks like it didn't like admin/crl or admin/remote-indexes:

Refreshing admin/conf-times OK
Refreshing data/ui/manager OK
Refreshing data/ui/nav OK
Refreshing data/ui/views OK
Refreshing admin/alert_actions OK
Refreshing admin/clusterconfig OK
Refreshing admin/collections-conf OK
Refreshing admin/commandsconf OK
Refreshing admin/conf-deploymentclient OK
Refreshing admin/conf-inputs OK
Refreshing admin/conf-times OK
Refreshing admin/conf-wmi OK
Refreshing admin/cooked OK
Refreshing admin/crl ResourceNotFound Invalid action for this internal handler (handler: crl, supported: list|_reload, wanted: list).
Refreshing admin/datamodel-files OK
Refreshing admin/datamodelacceleration OK
Refreshing admin/datamodeledit OK
Refreshing admin/dataset_consolidation_datamodeleditOK
Refreshing admin/deploymentserver OK
Refreshing admin/distsearch-peer OK
Refreshing admin/eventtypes OK
Refreshing admin/fields OK
Refreshing admin/fifo OK
Refreshing admin/fvtags OK
Refreshing admin/http OK
Refreshing admin/indexes OK
Refreshing admin/limits OK
Refreshing admin/livetail OK
Refreshing admin/localapps OK
Refreshing admin/lookup-table-files OK
Refreshing admin/macros OK
Refreshing admin/manager OK
Refreshing admin/messages-conf OK
Refreshing admin/modalerts OK
Refreshing admin/monitor OK
Refreshing admin/nav OK
Refreshing admin/panels OK
Refreshing admin/passwords OK
Refreshing admin/pools OK
Refreshing admin/quickstart OK
Refreshing admin/raw OK
Refreshing admin/remote_eventlogs OK
Refreshing admin/remote_indexes BadRequest The following required arguments are missing: repositoryLocation.
Refreshing admin/remote_monitor OK
Refreshing admin/remote_perfmon OK
Refreshing admin/remote_raw OK
Refreshing admin/remote_script OK
Refreshing admin/remote_udp OK
Refreshing admin/savedsearch OK
Refreshing admin/scheduledviews OK
Refreshing admin/script OK
Refreshing admin/search-head-bundles OK
Refreshing admin/serverclasses OK
Refreshing admin/shclusterconfig OK
Refreshing admin/sourcetypes OK
Refreshing admin/splunktcptoken OK
Refreshing admin/ssl OK
Refreshing admin/syslog OK
Refreshing admin/tcpout-default OK
Refreshing admin/tcpout-group OK
Refreshing admin/tcpout-server OK
Refreshing admin/telemetry OK
Refreshing admin/transforms-extract OK
Refreshing admin/transforms-lookup OK
Refreshing admin/transforms-reload OK
Refreshing admin/udp OK
Refreshing admin/ui-prefs OK
Refreshing admin/ui-tour OK
Refreshing admin/views OK
Refreshing admin/viewstates OK
Refreshing admin/vix-indexes OK
Refreshing admin/vix-providers OK
Refreshing admin/webhook OK
Refreshing admin/workflow-actions OK
DONE

0 Karma

adonio
Ultra Champion

do dashboards populate?

0 Karma

BrendanCO
Path Finder

That's a negative:
http://imgur.com/6BhN8Jw

0 Karma

adonio
Ultra Champion

can you try and change the time picker to 15 min?

0 Karma

BrendanCO
Path Finder
0 Karma

BrendanCO
Path Finder

I don't know if this means anything but when tailing splunkd.log I see this:

04-17-2017 15:00:37.206 -0600 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/splunkdata/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
04-17-2017 15:00:37.207 -0600 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/splunkdata/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
04-17-2017 15:00:37.208 -0600 INFO DatabaseDirectoryManager - idx=main Writing a bucket manifest in hotWarmPath='/splunkdata/defaultdb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'

0 Karma

adonio
Ultra Champion

try and search: index = _internal log_level = WARN OR log_level = ERROR

0 Karma

BrendanCO
Path Finder

Ok, here's what I'm showing... I took 5 screen shots of logs. Being newer to Splunk, I was wondering how to query multiple log files from the console! Thanks for that trick. Looks like there's a BUNCH of stuff going on.

http://imgur.com/Mrt7Cdt
http://imgur.com/GmchHsN
http://imgur.com/NqRtDkq
http://imgur.com/L6wdL03
http://imgur.com/uUTMFs5

0 Karma

BrendanCO
Path Finder

Due to the error on transparent huge pages, I followed the instructions at this link to disable that...
https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

0 Karma

adonio
Ultra Champion

looks like some error with paloalto.py
can you search: index = _internal log_level = ERROR OR log_level = WARN paloalto
also, can you verify your PAN apps (app + TA) are set to global?

0 Karma

BrendanCO
Path Finder

Ok, I just changed SplunkforPaloAltoNetworks to Global. It was set on "App". Splunk_TA_paloalto was correctly set to global.

Running the search you have provided, here are the results:
http://imgur.com/Z7UCmm7

0 Karma

adonio
Ultra Champion

try and see if dashboards populate after set apps to global

0 Karma

adonio
Ultra Champion

also, fix the error, locate the deploy_splunk_ta_paloalto.py and move it to the correct directory: $SPLUNK_HOME/bin/scripts

0 Karma

BrendanCO
Path Finder

Ok, I copied the file to the correct directory. The dashboards are still not populating.

0 Karma

BrendanCO
Path Finder

I also just fixed the hard fd error, via this article:

https://splunkgeek.blogspot.com/2015/05/the-hard-fd-limit-is-lower-than.html

0 Karma

adonio
Ultra Champion

can you try and run this search: pan_logs sourcetype="pan:" log_subtype="" | timechart span=10s count by log_subtype`

0 Karma

adonio
Ultra Champion

pan_logs is a macro so add a tic "`" in each end of it

0 Karma

BrendanCO
Path Finder

I must be missing something. I put in:
`pan_logs sourcetype="pan:" log_subtype="" | timechart span=10s count by log_subtype`

and it says:
Error in 'SearchParser': You must provide a macro expression.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...