All Apps and Add-ons

Splunk for Palo Alto Networks 4.2: Data indexed, but how to troubleshoot why are dashboards still empty?

SCSM_Martin
Explorer

Hi,

I just did a brand new installation of Splunk Enterprise 6.2 on FreeBSD 9.3 and installed the Palo Alto app version 4.2. I followed the instructions for the installation and was able to ran successfully the search commands to validate my installation. "index=pan_logs sourcetype=pan_config" and "index=pan_logs" returns me all the logs from the FW. I looked in Settings--Data Model and the Firewall Logs acceleration is 100% completed.

What I find strange is that when I look at the "Data Summary" in the Search tab, I don't see anything related to Palo Alto. I can only see the other data that was injected in Splunk (CSV files and syslog which are both working fine). Is it normal? What should I be looking at to troubleshoot further?

Thank you!

Martin,

0 Karma
1 Solution

SCSM_Martin
Explorer

I finally found the problem. NTP wasn't working correctly because of firewall rules, so the system time of the Splunk server was wrong. I soon as I had NTP synced up, the dashboards started to work correctly. The system time was about 5 hours off, so that screwed up the dashboards that are using last 60 minutes by default.

Thanks for your help!

View solution in original post

SCSM_Martin
Explorer

I finally found the problem. NTP wasn't working correctly because of firewall rules, so the system time of the Splunk server was wrong. I soon as I had NTP synced up, the dashboards started to work correctly. The system time was about 5 hours off, so that screwed up the dashboards that are using last 60 minutes by default.

Thanks for your help!

neelamssantosh
Contributor

Hi,
I too was facing the same concern
Data is showing up in Search but not in dashboard.

After reading few Splunk community answers,
instead of upgrading directly, download the app and place it it in /etc/apps and restart the machine.

yet to work on few data model, to render the results in Dashboard.

0 Karma

SCSM_Martin
Explorer

I ran "index=pan_logs|head 10" and got events listed in local time. I think the problem is related to timezones because I'm GMT -5 (Eastern) and when I search for events in the last 5 hours, I only get events for the current minute (223 events (12/9/14 4:20:41.000 PM to 12/9/14 9:20:41.000 PM). It is currently 4:20 PM. Is there a way to correct the timezone behaviour? Thank you!

SCSM_Martin
Explorer

Thanks for taking the time to help me with this issue. I think the problem lies with relative dates. When I click on the dashboard to open the query in the Search tab, I don't have any result. But, when I change the time definition from "Last 60 minutes" to "All time", I have results! I have the same problem with a simple "index=pan_logs" query. As soon as I choose a "Last X minutes" query, I get no results. Any clue on what can be causing this? I'm a Splunk newb, but really eager to learn.

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

I guess the question is whether there is actually a delay (so relative time has to accommodate that) or if it's something weirder... so rather than all time, take a look at the most recent stuff |head 10

use the "relative" bar on the time picker... or try "Today" or "Yesterday" or "last week" etc...

I'm wondering if you've got a server time and local time that are throwing things off... (meaning... the server time is off so Splunk, which assumes epoch time and then displays local is being hamstrung)
Once you see the timestamp of the most recent stuff you'll get an idea of exactly what's playing tricks on you.
If you're reading in archive data... you might still be in last month, you know?
I had that scenario with a customer and we had to wait for it to catch up to be able to use the relative time attributes.
I've also had situations where someone set server time to local time... meaning the timestamp shows up in epoch time but relative to their own timezone rather than GMT. which then makes it "Garbage" rather than "Epoch" since Epoch is relative to GMT or nothing. Think outside the box. 🙂

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

SCSM_Martin
Explorer

Hi! Unfortunately, I'm still in the same state. I can see the logs in the Search tab, but all dashboards are still blank (0 or No results found). Thanks for your help.

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

in the doc (the documentation tab on the app page) there is a link to a troubleshooting doc. here

There is a section that talks about the troubleshooting the dashboards...
I see a couple of nuances there you might have missed.

One thing you can try is to deconstruct the panels... I have the app installed but no data coming in. The way the dashboards are created I can still hover at the lower left hand side of the panel and make the little magnifying glass appear. Click on that and it will show you the search in the search dash. note things like the time picker and whether there is a macro that is pointing to something you don't have...
The panels are expecting the field extractions to be working a certain way. perhaps you've got data but there is something slightly off so the dashboards can't find the fields?

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Customers have shared that it sometimes takes about 24 hours for the Accelerated Data Model to fully "form". Are you now getting data?

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...