Syslog traffic not showing in Splunk Search and Reporting

[BrendanCO]

Hello. I've set up a few Palo to Splunk instances in the past. I've never had a problem getting a syslog feed from the Palo to Splunk, port 514. Everything is set up on the FW correctly. When I tcpdump port 514, I see the traffic trying to come in but when I go to Search for source="udp:514", nothing is showing up.

inputs.conf
connection_host = ip
sourcetype = pan:log
no_appending_timestamp = true
index = paloalto
disabled = 0

UFW status
To Action From

---

22/tcp ALLOW Anywhere
514 ALLOW Anywhere
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
514/udp ALLOW Anywhere
8000 ALLOW Anywhere
21/tcp ALLOW Anywhere
5514/udp ALLOW Anywhere
22/tcp (v6) ALLOW Anywhere (v6)
514 (v6) ALLOW Anywhere (v6)
80/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
514/udp (v6) ALLOW Anywhere (v6)
8000 (v6) ALLOW Anywhere (v6)
21/tcp (v6) ALLOW Anywhere (v6)

I'm sure I'm missing something stupid here. Any thoughts? I went back to the install guide and checked off every box. Confused.

[BrendanCO]

It's running as splunk. I know that I've seen this recommendation before, but I never had an issue with it. The traffic is local and it's a small office so I wasn't too concerned about any security risk.
I actually am now seeing some events in the Palo dashboards but nothing still in Search.

So, I will try another port and let's see if that does the trick. Weird that the dashboard is now seeing some traffic tho...

[richgalloway]

If you see data in the PA app, but not in the Search & Reporting app then there may be permissions issues. Verify all of the knowledge objects you use in your search are exported globally.

[BrendanCO]

Progress! I put in the Search field "index=paloalto" and now I see data. Strange that just searching for just "*" doesn't show this data. Dashboards still not populating...

[richgalloway]

Examine the dashboards to ensure the assumptions the author made are valid for your environment.

[BrendanCO]

So I've swapped the port to 5514, updated my inputs.conf. I restarted the server and when I tcpdump port 5514, I see the traffic coming in. I then go to the Search app and look for source="udp:5514" and I get nothin...
Checked the index paloalto. It's set up an accelerated. I added the data input in the GUI as well (not sure if this is redundant). Hmmmmm.

[richgalloway]

What user is Splunk running as? If it's not root then it won't be able to open port 514.
Sending syslog directly to Splunk is not Best Practice. See http://www.georgestarcher.com/splunk-success-with-syslog/