All Apps and Add-ons

Palo Alto Networks App for Splunk: Reading Palo Alto logs from a file, why are events not separated as distinct eventtypes?

craigkleen
Communicator

I've set up my PAN to send its logs to a syslog server, which has a Splunk lightweight forwarder running on it (6.3.3). The forwarder sends data up to an Indexer. I'm getting logs, but certain eventtypes don't seem to be getting fully translated. The logs look like:

Jul  7 10:43:43 pan-1 1,2016/07/07 10:43:43,010108001192,SYSTEM,general,0,2016/07/07 10:43:43,,general,,0,0,general,high,Commit job succeeded ,1968,0x0,0,0,0,0,,pan-1
Jul  7 10:43:14 pan-1 1,2016/07/07 10:43:14,010108001182,CONFIG,0,0,2016/07/07 10:43:14,10.0.0.1,,commit,000109354,Web,Submitted,,979,0x0,0,0,0,0,,pan-1

Shouldn't those two events be separate eventtypes? All I'm getting right now is eventtype=pan, so the app isn't distinguishing them. I'm running Splunk 6.3.3, Splunk_TA_paloalto 3.6.0, and SplunkforPaloAltoNetworks 5.1.0.

0 Karma
1 Solution

craigkleen
Communicator

Good grief. So, I went one step further and installed the Splunk_TA_paloalto app directly onto the forwarder. After, of course, I remembered that I'm not using UniversalForwarder on my syslog server (when it's almost a year between certain kinds of updates, you forget things). It's a HeavyForwarder. So, once that was added, I'm now getting logs tagged right. Silly.

Thanks for your help~

View solution in original post

0 Karma

craigkleen
Communicator

Good grief. So, I went one step further and installed the Splunk_TA_paloalto app directly onto the forwarder. After, of course, I remembered that I'm not using UniversalForwarder on my syslog server (when it's almost a year between certain kinds of updates, you forget things). It's a HeavyForwarder. So, once that was added, I'm now getting logs tagged right. Silly.

Thanks for your help~

0 Karma

panguy
Contributor

Ah ok. That makes sense yes. I only want you to use UDP so that we can narrow down the issue to be related to the forwarder or rsyslogd. I'm glad you got it working!

0 Karma

craigkleen
Communicator

Okay, going directly to the indexer seems to have worked. The format looks a little different:

Jul 8 14:34:21 10.0.0.1 1 2016-07-08T14:34:21-07:00 pan-1 - - - - 1,2016/07/08 14:34:21,010108001182,CONFIG,0,0,2016/07/08 14:34:21,10.0.0.2,,delete,panadmin,Web,Succeeded, deviceconfig system ntp-servers secondary-ntp-server,1013,0x0,0,0,0,0,,pan-1

There are 4 different timestamps in that message! Seriously, what a waste of splunk license space.

Thanks. It's not really my ideal solution, but I guess it'll have to do.

0 Karma

craigkleen
Communicator

Maybe the log is different? Our PAN-OS is 7.0.6.

0 Karma

panguy
Contributor

Your pan os is fine. it shouldn't affect anything.

0 Karma

panguy
Contributor

I can double check that. Can we try sending the logs directly to the indexer using UDP?

0 Karma

craigkleen
Communicator

On the Universal forwarder, the ~/etc/outputs.conf file looks like:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = 0
server = splunkindexer:9997

[tcpout-server://splunkindexer:9997]

0 Karma

craigkleen
Communicator

I don't know how it can get any more "raw" either. A tcpdump with -vv shows:

16:14:57.478070 IP (tos 0x0, ttl 61, id 26749, offset 0, flags [DF], proto UDP (17), length 2xx)
pan-1.58806 > syslogserver.syslog: [udp sum ok] SYSLOG, length: 235
Facility local0 (16), Severity info (6)
Msg: Jul 7 16:14:57 pan-1 1,2016/07/07 16:14:57,010108001182,CONFIG,0,0,2016/07/07 16:14:57,10.32.49.2,,delete,000109354,Web,Succeeded, deviceconfig system ntp-servers secondary-ntp-server,1004,0x0,0,0,0,0,,pan-1

So, the pre-pended date and hostname are there in the raw message. I don't see that rsyslog is modifying anything at all. The bracketed "134" is just a decimalization of the Facility and Severity.

0 Karma

panguy
Contributor

Thats as raw as it will get.

How does your output.conf look on the universal forwarder?

0 Karma

craigkleen
Communicator

The syntax for the template is actually:

$template RawLogTemplate,"%rawmsg%\n"

I did that, and now the log looks like:

<131>Jul 7 16:04:15 pan-1 1,2016/07/07 16:04:15,010108001182,SYSTEM,general,0,2016/07/07 16:04:15,,general,,0,0,general,high,Commit job succeeded for user username,3210,0x0,0,0,0,0,,pan-1

Still though, it's only being seen as sourcetype pan:log.

0 Karma

panguy
Contributor

in syslog-ng i also have to set a template which speicfies raw-message only.

You will need to set a template as well. Try this in rsyslog.conf:

$template, RawLogTemplate,"%rawmsg\n"
. /var/log/somefile.log;RawLogTemplate

http://man7.org/linux/man-pages/man5/rsyslog.conf.5.html#TEMPLATES

0 Karma

craigkleen
Communicator

Yes, input.conf is monitoring a file:

[monitor:///var/log/pan.log]
disabled = false
index = pan

Not using syslog-ng here, just rsyslogd that comes with RedHat. I don't think it would modify anything by default, would it?

I've removed the no_appending_timestamp. I'll see what it returns now. In a plain-vanilla search, just including "sourcetype=pan:log" returns data. But pan:system does not. Hrmmm

0 Karma

craigkleen
Communicator

Sorry, I missed a cut and paste. "sourcetype = pan:log" was part of the [monitor] stanza. Should that not be there, I guess?

0 Karma

panguy
Contributor

sourcetype=pan:log SHOULD be in the monitor stanza. It's possible the rsyslogd is changing the log which would explain why the logs are not getting parsed. I haven't done a setup with rsyslogd but when setting up syslog-ng I had the same issue because syslog-ng was parsing the log.

I don't have the answer to how to get rsyslogd to not parse the logs. You will need to find out if you can configure it to not parse logs.

0 Karma

craigkleen
Communicator

Without that sourcetype declared, it becomes "sourcetype=local" in my search...

0 Karma

craigkleen
Communicator

More info: When I restarted the UniversalForwarder on the syslog server, I get the following:

    Checking conf files for problems...
            Invalid key in stanza [monitor:///var/log/pan.log] in /opt/splunk/etc/apps/_server_app_paloalto/local/inputs.conf, line 5: no_appending_timestamp  (value:  true).

The indexer is acting at a deployment server.

0 Karma

panguy
Contributor

Hi,

Is your input.conf monitoring a log file?
The monitor stanza does not use the no_appending_timestamp. You can remove that. That should get rid of that error message.

Here is the documentation on setting up syslog-ng and universal forwarder:
http://pansplunk.readthedocs.io/en/latest/universal_forwarder.html

As for the eventtype issue. Take a look at the sourcetype. If you are getting sourcetype=pan:log then the parser is not parsing your logs. If you see sourcetype=pan:system then your logs are getting parsed.

Try the troubleshooting section here:
http://pansplunk.readthedocs.io/en/latest/troubleshoot.html

Paul

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...