Palo Alto Networks App for Splunk: Reading Palo Alto logs from a file, why are events not separated as distinct eventtypes?

[craigkleen]

I've set up my PAN to send its logs to a syslog server, which has a Splunk lightweight forwarder running on it (6.3.3). The forwarder sends data up to an Indexer. I'm getting logs, but certain eventtypes don't seem to be getting fully translated. The logs look like:

Jul  7 10:43:43 pan-1 1,2016/07/07 10:43:43,010108001192,SYSTEM,general,0,2016/07/07 10:43:43,,general,,0,0,general,high,Commit job succeeded ,1968,0x0,0,0,0,0,,pan-1
Jul  7 10:43:14 pan-1 1,2016/07/07 10:43:14,010108001182,CONFIG,0,0,2016/07/07 10:43:14,10.0.0.1,,commit,000109354,Web,Submitted,,979,0x0,0,0,0,0,,pan-1

Shouldn't those two events be separate eventtypes? All I'm getting right now is eventtype=pan, so the app isn't distinguishing them. I'm running Splunk 6.3.3, Splunk_TA_paloalto 3.6.0, and SplunkforPaloAltoNetworks 5.1.0.

[craigkleen · Accepted solution]

Good grief. So, I went one step further and installed the Splunk_TA_paloalto app directly onto the forwarder. After, of course, I remembered that I'm not using UniversalForwarder on my syslog server (when it's almost a year between certain kinds of updates, you forget things). It's a HeavyForwarder. So, once that was added, I'm now getting logs tagged right. Silly.

Thanks for your help~

[panguy]

Ah ok. That makes sense yes. I only want you to use UDP so that we can narrow down the issue to be related to the forwarder or rsyslogd. I'm glad you got it working!

[craigkleen]

Okay, going directly to the indexer seems to have worked. The format looks a little different:

Jul 8 14:34:21 10.0.0.1 1 2016-07-08T14:34:21-07:00 pan-1 - - - - 1,2016/07/08 14:34:21,010108001182,CONFIG,0,0,2016/07/08 14:34:21,10.0.0.2,,delete,panadmin,Web,Succeeded, deviceconfig system ntp-servers secondary-ntp-server,1013,0x0,0,0,0,0,,pan-1

There are 4 different timestamps in that message! Seriously, what a waste of splunk license space.

Thanks. It's not really my ideal solution, but I guess it'll have to do.

[craigkleen]

Maybe the log is different? Our PAN-OS is 7.0.6.

[panguy]

Your pan os is fine. it shouldn't affect anything.

[panguy]

I can double check that. Can we try sending the logs directly to the indexer using UDP?

[craigkleen]

On the Universal forwarder, the ~/etc/outputs.conf file looks like:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = 0
server = splunkindexer:9997

[tcpout-server://splunkindexer:9997]

[craigkleen]

I don't know how it can get any more "raw" either. A tcpdump with -vv shows:

16:14:57.478070 IP (tos 0x0, ttl 61, id 26749, offset 0, flags [DF], proto UDP (17), length 2xx)
pan-1.58806 > syslogserver.syslog: [udp sum ok] SYSLOG, length: 235
Facility local0 (16), Severity info (6)
Msg: Jul 7 16:14:57 pan-1 1,2016/07/07 16:14:57,010108001182,CONFIG,0,0,2016/07/07 16:14:57,10.32.49.2,,delete,000109354,Web,Succeeded, deviceconfig system ntp-servers secondary-ntp-server,1004,0x0,0,0,0,0,,pan-1

So, the pre-pended date and hostname are there in the raw message. I don't see that rsyslog is modifying anything at all. The bracketed "134" is just a decimalization of the Facility and Severity.

[panguy]

Thats as raw as it will get.

How does your output.conf look on the universal forwarder?

[craigkleen]

The syntax for the template is actually:

$template RawLogTemplate,"%rawmsg%\n"

I did that, and now the log looks like:

<131>Jul 7 16:04:15 pan-1 1,2016/07/07 16:04:15,010108001182,SYSTEM,general,0,2016/07/07 16:04:15,,general,,0,0,general,high,Commit job succeeded for user username,3210,0x0,0,0,0,0,,pan-1

Still though, it's only being seen as sourcetype pan:log.

[panguy]

in syslog-ng i also have to set a template which speicfies raw-message only.

You will need to set a template as well. Try this in rsyslog.conf:

$template, RawLogTemplate,"%rawmsg\n"
. /var/log/somefile.log;RawLogTemplate

http://man7.org/linux/man-pages/man5/rsyslog.conf.5.html#TEMPLATES

[craigkleen]

Yes, input.conf is monitoring a file:

[monitor:///var/log/pan.log]
disabled = false
index = pan

Not using syslog-ng here, just rsyslogd that comes with RedHat. I don't think it would modify anything by default, would it?

I've removed the no_appending_timestamp. I'll see what it returns now. In a plain-vanilla search, just including "sourcetype=pan:log" returns data. But pan:system does not. Hrmmm

[craigkleen]

Sorry, I missed a cut and paste. "sourcetype = pan:log" was part of the [monitor] stanza. Should that not be there, I guess?

[panguy]

sourcetype=pan:log SHOULD be in the monitor stanza. It's possible the rsyslogd is changing the log which would explain why the logs are not getting parsed. I haven't done a setup with rsyslogd but when setting up syslog-ng I had the same issue because syslog-ng was parsing the log.

I don't have the answer to how to get rsyslogd to not parse the logs. You will need to find out if you can configure it to not parse logs.

[craigkleen]

Without that sourcetype declared, it becomes "sourcetype=local" in my search...

[craigkleen]

More info: When I restarted the UniversalForwarder on the syslog server, I get the following:

    Checking conf files for problems...
            Invalid key in stanza [monitor:///var/log/pan.log] in /opt/splunk/etc/apps/_server_app_paloalto/local/inputs.conf, line 5: no_appending_timestamp  (value:  true).

The indexer is acting at a deployment server.

[panguy]

Hi,

Is your input.conf monitoring a log file?
The monitor stanza does not use the no_appending_timestamp. You can remove that. That should get rid of that error message.

Here is the documentation on setting up syslog-ng and universal forwarder:
http://pansplunk.readthedocs.io/en/latest/universal_forwarder.html

As for the eventtype issue. Take a look at the sourcetype. If you are getting sourcetype=pan:log then the parser is not parsing your logs. If you see sourcetype=pan:system then your logs are getting parsed.

Try the troubleshooting section here:
http://pansplunk.readthedocs.io/en/latest/troubleshoot.html

Paul