Splunk App for Enterprise Security: How to fix Palo Alto sourcetype naming conflict that is causing empty dashboards?

[hopnscotch]

This is a new install of ES (a few months old) that was added to an existing base Splunk instance. All of the web and proxy related dashboards are empty.

According to this, http://docs.splunk.com/Documentation/ES/latest/CreateTA/Out-of-the-boxsourcetypes, the sourcetype for the PA logs SHOULD be "pan:threat", "pan:traffic", etc, however they are named with an underscore instead of the colon: "pan_threat", "pan_traffic", etc.

The PA logs have existed in the system for a while so I'm not sure how they got named, but looking at questions around the Palo Alto app it looks like this app gives them the underscore version. Since this app is part of ES, it seems to be a conflict. I have tried tagging the existing sourcetype "pan_threat" with web and proxy, but the dashboards are still empty.

1. What is the best way to workaround this?
   
2. Do I need to manually override the sourcetype name (if so, I lose access to previous data?)?
3. (or) Will a simple "rename" fix it?
4. Also, where should this be done (at the indexer, or in ES)?

Or is this some other issue altogether?

Thanks

[esix_splunk]

Another solution is to change the PAN eventtypes on the ES SH. In the latest version they are actually looking for pan:* AND pan_*.. Check your eventypes. This would be the simplest fix. This redress your sourcetype naming, the new "standard" is type:subtype..

[joe_kraxner · Accepted solution]

The Splunk for Palo Alto Networks App (https://apps.splunk.com/app/491/) wants to sourcetype the PAN logs as "pan_log" but this conflicts with what ES is requiring just "pan". Digging into the ES TA-paloalto I see a reference for pan_log and "Added per SOLNESS-2728 for compatibility with SplunkforPaloAltoNetworks app". One could modify the apps, it's your warranty. 🙂

Now as far as your questions on sourcetyping:

What is the best way to workaround this?

Since you can't change the data in Splunk once it's written to disk (index), your previous data is stuck with that sourcetype. One caveat is if you have the data you can re-index if your kick the Fishbucket for the file (http://docs.splunk.com/Documentation/Splunk/6.2.0/Troubleshooting/CommandlinetoolsforusewithSupport#...).

Do I need to manually override the sourcetype name (if so, I lose access to previous data?)?

I make it a bit easy on myself since I have several other syslog data sources and use rsyslog with a fairly static configuration (see below). You will not lose the old data, it will still have the same previously written sourcetype.

(or) Will a simple "rename" fix it?

I'm not sure I know the exact answer to this. From my experience, the ES app includes the necessary props and transforms that require to be sourcetyped as "pan" or "pan_log" so the pan:traffic and pan:threat sub-sourcetyping can be performed. This is done at the input phase.

Also, where should this be done (at the indexer, or in ES)?

Preferribly applying the sourcetype is done at input time (usually a UF or whereever you're specifying the inputs.conf).

My setup:
I am using rsyslog [1] to collect my PAN firewall logs for ES and having them written to a location such as the following:

/data/splunk/logs/syslog/pan/acme-fw-01/2014-12-17_21.log

Config:

#rsyslog config snippet: /etc/rsyslog.d/splunk.conf
# Template
$template PAN,"/data/splunk/logs/syslog/pan/%fromhost%/%$year%-%$month%-%$day%_%$HOUR%.log"
# Palo Alto Networks Firewall
if $fromhost-ip == '10.0.4.1' or $fromhost-ip == '10.0.5.1' then -?PAN
& ~

Then I configure an input such as the following to force the sourcetype as "pan":

#From inputs.conf
[monitor:///data/splunk/logs/syslog/pan/]
sourcetype=pan
index=firewalls
host_segment=6
whitelist = \.log$

Hope this helps.

[jcoates_splunk]

there's two places making this decision -- TA-paloalto is in ES, and the Palo Alto App is not. One should do a transforms.conf sourcetype rename from the sourcetype set by the other.