topic Re: How to deploy the Palo Alto app in an Indexer Cluster environment in All Apps and Add-ons

How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Tue, 29 Sep 2020 07:29:54 GMT

Hey everyone,

I'm having trouble deploying the Palo Alto Networks app (4.2.2) in Splunk Enterprise (6.2.2). The setup is 1x Search Head, 1x Cluster Master, 2x Indexers, receiving data from a separate Universal Forwarder that reads off a directory populated by syslog-ng.

The Palo Alto app was deployed as a Distributed Configuration Bundle from the Search Head, and I saw it to be successfully deployed against the 2x indexers.

The Universal Forwarder has a input.conf stanza for the PAN data with:
index = pan_logs
sourcetype = pan_log

Data is coming into the system, and when searching (from Search Head):
index=pan_logs sourcetype=pan_log (shows every event)
index=pan_logs sourcetype=pan_config (shows no events)

In fact, I can see only one sourcetype in that index: pan_log, so it is not getting correctly parsed. I tried loading the syslog-ng data in my local laptop running the PAN app and it worked fine, as in, the sourcetype fields populate correctly. That means the data coming out of syslog-ng is correct.

I can also see the /slave-apps/ and /master-apps/ directories replicated correctly in the indexers. I haven't modified the transforms.conf or props.conf files, but I can see they are there, and contain the necessary rules to correctly assign the event's source type.

I think that for some reason, the transforms.conf and props.conf for the PAN app is not getting picked up by the indexers, thus not getting the correct sourcetypes.

I'm at a loss on how to troubleshoot this further. Any ideas would be greatly appreciated.

Dan.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Sun, 11 Oct 2015 02:25:10 GMT

Could it be that in a distributed, clustered environment I need to setup SplunkforPaloAltoNetwork/metadata/default.meta permissions to global rather than none?

I didn't have to do that in the all-in-one deployment, in which they all appear as none by default.

Default:

###PROPS
 [props]
 export = none
 ###TRANSFORMS
 [transforms]
 export = none
 [lookups]
 export = none

Proposed Change:

###PROPS
 [props]
 export = system
 ###TRANSFORMS
 [transforms]
 export = system
 [lookups]
 export = system

As a test in the customer environment, I'm planning to execute the change on the Search Head, as well as on the Cluster Master, Restarting both the Search Head and Cluster Master after having edited the default.meta file.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

rtoloczk — Tue, 13 Oct 2015 17:12:18 GMT

David,

Did this resolve your issue?

Thanks,

Robert

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Wed, 14 Oct 2015 05:12:02 GMT

It didn't. Developing another theory now...

The forwarders are full Splunk instances, as in, Heavy Forwarders.

I wonder whether the app must be installed in the forwarder only in the case of a heavy forwarder. I'm planning to test that next.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Wed, 14 Oct 2015 07:10:27 GMT

The theory now is that given the forwarder is a Heavy Forwarder, the app needs to be installed in the Forwarder as well.

This link (http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F) explains how both the Input and Parsing stages are performed in the Forwarder - when using a Heavy Forwarder:

Heavy Forwarder → Indexer
Input, Parsing → Indexing, Search

And how transforms.conf is looked at during the Parsing phase:

Parsing
- props.conf
- transforms.conf
- datetime.xml

and not during the Indexing stage:

Indexing
- props.conf
- indexes.conf
- segmenters.conf
- multikv.conf

Trying this next... will update.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

esix_splunk — Wed, 14 Oct 2015 07:21:03 GMT

The PAN app works at Search Time to break out all the different sourcetypes. These are search time transforms. As long as your data is getting indexed properly and pan_logs.

So it does depend on your ingest framework. If you're ingesting via UDP directly on indexers, you need the inputs and the props/transforms from the App.

If you're ingesting via syslog on a UF, you dont need the app, just set the sourcetype to pan_log. The indexers will need the APP though.

If you're ingesting via syslog / UDP on a HF, you need the props/transforms, same as an indexer.

Then on the SH, you also need the App. The recognition of the pan:traffic|system|threat|log|config sourcetypes are done at search time, not index time.

index=pan_logs earliest=-1h@h | stats count by sourcetype

Assuming you're using the standard configuration, that search should return the different groups of PAN sourcetypes.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Wed, 14 Oct 2015 07:27:24 GMT

Hi Eric,

So, just to clarify, when you say: "If you're ingesting via syslog / UDP on a HF, you need the props/transforms, same as an indexer."

Do you mean that one would need the Palo Alto App in the Search Head, Indexers, as well as the Heavy Forwarder? My previously proposed answer has my thinking on the why.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

esix_splunk — Wed, 14 Oct 2015 07:30:24 GMT

If you're ingesting on a HF, then you need the app there, and don't need it on the indexers.

That's not always the case however, but in this case, the HF is cooking the data and forwarding to the indexers with the relevant index time fields and metadata.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

rtoloczk — Tue, 29 Sep 2020 07:34:05 GMT

Thank you both.

However, I am still not getting the correct sourcetypes.

I have my PAs sending their syslogs to a Syslog-NG server with a UF. The UF's inputs.conf sets the index to pan_logs and the sourcetype to pan_log. Both the Search Heads and the Indexers have the app installed.

The standalone (non-clustered) Indexer is reassigning the correct sourcetypes (pan_traffic, pan_config, etc), however the indexers in the cluster are not. I only see the original set pan_log.

Is there a different required config for a clustered instance?

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

btorresgil — Tue, 29 Sep 2020 07:37:14 GMT

Hello,

The app needs to be installed on all searchheads, indexers, and heavy forwarders. Since the sourcetype for the events is pan_log, it means the events are not getting parsed by the app. 9 times out of 10 this is because the logs have been subtly modified by Syslog-NG so the props/transforms cannot recognize them. So...

Make sure the app is installed on all necessary Splunk nodes
Verify syslog-ng isn't adding any characters to the logs or modifying them in any way

Hope that helps!

Update: an earlier version of this answer said the sourcetype was pan_logs, but it is pan_log. This has been corrected. (thanks mbonsack)

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

mbonsack_splunk — Tue, 29 Sep 2020 07:34:13 GMT

Is the sourcetype pan_log or pan_logs(plural)? This is very confusing and could be the source of the problem. The index is pan_logs, correct?

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

btorresgil — Wed, 14 Oct 2015 17:15:56 GMT

The sourcetype is pan_log

The index is pan_logs

I initially typed it wrong in my answer, but corrected it now. Thanks for the catch.

Yes, this is confusing, and it will be changed in the next version of the app (version 5.0) coming out soon.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

rtoloczk — Tue, 29 Sep 2020 07:34:16 GMT

The sourcetype is set to pan_log . The following is the inputs.conf stanza for my syslog-ng server

Push PaloAlto Syslog to Splunk:

[monitor:///var/log/syslog-ng/paloalto]
disabled = false
index = pan_logs
sourcetype = pan_log
no_appending_timestamp=true
host_segment = 5

The architecture is now:
- PA sends syslog to syslog-ng server with UF
- UF forwards to indexer cluster load balancing between three indexers
- Each indexer has the app in the $SPLUNK_HOME/etc/slave-apps/_cluster directory
- The search head has the app in $SPLUNK_HOME/etc/apps

Our 2nd site with just PA --> UF --> Standalone indexer <-- Search Head parses the logs just fine with the correct sourcetypes.

Our primary site did have a stand alone indexer and it worked. When we changed to the indexer cluster is when the parsing/sourcetyping stopped working.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

rtoloczk — Wed, 14 Oct 2015 17:54:00 GMT

As a test, I copied the app from .../etc/slave-app/_cluster to /etc/apps on one of the clustered indexers and rebooted the box.

The sourcetyping is now correct for indexer02, but incorrect on indexer01 and indexer03

There appears to be something unique with the clustered indexer setup

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

mbonsack_splunk — Wed, 14 Oct 2015 17:56:12 GMT

Permissions?

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

rtoloczk — Wed, 14 Oct 2015 18:07:49 GMT

The app in both /etc/apps and /etc/slave-apps have the same OS level permissions and the same Splunk app permissions as listed below

Application-level permissions

[]
access = read : [ * ], write : [ admin, power ]

EVENT TYPES

[eventtypes]
export = system

PROPS

[props]
export = system

TRANSFORMS

[transforms]
export = system

[lookups]
export = system

OTHER

[savedsearches]
export = none

[commands]
export = system

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Thu, 15 Oct 2015 06:23:09 GMT

Installed the app in SH, Cluster Master, and Heavy Forwarders and the thing started working as expected. So copying the app folder into the HF and restarting did the job.

A little confused though... All data seemed to be parsed into the right sourcetypes now, including last week's data, which hadn't been parsed correctly. I was expecting only today's data to be parsed correctly. (Not that I'm complaining! ha!, but would like to understand why..)

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

esix_splunk — Thu, 15 Oct 2015 07:22:08 GMT

The App applies field extractions and parsing at Search Time, not index time. This is why historical data is working correctly.

Where on the CM did you put this?

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

esix_splunk — Thu, 15 Oct 2015 07:22:55 GMT

Configuration for clustered indexers needs to be applied via the cluster bundles under $splunk_home/etc/master-apps/.

Re: How to deploy the Palo Alto app in an Indexer Cluster environment

dmartinez_splun — Thu, 15 Oct 2015 10:44:25 GMT

Thanks Eric,

It's in the /master-apps/ in the CM, but it was pushed there via the UI, using the Distributed Configuration Bundle from the Search Head. It's replicated correctly in the /slave-apps/ directories.