Setup assistance - Panorama to UF with syslog-ng to Splunk Enterprise at AWS

[BrendanCO]

Hi all! I've always had a pretty straight forward approach to bringing in my Palo logs straight to an on-prem Search Head / Indexer just via port 514 / syslog. That's a pretty straight forward setup.

I'm trying to set up the more recommended way, now that my Splunk Search Head / Indexer is hosted at AWS. SO, I set up a Universal Forwarder on an Ubuntu Server on the same network as my Panorama instance and am sending the Panorama syslog feed to the UF - running syslog-ng. I see those log files coming in and saving to /var/log/udp514.log. I set up the UF to connect as a forwarder to the Splunk instance on port 9997 and have added /var/log/514.log to be monitored via "./splunk/add monitor /var/log/udp514.log". I see that logging on the UF and then coming in to Splunk, but it is all logging to the main index. 

Logically I know that either on the UF or on the Splunk indexer I need to use the PA app to tell it to log to my paloalto index, but I don't know where. I can't seem to add a data input to also listen on 9997. That seems to be a conflict and my normal method of looking for logs on port 514 doesn't apply anymore... 

So how do I tell my Splunk indexer to look at the stream coming in on 9997 and move the logs associated with Palo Alto over where the app is looking for it (index=paloalto)? I'll also be logging much more to the UF soon as well...

[johnhuang]

It's possible that syslog-ng is modifying the raw logs. What sourcetype is it showing up as?

index=* sourcetype=pan*
| stats count as event_count by index source sourcetype

Your _raw logs should begin with:

Jan 28 19:37:41....................

[BrendanCO]

Hi Johnhua. Here's what I get from that search. 

[PickleRick]

As far as splunk is concerned, the input is the logfile (/var/log/514.log). So in the definition of this input, on your UF you have to specify the proper index and sourcetype to which the events should go to.

It's the input phase when you decide (unless you override it further down the pipeline with some transforms) which index the data goes to. If you don't specify the index, splunk assumes the default main index.

[BrendanCO]

Ok, that makes sense. So I need to modify the default inputs.conf on the UF it sounds like. I've looked for examples and am having trouble modifying for this situation. 

[PickleRick]

I don't know what you mean by "default inputs" but as a general rule you shouldn't touch anything in directories named default (either system/default or apps/<app>/default). Those settings will get overwritten with upgrade, app redeployment and so on. You should define your settings in local.

In your case, this whole input stanza should be created either in system/local/inputs.conf or you should do a separate app for it (and that's the good practice i  case of a bigger managed configuration so you have separate pieces of config in separate apps and you can manage them independently) - create apps/my_forwarded_syslog_events_input (or however you want to call it), within this directory create a directory named local and there put your inputs.conf.

[SinghK]

apologies for incorrect info. I think I just mixed up 2 posts.

[BrendanCO]

Hi. On the Splunk Indexer / Search Head:

/opt/splunk/etc/apps/Splunk_TA_paloalto/local# more inputs.conf

[splunktcp:9997]
sourcetype = pan:firewall
index = paloalto
disabled = 0

Everything is still going in to the main index. 

[BrendanCO]

On the UF's inputs.conf, or the Splunk Instance? Default one or in the Palo App?

[SinghK]

You can create a inputs.conf in paloaltos ta on splunk forwarder in  local folder. if there is no inputs.conf create one and on Linux  file  should be owned by splunk user  and put this there restart splunk service once and you are all set.

[BrendanCO]

Ok, so on the Universal forwarder I need to install the Splunk app? This is confusing because the installation guide says to not install it on the UF...

https://splunk.paloaltonetworks.com/installation.html

Just in case I misunderstood, I changed the inputs.conf on the Splunk indexer/search head. This didn't prove successful. 

/opt/splunk/etc/apps/Splunk_TA_paloalto/local# more inputs.conf

[splunktcp:9997]
sourcetype = pan:firewall
index = paloalto
disabled = 0

[PickleRick · Accepted solution]

Firstly, there is a little confusion about naming.

From splunk's point of view, an app is just a collection of files. The add-on is also an app. 🙂

So in general, if anyone says "deploy an app" or "create an app" or something like that, not referring to a particular app, this doesn't mean the "App for XXXX".

To add to the confusion, indeed typically support for a particular solution with Splunk is divided into two apps of which one is usually called Technical Add-On (TA) and is responsible for input, parsing and normalization (usually - mapping to CIM), and another is called App and contains some dashboards, reports and so on.

In your case - you don't need either the TA nor the App on your UF from what I see on the webpage you showed us.

That's about apps.

About the inputs...

Don't define inputs on search heads or indexers. Why would you do that? An input is the "point" where you're receiving data. So you need it on your Universal Forwarder. Everything from there is already covered by the rest of your config.

And, someone here provided _wrong_ solution of configuring the splunktcp:9997 input. The splunktcp:9997 input is for internal splunk-to-splunk communication and data flowing there should already be properly prepared so you should never need to touch that!

I never use the "splunk add monitor" command so I don't remember where it puts the config but I suppose it goes to /opt/splunk/etc/system/local/inputs.conf if not, check other inputs.conf instances in /opt/splunk/etc

Anyway, you'll have a stanza

[monitor:///var/log/udp514.log]
disabled=0

Maybe there are also other settings added to that stanza, honestly - I always edit config by hand so I don't remember 🙂

Into this stanza you have to add

sourcetype=pan:firewall
index=your_destination_index

 And restart the UF

[BrendanCO]

Thank you everyone for your help here! It is TRULY appreciated. I've done what @PickleRick spelled out and it now works. 

So to summarize I only changed the inputs.conf on the Universal Forwarder to include:

[monitor:///var/log/udp514.log]
disabled=0

sourcetype=pan:firewall
index=your_destination_index

The documentation really doesn't say to do that only on the UF and with your explanation of where the logs should be recognized and sourcetype'd, now I get it. 

 I'll email the Palo folks and see if they can make that more clear in their docs. 

Love this Community. Thanks again to everyone who responded. It ALL helps. 

[PickleRick]

You have to get used to it. Apps authors usually assume some particular way of transporting the events from the source system to splunk which might or might not be the most effective or the most convenient for you.

In this case Palo Alto assumed that you'd be receiving syslog events directly by the Splunk component on tcp or udp input. If you want to do it some other way (like using syslog daemon to write to files or using SC4S or any other way) it's up to you to adjust the process and input settings accordingly.

[BrendanCO]

Indeed! Looking forward to trying out SC4S sometime soon. Worth it?

[PickleRick]

To be fully honest with you - I haven't tried SC4S myself. I use rsyslog infrastructure (SC4S is syslog-ng based) to route events from syslog to HEC but from what people are saying it seems that it's a decent solution.

I find rsyslog configuration more straightforward and flexible than syslog-ng/SC4S but maybe that's just me.

[BrendanCO]

Ok. I very much appreciate this information. Yes, it didn't make sense to do this on the search head to me as well! So I will take that stanza out of my /opt/splunk/etc/apps/Splunk_TA_paloalto/local/inputs.conf. 

I will add that stanza to the UF right now.