Getting Data In

Creating new soucertype using Props.conf and transform.conf

raomu
Explorer

All my network data comes to default source type irrespective of type of devices.

index = network
sourcetype = network

I have define props.conf and transforms.conf to separate the firewall ( Palo Alto logs ) comes to different soucertype pan:log

The new soucertype "pan:log" will take place before indexing or ?

Trasnforms.conf

[PaloAlto_sourcetype_setting]
REGEX = ^\w+\s+\d+\s+\d+:\d+:\d+\s+(\w\w-\w+-panorama)
FORMAT = sourcetype::pan:log
DEST_KEY = MetaData:Sourcetype

Tags (1)
0 Karma

micahkemp
Champion

I see you are referencing the Palo Alto TA sourcetype, which does additional sourcetype rewriting when events come in. I strongly advise you to have your events first come in as the necessary pan:log, instead of rewriting them to pan:log after they arrive.

Please reference this ongoing answers post about this topic.

0 Karma

mayurr98
Super Champion

hey @raomu
sourcetype override occurs at parse-time, it works only on an indexer or heavy forwarder, not on a universal forwarder which means before indexing
This is written in
https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

you can have look at props.conf Splunk_TA_paloalto
https://github.com/PaloAltoNetworks/Splunk_TA_paloalto/blob/master/default/props.conf

Refer this link to create new sourcetype
https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

[pan:log]
REGEX = <your_regex>
FORMAT = sourcetype::<new_sourcetype>
DEST_KEY = MetaData:Sourcetype

Also look at
https://answers.splunk.com/answers/210347/how-to-get-palo-alto-app-transforms-working.html
I hope this helps!

0 Karma

raomu
Explorer

Thanks for your response.

I have all the Palo Alto settings you shared. My question is if i am going to force these settings in transforms.conf will this take place before indexing ? or after indexing ?

As you see my Inputs.conf I am giving the soucertype "network" so it will index all the data to "network" soucertype first and then we using the transforms.conf to filter logs for Palo Alto and putting them in another soucetype. question here is the change of soucertype will happen during search time to Index time ?

0 Karma

mayurr98
Super Champion

hey i have edited my answer
so basically whatever you write in transforms.conf happens in parsing phase i.e. before indexing
see data pipeline flow
http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Configurationparametersandthedatapipeline

I hope this solves your query!

0 Karma
Get Updates on the Splunk Community!

Now Playing: Splunk Education Summer Learning Premieres

It’s premiere season, and Splunk Education is rolling out new releases you won’t want to miss. Whether you’re ...

The Visibility Gap: Hybrid Networks and IT Services

The most forward thinking enterprises among us see their network as much more than infrastructure – it's their ...

Get Operational Insights Quickly with Natural Language on the Splunk Platform

In today’s fast-paced digital world, turning data into actionable insights is essential for success. With ...