topic Re: Creating new soucertype using Props.conf and transform.conf in Getting Data In

Creating new soucertype using Props.conf and transform.conf

raomu — Tue, 29 Sep 2020 17:32:40 GMT

All my network data comes to default source type irrespective of type of devices.

index = network
sourcetype = network

I have define props.conf and transforms.conf to separate the firewall ( Palo Alto logs ) comes to different soucertype pan:log

The new soucertype "pan:log" will take place before indexing or ?

Trasnforms.conf

[PaloAlto_sourcetype_setting]
REGEX = ^\w+\s+\d+\s+\d+:\d+:\d+\s+(\w\w-\w+-panorama)
FORMAT = sourcetype::pan:log
DEST_KEY = MetaData:Sourcetype

Re: Creating new soucertype using Props.conf and transform.conf

mayurr98 — Tue, 29 Sep 2020 17:32:42 GMT

hey @raomu
sourcetype override occurs at parse-time, it works only on an indexer or heavy forwarder, not on a universal forwarder which means before indexing
This is written in
https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

you can have look at props.conf Splunk_TA_paloalto
https://github.com/PaloAltoNetworks/Splunk_TA_paloalto/blob/master/default/props.conf

Refer this link to create new sourcetype
https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

[pan:log]
REGEX = <your_regex>
FORMAT = sourcetype::<new_sourcetype>
DEST_KEY = MetaData:Sourcetype

Also look at
https://answers.splunk.com/answers/210347/how-to-get-palo-alto-app-transforms-working.html
I hope this helps!

Re: Creating new soucertype using Props.conf and transform.conf

raomu — Thu, 11 Jan 2018 06:40:24 GMT

Thanks for your response.

I have all the Palo Alto settings you shared. My question is if i am going to force these settings in transforms.conf will this take place before indexing ? or after indexing ?

As you see my Inputs.conf I am giving the soucertype "network" so it will index all the data to "network" soucertype first and then we using the transforms.conf to filter logs for Palo Alto and putting them in another soucetype. question here is the change of soucertype will happen during search time to Index time ?

Re: Creating new soucertype using Props.conf and transform.conf

mayurr98 — Thu, 11 Jan 2018 06:44:29 GMT

hey i have edited my answer
so basically whatever you write in transforms.conf happens in parsing phase i.e. before indexing
see data pipeline flow
http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Configurationparametersandthedatapipeline

I hope this solves your query!

Re: Creating new soucertype using Props.conf and transform.conf

micahkemp — Thu, 11 Jan 2018 15:31:01 GMT

I see you are referencing the Palo Alto TA sourcetype, which does additional sourcetype rewriting when events come in. I strongly advise you to have your events first come in as the necessary pan:log, instead of rewriting them to pan:log after they arrive.

Please reference this ongoing answers post about this topic.