Creating new soucertype using Props.conf and transform.conf

[raomu]

All my network data comes to default source type irrespective of type of devices.

index = network
sourcetype = network

I have define props.conf and transforms.conf to separate the firewall ( Palo Alto logs ) comes to different soucertype pan:log

The new soucertype "pan:log" will take place before indexing or ?

Trasnforms.conf

[PaloAlto_sourcetype_setting]
REGEX = ^\w+\s+\d+\s+\d+:\d+:\d+\s+(\w\w-\w+-panorama)
FORMAT = sourcetype::pan:log
DEST_KEY = MetaData:Sourcetype

[micahkemp]

I see you are referencing the Palo Alto TA sourcetype, which does additional sourcetype rewriting when events come in. I strongly advise you to have your events first come in as the necessary pan:log, instead of rewriting them to pan:log after they arrive.

Please reference this ongoing answers post about this topic.

[mayurr98]

hey @raomu
sourcetype override occurs at parse-time, it works only on an indexer or heavy forwarder, not on a universal forwarder which means before indexing
This is written in
https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

you can have look at props.conf Splunk_TA_paloalto
https://github.com/PaloAltoNetworks/Splunk_TA_paloalto/blob/master/default/props.conf

Refer this link to create new sourcetype
https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

[pan:log]
REGEX = <your_regex>
FORMAT = sourcetype::<new_sourcetype>
DEST_KEY = MetaData:Sourcetype

Also look at
https://answers.splunk.com/answers/210347/how-to-get-palo-alto-app-transforms-working.html
I hope this helps!

[raomu]

Thanks for your response.

I have all the Palo Alto settings you shared. My question is if i am going to force these settings in transforms.conf will this take place before indexing ? or after indexing ?

As you see my Inputs.conf I am giving the soucertype "network" so it will index all the data to "network" soucertype first and then we using the transforms.conf to filter logs for Palo Alto and putting them in another soucetype. question here is the change of soucertype will happen during search time to Index time ?

[mayurr98]

hey i have edited my answer
so basically whatever you write in transforms.conf happens in parsing phase i.e. before indexing
see data pipeline flow
http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Configurationparametersandthedatapipeline

I hope this solves your query!