Am I over utilizing Props/Transforms?

ekenne06 — Mon, 15 Feb 2021 21:13:03 GMT

I'm going to describe a typical use case.

The Software team will have one log file for most of it's outputs, lets call this HOUSES. This data will be generic health status information, transaction information, and some times data payloads in XML and some data in JSON. My practice has been to use a UF to monitor that file. Then on the Indexers i'll use Props to go through that data and set my time, linebreakser, and transforms.

Then in my transforms I use REGEX to match UNIQ fields in the data like, for example, RED-HOUSE. This will grab all data from that original file that contains RED-HOUSE. I then do a sourcetype override and make the new source type HOUSES:RED-HOUSE.

Then on my Searchhead i'll define my props/transforms (Using EXTRACT/REPORT) for field extractions.

I'll try to show a brief example:

Data:

lets say sourcetype is LOGS

dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS START PERIODIC PERIODIC TIMER_START .D00002 dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS SEND PERIODIC CONTRACT REQUEST #502, .D00002 dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS PERIODIC PERIODIC CONTRACT TIMER_EXPIRED 0,.D00002

On indexing Tier:

Props:

[LOGS] SHOULD_LINEMERGE = False TRANSFORMS-ADS = ADS, EFG, XYZ

Tranforms:

[ADS] REGEX = ADS DEST_KEY = MetaData:sourcetype FORMAT = sourcetype::LOGS:ADS

Search head:

props:

[ADS] REPORT-ADS = ADS_EXTRACTIONS

(lets assume several data points use the same extraction, so REPORT should be used

TRANSFORMS:

[ADS_EXTRACTIONS] REGEX = ADS (?P<ADS_Method>\w+\s\w+)\s(?P<ADS_Method_Type>\w+)\s(?P<Method_Message>\w+)(?:.*)(?P<tail_no>.{7}$)

Is this overkill? Or is this taking the right approach? I basically try to make a sourcetype for every differential of field extractions format there is.

Re: Am I over utilizing Props/Transforms?

richgalloway — Tue, 16 Feb 2021 01:16:09 GMT

Sourcetypes tell Splunk how to parse events into fields. If an event doesn't work with the existing sourcetypes (and the sourcetype can't be made to work) then create a new one that does.

Consider un-friending your developers until they come up with a sane method of logging. 😀

topic Am I over utilizing Props/Transforms? in Getting Data In

Am I over utilizing Props/Transforms?

Re: Am I over utilizing Props/Transforms?