I'm going to describe a typical use case.
The Software team will have one log file for most of it's outputs, lets call this HOUSES. This data will be generic health status information, transaction information, and some times data payloads in XML and some data in JSON. My practice has been to use a UF to monitor that file. Then on the Indexers i'll use Props to go through that data and set my time, linebreakser, and transforms.
Then in my transforms I use REGEX to match UNIQ fields in the data like, for example, RED-HOUSE. This will grab all data from that original file that contains RED-HOUSE. I then do a sourcetype override and make the new source type HOUSES:RED-HOUSE.
Then on my Searchhead i'll define my props/transforms (Using EXTRACT/REPORT) for field extractions.
I'll try to show a brief example:
Data:
lets say sourcetype is LOGS
dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS START PERIODIC PERIODIC TIMER_START .D00002
dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS SEND PERIODIC CONTRACT REQUEST #502, .D00002
dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS PERIODIC PERIODIC CONTRACT TIMER_EXPIRED 0,.D00002
On indexing Tier:
Props:
[LOGS]
SHOULD_LINEMERGE = False
TRANSFORMS-ADS = ADS, EFG, XYZ
Tranforms:
[ADS]
REGEX = ADS
DEST_KEY = MetaData:sourcetype
FORMAT = sourcetype::LOGS:ADS
Search head:
props:
[ADS]
REPORT-ADS = ADS_EXTRACTIONS
(lets assume several data points use the same extraction, so REPORT should be used
TRANSFORMS:
[ADS_EXTRACTIONS]
REGEX = ADS (?P<ADS_Method>\w+\s\w+)\s(?P<ADS_Method_Type>\w+)\s(?P<Method_Message>\w+)(?:.*)(?P<tail_no>.{7}$)
Is this overkill? Or is this taking the right approach? I basically try to make a sourcetype for every differential of field extractions format there is.
