Getting Data In

Am I over utilizing Props/Transforms?

ekenne06
Path Finder

I'm going to describe a typical use case.

 

The Software team will have one log file for most of it's outputs, lets call this HOUSES. This data will be generic health status information, transaction information, and some times data payloads in XML and some data in JSON. My practice has been to use a UF to monitor that file. Then on the Indexers i'll use Props to go through that data and set my time, linebreakser, and transforms. 

Then in my transforms I use REGEX to match UNIQ fields in the data like, for example, RED-HOUSE. This will grab all data from that original file that contains RED-HOUSE. I then do a sourcetype override and make the new source type HOUSES:RED-HOUSE. 

Then on my Searchhead i'll define my props/transforms (Using EXTRACT/REPORT) for field extractions. 

I'll try to show a brief example:

 

Data:

lets say sourcetype is LOGS

 

dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS START PERIODIC PERIODIC TIMER_START .D00002
​
dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS SEND PERIODIC CONTRACT REQUEST #502,  .D00002
​
dcnsnoctads-1 2021/02/12 01:59:59.105 GMT-FANS ADS PERIODIC PERIODIC CONTRACT TIMER_EXPIRED 0,.D00002

 

On indexing Tier:

Props:

 

[LOGS]
SHOULD_LINEMERGE = False
TRANSFORMS-ADS = ADS, EFG, XYZ

 

Tranforms:

 

[ADS]
REGEX = ADS
DEST_KEY = MetaData:sourcetype
FORMAT = sourcetype::LOGS:ADS

 

 

Search head:

props:

 

[ADS]

REPORT-ADS = ADS_EXTRACTIONS

 

(lets assume several data points use the same extraction, so REPORT should be used 

TRANSFORMS:

 

[ADS_EXTRACTIONS]
REGEX = ADS (?P<ADS_Method>\w+\s\w+)\s(?P<ADS_Method_Type>\w+)\s(?P<Method_Message>\w+)(?:.*)(?P<tail_no>.{7}$)

 

 

 

Is this overkill? Or is this taking the right approach? I basically try to make a sourcetype for every differential of field extractions format there is. 

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Sourcetypes tell Splunk how to parse events into fields.  If an event doesn't work with the existing sourcetypes (and the sourcetype can't be made to work) then create a new one that does.

Consider un-friending your developers until they come up with a sane method of logging.  😀

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Sourcetypes tell Splunk how to parse events into fields.  If an event doesn't work with the existing sourcetypes (and the sourcetype can't be made to work) then create a new one that does.

Consider un-friending your developers until they come up with a sane method of logging.  😀

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...