topic Re: heterogenous sourcetype in log file in Getting Data In

heterogenous sourcetype in log file

riotto — Mon, 03 Oct 2016 14:27:20 GMT

I have a log file that has multiple sourcetypes or entries defined by a different format. Each entry in the log has a field that tells me
the type of data (the fields) or format for that line (or event). For example: entries could be something like
ABCD, aa, bb, cc, dd
XYZ, 1, 2, 3, 4
LMNOP, 1.45, 2.23, 3.89, 444, 5.67, 6.1
GHIK,1, 2, 3, 4, 5, 6, 7, 8, 9
What is the best way of defining a sourcetype for each entry and the field names (or its header) for them.

Re: heterogenous sourcetype in log file

somesoni2 — Mon, 03 Oct 2016 14:53:30 GMT

Could we have some real sample entries?

Re: heterogenous sourcetype in log file

gcusello — Mon, 03 Oct 2016 15:06:01 GMT

You have to define a sourcetype to acquire your file (e.g.: original).
after you have to configure a transformation for each kind of sourcetype finding a regular expression to identify events for each sourcetype.
To do this, modify your props.conf
[original]
TRANSFORMS-sourcetype1 = sourcetype1
TRANSFORMS-sourcetype2 = sourcetype2
TRANSFORMS-sourcetype3 = sourcetype3

After modify transforms.conf
[sourcetype1]
REGEX = myregex1
FORMAT = sourcetype::newsourcetype1
DEST_KEY = MetaData:Sourcetype

[sourcetype2]
REGEX = myregex2
FORMAT = sourcetype::newsourcetype2
DEST_KEY = MetaData:Sourcetype

[sourcetype3]
REGEX = myregex3
FORMAT = sourcetype::newsourcetype3
DEST_KEY = MetaData:Sourcetype

In this way you override the original sourcetype value with the requested ones.

You can find information at http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Advancedsourcetypeoverrides

Bye.
Giuseppe

Re: heterogenous sourcetype in log file

riotto — Mon, 03 Oct 2016 15:23:41 GMT

I was trying to avoid the TRANSFORMS-class because Splunk says it's a big performance hit. Is there a way to use the REPORT-class?

Re: heterogenous sourcetype in log file

gcusello — Mon, 03 Oct 2016 15:30:47 GMT

I don't think that there's a different way to override sourcetypes!
maybe you could extract different fields using regex and use them, but it's less easy and you have to verify them in detail to not have wrong data.
Bye.
Giuseppe

Re: heterogenous sourcetype in log file

riotto — Mon, 03 Oct 2016 20:35:44 GMT

How does it know what the fields names are for each sourcetype? Is there something else I have to add to the transforms,conf?

Re: heterogenous sourcetype in log file

ddrillic — Mon, 03 Oct 2016 21:41:07 GMT

Maybe the following can help - sourcetype best practices

@hexx says -

Re: heterogenous sourcetype in log file

riotto — Mon, 03 Oct 2016 21:58:59 GMT

When I search the data or trend a value in the SPLUNK indexer don't I need to give a field name to each value? This is where I am struggling.
Do I need a FIELDS = in the transform.conf ?
so that the event ABCD, aa, bb, cc, dd --- value aa has attribute name - Credit_Rating

value bb has attribute name - FICO. ABCD in the event tells me the data is Credit data...I hope this make sense