heterogenous sourcetype in log file

[riotto]

I have a log file that has multiple sourcetypes or entries defined by a different format. Each entry in the log has a field that tells me
the type of data (the fields) or format for that line (or event). For example: entries could be something like
ABCD, aa, bb, cc, dd
XYZ, 1, 2, 3, 4
LMNOP, 1.45, 2.23, 3.89, 444, 5.67, 6.1
GHIK,1, 2, 3, 4, 5, 6, 7, 8, 9
What is the best way of defining a sourcetype for each entry and the field names (or its header) for them.

[ddrillic]

Maybe the following can help - sourcetype best practices

@hexx says -

[riotto]

When I search the data or trend a value in the SPLUNK indexer don't I need to give a field name to each value? This is where I am struggling.
Do I need a FIELDS = in the transform.conf ?
so that the event ABCD, aa, bb, cc, dd --- value aa has attribute name - Credit_Rating

value bb has attribute name - FICO. ABCD in the event tells me the data is Credit data...I hope this make sense

[gcusello]

You have to define a sourcetype to acquire your file (e.g.: original).
after you have to configure a transformation for each kind of sourcetype finding a regular expression to identify events for each sourcetype.
To do this, modify your props.conf
[original]
TRANSFORMS-sourcetype1 = sourcetype1
TRANSFORMS-sourcetype2 = sourcetype2
TRANSFORMS-sourcetype3 = sourcetype3

After modify transforms.conf
[sourcetype1]
REGEX = myregex1
FORMAT = sourcetype::newsourcetype1
DEST_KEY = MetaData:Sourcetype

[sourcetype2]
REGEX = myregex2
FORMAT = sourcetype::newsourcetype2
DEST_KEY = MetaData:Sourcetype

[sourcetype3]
REGEX = myregex3
FORMAT = sourcetype::newsourcetype3
DEST_KEY = MetaData:Sourcetype

In this way you override the original sourcetype value with the requested ones.

You can find information at http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Advancedsourcetypeoverrides

Bye.
Giuseppe

[riotto]

I was trying to avoid the TRANSFORMS-class because Splunk says it's a big performance hit. Is there a way to use the REPORT-class?

[gcusello]

I don't think that there's a different way to override sourcetypes!
maybe you could extract different fields using regex and use them, but it's less easy and you have to verify them in detail to not have wrong data.
Bye.
Giuseppe

[riotto]

How does it know what the fields names are for each sourcetype? Is there something else I have to add to the transforms,conf?

[somesoni2]

Could we have some real sample entries?