Getting Data In

heterogenous sourcetype in log file

riotto
Path Finder

I have a log file that has multiple sourcetypes or entries defined by a different format. Each entry in the log has a field that tells me
the type of data (the fields) or format for that line (or event). For example: entries could be something like
ABCD, aa, bb, cc, dd
XYZ, 1, 2, 3, 4
LMNOP, 1.45, 2.23, 3.89, 444, 5.67, 6.1
GHIK,1, 2, 3, 4, 5, 6, 7, 8, 9
What is the best way of defining a sourcetype for each entry and the field names (or its header) for them.

Tags (1)
0 Karma

ddrillic
Ultra Champion

Maybe the following can help - sourcetype best practices

@hexx says -

alt text

0 Karma

riotto
Path Finder

When I search the data or trend a value in the SPLUNK indexer don't I need to give a field name to each value? This is where I am struggling.
Do I need a FIELDS = in the transform.conf ?
so that the event ABCD, aa, bb, cc, dd --- value aa has attribute name - Credit_Rating

value bb has attribute name - FICO. ABCD in the event tells me the data is Credit data...I hope this make sense

0 Karma

gcusello
SplunkTrust
SplunkTrust

You have to define a sourcetype to acquire your file (e.g.: original).
after you have to configure a transformation for each kind of sourcetype finding a regular expression to identify events for each sourcetype.
To do this, modify your props.conf
[original]
TRANSFORMS-sourcetype1 = sourcetype1
TRANSFORMS-sourcetype2 = sourcetype2
TRANSFORMS-sourcetype3 = sourcetype3

After modify transforms.conf
[sourcetype1]
REGEX = myregex1
FORMAT = sourcetype::newsourcetype1
DEST_KEY = MetaData:Sourcetype

[sourcetype2]
REGEX = myregex2
FORMAT = sourcetype::newsourcetype2
DEST_KEY = MetaData:Sourcetype

[sourcetype3]
REGEX = myregex3
FORMAT = sourcetype::newsourcetype3
DEST_KEY = MetaData:Sourcetype

In this way you override the original sourcetype value with the requested ones.

You can find information at http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Advancedsourcetypeoverrides

Bye.
Giuseppe

0 Karma

riotto
Path Finder

I was trying to avoid the TRANSFORMS-class because Splunk says it's a big performance hit. Is there a way to use the REPORT-class?

0 Karma

gcusello
SplunkTrust
SplunkTrust

I don't think that there's a different way to override sourcetypes!
maybe you could extract different fields using regex and use them, but it's less easy and you have to verify them in detail to not have wrong data.
Bye.
Giuseppe

0 Karma

riotto
Path Finder

How does it know what the fields names are for each sourcetype? Is there something else I have to add to the transforms,conf?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Could we have some real sample entries?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...