Getting Data In

Indexing a CSV with mixed sourcetypes.

BobM
Builder

In SPLUNK, can we index and search data with varying formats?

We have a csv file containg events with different formats. Approximately there are 100 formats available. There is also a config file which can be used as a lookup to determine the column names of the corresponding record.

 
If we need to put such data into Splunk, where and how we can specify the format for each event to be handled?

The data format is something like this:
 

Dept01,ABC,20120904,001500,Gn,3,ggss Test-City_gtpp,1,103031    Dept01,DEF,20120904,1500,1,13.17,86.83,277,1876288,34078720,2,3.34,96.66,235,1681068   Dept01,PQR,20120904,001500,24,3,0
Dept01,XYZ12,20120904,001500,Gi,2,abc-xyz.net.P2,67651,0,0,63419,R,0,G,10.208.0.0,abc-xyz.net.GP

  
The second column (ABC, PQR) are different source types, and each has varying column names and numbers. Since there are no key-value pairs, we cannot use EXTRACT.
In this case, how can we populate the fields?
 

Tags (3)
1 Solution

BobM
Builder

I found this works. Set the sourcetype with a transform and regex then set the field names at search time.

props.conf

[mycsvsourcetype]
TRANSFORMS-mysourcetyper = csvsourcetypes

transforms.conf

[csvsourcetypes]
DEST_KEY = MetaData:Sourcetype
REGEX = ^\w*,(\w+),
FORMAT = sourcetype::$1

then back in the props.conf file for each source type

[ABC]
REPORT-abc = ABCfields

[DEF]
REPORT-def = DEFfields

and back to transforms.conf to define the filed names

[ABCfields]
DELIMS = ","
FIELDS = "field1", "field2", "field3"

[DEFfields]
DELIMS = ","
FIELDS = "fieldA", "fieldB", "fieldC"

View solution in original post

BobM
Builder

I found this works. Set the sourcetype with a transform and regex then set the field names at search time.

props.conf

[mycsvsourcetype]
TRANSFORMS-mysourcetyper = csvsourcetypes

transforms.conf

[csvsourcetypes]
DEST_KEY = MetaData:Sourcetype
REGEX = ^\w*,(\w+),
FORMAT = sourcetype::$1

then back in the props.conf file for each source type

[ABC]
REPORT-abc = ABCfields

[DEF]
REPORT-def = DEFfields

and back to transforms.conf to define the filed names

[ABCfields]
DELIMS = ","
FIELDS = "field1", "field2", "field3"

[DEFfields]
DELIMS = ","
FIELDS = "fieldA", "fieldB", "fieldC"

BobM
Builder

After testing this, I found I have to use REPORT and not TRANSFORMS as it is a search time function. Editing original answer to reflect this

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...