Splunk Search

Select only some fields from csv to index

fabrizioalleva
Path Finder

Hi all,
I'm in enviroment so configured:

1 uf, 1 hf, 4 indexers, 1 search head, 1 master cluster.

I've to index a large CSV, read from the universal forwarder, which forwards data to the HF which pass the data to the indexer.

The CSV has 150 fields and I want to index only 10 of these. So I've configured these things:

on universal forwarder:

------------------

inputs.conf

------------------

[monitor:///myfolder/Interface*]
disabled = 0
index = interface_metrics
sourcetype = if_csv

on heavy forwarder

------------------

inputs.conf

------------------

[splunktcp://9996]
index=interface_metrics
sourcetype = if_csv

------------------

props.conf

------------------

[if_csv]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
HEADER_FIELD_DELIMITER =,
FIELD_DELIMITER=,
HEADER_FIELD_LINE_NUMBER = 0
TRANSFORMS-set=setnull, setparsing, nullhead

------------------

transforms.conf

------------------

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[nullhead]
REGEX = ifInDiscardsDelta
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^([^,]),([^,]),(?:[^,]+,\s*)([^,]),([^,])(?:[^,]+,\s*){5}([^,]),([^,])(?:[^,]+,\s*){3}([^,]),([^,]),(?:[^,]+,\s*){2}([^,]),([^,])(?:[^,]+,\s*){7}([^,])(?:(?:[^,]+,?\s)|(?:[,,])){123}([^,]),([^,])
DEST_KEY = queue
FORMAT = indexQueue

example CSV row :
0ef1fa5f-586c-48a4-a902-827aef967f47,1569309580446,300.0,100,9,0,0,0,0,6.6107712E7,5.0463189E7,151356.0,150857.0,0.176,0.135,0.0,0.0,0.0,0.0,0,0,0,0,0,0,4b16e13e-c391-4626-b364-2890fe5a009a,0,0,0,0,,,151351,149267,0,0,451,5,1139,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,039550ed-1d39-487f-9b12-276ad9472771,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,3.0,3.0,,,0.056,3.4E-4,0.2,3.0,3.0,,,0.056,3.4E-4,0.2,1569309300000,300

I want to keep the fields:
1,2,4,5,10,11,14,15,26,149,150

I don't succeed in indexing only the fields that I choose, but the whole row.

What I'm wrong ?

Thanks
Fabrizio

0 Karma
1 Solution

fabrizioalleva
Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

View solution in original post

0 Karma

fabrizioalleva
Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

0 Karma
Get Updates on the Splunk Community!

The Payment Operations Wake-Up Call: Why Financial Institutions Can't Afford ...

The same scenario plays out across financial institutions daily. A payment system fails at 11:30 AM on a busy ...

Make Your Case: A Ready-to-Send Letter for Getting Approval to Attend .conf25

Hello Splunkers, Want to attend .conf25 in Boston this year but not sure how to convince your manager? We've ...

Community Spotlight: A Splunk Expert's Journey

In the world of data analytics, some journeys leave a lasting impact not only on the individual but on the ...