Archive

Select only some fields from csv to index

Path Finder

Hi all,
I'm in enviroment so configured:

1 uf, 1 hf, 4 indexers, 1 search head, 1 master cluster.

I've to index a large CSV, read from the universal forwarder, which forwards data to the HF which pass the data to the indexer.

The CSV has 150 fields and I want to index only 10 of these. So I've configured these things:

on universal forwarder:

------------------

inputs.conf

------------------

[monitor:///myfolder/Interface*]
disabled = 0
index = interfacemetrics
sourcetype = if
csv

on heavy forwarder

------------------

inputs.conf

------------------

[splunktcp://9996]
index=interfacemetrics
sourcetype = if
csv

------------------

props.conf

------------------

[ifcsv]
INDEXED
EXTRACTIONS = CSV
HEADERFIELDLINENUMBER=1
HEADER
FIELDDELIMITER =,
FIELD
DELIMITER=,
HEADERFIELDLINE_NUMBER = 0
TRANSFORMS-set=setnull, setparsing, nullhead

------------------

transforms.conf

------------------

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[nullhead]
REGEX = ifInDiscardsDelta
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^([^,]),([^,]),(?:[^,]+,\s)([^,]),([^,])(?:[^,]+,\s){5}([^,]),([^,])(?:[^,]+,\s){3}([^,]),([^,]),(?:[^,]+,\s){2}([^,]),([^,])(?:[^,]+,\s){7}([^,])(?:(?:[^,]+,?\s)|(?:[,,])){123}([^,]),([^,]*)
DEST_KEY = queue
FORMAT = indexQueue

example CSV row :
0ef1fa5f-586c-48a4-a902-827aef967f47,1569309580446,300.0,100,9,0,0,0,0,6.6107712E7,5.0463189E7,151356.0,150857.0,0.176,0.135,0.0,0.0,0.0,0.0,0,0,0,0,0,0,4b16e13e-c391-4626-b364-2890fe5a009a,0,0,0,0,,,151351,149267,0,0,451,5,1139,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,039550ed-1d39-487f-9b12-276ad9472771,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,3.0,3.0,,,0.056,3.4E-4,0.2,3.0,3.0,,,0.056,3.4E-4,0.2,1569309300000,300

I want to keep the fields:
1,2,4,5,10,11,14,15,26,149,150

I don't succeed in indexing only the fields that I choose, but the whole row.

What I'm wrong ?

Thanks
Fabrizio

0 Karma
1 Solution

Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

View solution in original post

0 Karma

Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

View solution in original post

0 Karma