Splunk Search

Select only some fields from csv to index

fabrizioalleva
Path Finder

Hi all,
I'm in enviroment so configured:

1 uf, 1 hf, 4 indexers, 1 search head, 1 master cluster.

I've to index a large CSV, read from the universal forwarder, which forwards data to the HF which pass the data to the indexer.

The CSV has 150 fields and I want to index only 10 of these. So I've configured these things:

on universal forwarder:

------------------

inputs.conf

------------------

[monitor:///myfolder/Interface*]
disabled = 0
index = interface_metrics
sourcetype = if_csv

on heavy forwarder

------------------

inputs.conf

------------------

[splunktcp://9996]
index=interface_metrics
sourcetype = if_csv

------------------

props.conf

------------------

[if_csv]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
HEADER_FIELD_DELIMITER =,
FIELD_DELIMITER=,
HEADER_FIELD_LINE_NUMBER = 0
TRANSFORMS-set=setnull, setparsing, nullhead

------------------

transforms.conf

------------------

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[nullhead]
REGEX = ifInDiscardsDelta
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^([^,]),([^,]),(?:[^,]+,\s*)([^,]),([^,])(?:[^,]+,\s*){5}([^,]),([^,])(?:[^,]+,\s*){3}([^,]),([^,]),(?:[^,]+,\s*){2}([^,]),([^,])(?:[^,]+,\s*){7}([^,])(?:(?:[^,]+,?\s)|(?:[,,])){123}([^,]),([^,])
DEST_KEY = queue
FORMAT = indexQueue

example CSV row :
0ef1fa5f-586c-48a4-a902-827aef967f47,1569309580446,300.0,100,9,0,0,0,0,6.6107712E7,5.0463189E7,151356.0,150857.0,0.176,0.135,0.0,0.0,0.0,0.0,0,0,0,0,0,0,4b16e13e-c391-4626-b364-2890fe5a009a,0,0,0,0,,,151351,149267,0,0,451,5,1139,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,039550ed-1d39-487f-9b12-276ad9472771,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,3.0,3.0,,,0.056,3.4E-4,0.2,3.0,3.0,,,0.056,3.4E-4,0.2,1569309300000,300

I want to keep the fields:
1,2,4,5,10,11,14,15,26,149,150

I don't succeed in indexing only the fields that I choose, but the whole row.

What I'm wrong ?

Thanks
Fabrizio

0 Karma
1 Solution

fabrizioalleva
Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

View solution in original post

0 Karma

fabrizioalleva
Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...