topic How to omit columns from CSV-style event input? in Getting Data In

How to omit columns from CSV-style event input?

ziq — Thu, 01 Feb 2018 00:11:33 GMT

I intend to import a CSV-style file into Splunk. The file has about 30 columns, about 120 million lines and is about 150GB of size. Of the file, I only require a subset of columns.

The file's contents shall be imported as events and not as a CSV lookup file.

For sake of simplicity, assume the structure below:

src,dest_ip,dest_port,dest_user,dest_zone
10.50.60.80,192.0.2.92,443,emily,Internet
10.50.60.53,203.0.113.12,389,brian,Intranet
10.33.118.40,198.51.100.65,80,john,Internet
...

Is there any way to exclude the "dest_user" column from the import?

(Running the data through a sed/awk/perl script beforehand is certainly possible, but given the size of the file this would be computationally expensive. And as Splunk already extracts the field headers, it appears to me that excluding columns from import would be the cleaner and more efficient approach. Furthermore, it is likely that I will have to deal with similar files (that have different fieldsets or column orders) in the future.)

Re: How to omit columns from CSV-style event input?

p_gurav — Thu, 01 Feb 2018 05:22:11 GMT

Hi ziq,

You can use script to create new csv with selected fields and then index into splunk.

Re: How to omit columns from CSV-style event input?

ziq — Fri, 02 Feb 2018 13:32:38 GMT

I'm aware and mentioned the script option in my question. But running the data through a script beforehand would be twice as computationally expensive (or more). I also anticipate similar files with different field sets in the future, hence I would need to touch/modify the script for each of these files.

(Running the data through a
sed/awk/perl script beforehand is
certainly possible, but given the size
of the file this would be
computationally expensive. And as
Splunk already extracts the field
headers, it appears to me that
excluding columns from import would be
the cleaner and more efficient
approach. Furthermore, it is likely
that I will have to deal with similar
files (that have different fieldsets
or column orders) in the future.)