I intend to import a CSV-style file into Splunk. The file has about 30 columns, about 120 million lines and is about 150GB of size. Of the file, I only require a subset of columns.
The file's contents shall be imported as events and not as a CSV lookup file.
For sake of simplicity, assume the structure below:
src,dest_ip,dest_port,dest_user,dest_zone
10.50.60.80,192.0.2.92,443,emily,Internet
10.50.60.53,203.0.113.12,389,brian,Intranet
10.33.118.40,198.51.100.65,80,john,Internet
...
Is there any way to exclude the "dest_user" column from the import?
(Running the data through a sed/awk/perl script beforehand is certainly possible, but given the size of the file this would be computationally expensive. And as Splunk already extracts the field headers, it appears to me that excluding columns from import would be the cleaner and more efficient approach. Furthermore, it is likely that I will have to deal with similar files (that have different fieldsets or column orders) in the future.)
... View more