topic How to ignore concrete rows from csv file before indexing? in Splunk Search

How to ignore concrete rows from csv file before indexing?

spisiakmi — Tue, 10 Sep 2019 10:06:24 GMT

Hi,

I spent really a lot of time, but found no solution. Here is my problem. There is CSV file, which should be indexed:

Lötprotokoll Version 1.0
Laufende Nummer;Version Lötprotokoll;Lötprogramm Version
1;1;1;
2;3;4;
937;381.000000;110.000000;
938;382.000000;113.000000;
565136;1;1;9;

I want to sent it to the Splunk using forwarder. I want, that the rows 1, 3, 4, 5, 6 will be ignored and not indexed. Can anybody help me, please, how to do it?

Re: How to ignore concrete rows from csv file before indexing?

spisiakmi — Wed, 11 Sep 2019 06:19:48 GMT

I can create 2 very similar regex.
1. regex with 2 captured groups, which should be indexed
.+\n(.+\n).+\n.+\n.+\n.+\n(.+\n)
2. regex with 2 captured groups, which should be ignored
(.+\n).+\n(.+\n.+\n.+\n.+\n).+\n

I have probably 2 possible choises.

preamble_regex: the question is, if it is possible to put here regex #2, and if the captured groups will be accepted
transforms.conf and parsing [stanza]: [setparsing] REGEX = regex #1 DEST_KEY = queue FORMAT = indexQueue the same question, if I use regex #1, if the captured groups will be accepted

Re: How to ignore concrete rows from csv file before indexing?

spisiakmi — Wed, 30 Sep 2020 02:06:34 GMT

I tried to import the file manualy. I created my own sourcetype and added the parameter PREAMBLE_REGEX. It has no influence on the previewed content of indexed csv file. Even I tried simple regular expressions. The question is, if the parameter PREAMBLE_REGEX is either working properly.

Re: How to ignore concrete rows from csv file before indexing?

spisiakmi — Mon, 14 Oct 2019 13:33:57 GMT

Finally I used the Powershell script, which picks only (indexed from 0) rows 1, 6:

Get-ChildItem *\*.protocol | ForEach { Get-Content $_.Fullname | Select-Object -Index 1,6 | Out-File "$($_.Directory)\$($_.BaseName).csv" -Encoding utf8}