topic Re: CSV Event Separation in Getting Data In

How to do CSV Event Separation?

JGP — Tue, 01 Aug 2023 18:33:54 GMT

We want event to separated for each header whenever there is new entry in the csv file. what would be the props applied to the sourcetype to have a single event

sample file

want details in one event whenever there is header inserted in csv file

please suggest

Re: CSV Event Separation

dhruv — Tue, 01 Aug 2023 10:35:11 GMT

Hi JGP,

You can create your own custom sourcetype to parse the events separately.
Let me give you an example :
My Sample Data -
Test ID,name
1,test1
2,test2
Test ID,name
3,test3
4,test4
Test ID,name
5,test5
6,test6

My Sourcetype configuration -
[test_csv]
LINE_BREAKER=(\n+)Test
SHOULD_LINEMERGE=true

NOTE - You can define the other parameters as per your need.

This is how it parses the data :

If you want to extract fields, You can write your own REGEX to do it.
Here is how I did it using "REX" command. Just to showcase :

The query :
source="testing.csv" host="so1" index="test_csv" sourcetype="test_csv"
| rex "(?<Test_ID>\d+),(?<name>.*)" max_match=0
| table Test_ID,name

Hope this helps.

Thanks.

Re: CSV Event Separation

Praz_123 — Tue, 01 Aug 2023 12:26:06 GMT

@JGP

Will you please elaborate the query.

Re: CSV Event Separation

dhruv — Tue, 01 Aug 2023 13:45:59 GMT

source="testing.csv" host="so1" index="test_csv" sourcetype="test_csv"
| rex "(?<Test_ID>\d+),(?<name>.*)" max_match=0
| table Test_ID,name

1. source="testing.csv" host="so1" index="test_csv" sourcetype="test_csv" - This is just searching for a particular index, source, sourcetype.

2. | rex "(?<Test_ID>\d+),(?<name>.*)" max_match=0 -

rex - It extracts fields inline from the events. The field extraction is not permanent. https://docs.splunk.com/Documentation/SplunkCloud/9.0.2305/SearchReference/Rex

"(?<Test_ID>\d+),(?<name>.*)" - It is regular expression as per my sample data. Try https://regex101.com/ for building the regex as per your data.

max_match=0 - It will allow rex command to match all matching field-values. Otherwise rex will only match the first occurrence.