Getting Data In

How to do CSV Event Separation?

JGP
Explorer

We want event to separated for each header whenever there is new entry in the csv file. what would be the props applied to the sourcetype to have a single event 

sample file

JGP_0-1690466270798.png

 

want details in one event whenever there is header inserted in csv file

JGP_1-1690466493598.png

please suggest

Labels (1)
Tags (1)
0 Karma

Praz_123
Communicator

@JGP 

Will you please elaborate the query. 

0 Karma

dhruv
Explorer

source="testing.csv" host="so1" index="test_csv" sourcetype="test_csv"
| rex "(?<Test_ID>\d+),(?<name>.*)" max_match=0
| table Test_ID,name

 

1. source="testing.csv" host="so1" index="test_csv" sourcetype="test_csv" - This is just searching for a particular index, source, sourcetype.

2. | rex "(?<Test_ID>\d+),(?<name>.*)" max_match=0 - 

rex - It extracts fields inline from the events. The field extraction is not permanent. https://docs.splunk.com/Documentation/SplunkCloud/9.0.2305/SearchReference/Rex

"(?<Test_ID>\d+),(?<name>.*)" - It is regular expression as per my sample data. Try https://regex101.com/ for building the regex as per your data.

max_match=0 - It will allow rex command to match all matching field-values. Otherwise rex will only match the first occurrence.

0 Karma

dhruv
Explorer

Hi JGP,

You can create your own custom sourcetype to parse the events separately.
Let me give you an example : 
My Sample Data - 
Test ID,name
1,test1
2,test2
Test ID,name
3,test3
4,test4
Test ID,name
5,test5
6,test6

My Sourcetype configuration - 
[test_csv]
LINE_BREAKER=(\n+)Test
SHOULD_LINEMERGE=true

NOTE - You can define the other parameters as per your need.

This is how it parses the data : 

dhruv_0-1690885965447.png

If you want to extract fields, You can write your own REGEX to do it.
Here is how I did it using "REX" command. Just to showcase : 

dhruv_1-1690886082812.png


The query : 
source="testing.csv" host="so1" index="test_csv" sourcetype="test_csv"
| rex "(?<Test_ID>\d+),(?<name>.*)" max_match=0
| table Test_ID,name

Hope this helps.

Thanks.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mastering Threat Intelligence in ES 8.5, Splunk AI Assistant v2, and More from Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

Break the Build: Inside the KubeDoom Lounge at .conf26

    You step up to the machine. The pixelated corridors of a certain 1993 FPS load in front of you, EMP Pulse ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...