Getting Data In

Extract some lines of an event from a CSV file and index in separate sourcetype OR Index

mala_splunk_91
Explorer

Hi guys,

Please provide your input on the below scenario.
I have some events like below. Here , I want to extract some part of event which is in CSV format and that is starting after "#" till the end of an event and store them in separate new index/sourcetype, either by using props/transforms conf OR using query.

I have questions like
1)Is there any way to split/extract some part of an event and store it in separate index/sourcetype?
2)How can I extract only CSV event part and display/View it in table format in Splunk?

Final result I need is:
Extract CSV format events separately from the below events and display it in table format OR store in lookup file.(Simply, to make it human readable).

sample.log:

sep-12 02:45:56 This message is received from printer,something like this as a eveent.
sep-12 02:46:56 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xyz,5768,city1,fail,0,0,
mno,7898,city3,done,0,0,
.
.
.
.
.
tno,7459,cityx,done,0,0,
sep-1:3 01:45:56 This message is received from printer,something like this as a event.
sep-1:3 02:05:52 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xez,5718,city1,fail,0,0,
kno,7878,city3,done,0,0,
.
.
.
.
.
mno,1459,cityx,done,0,0,

Kindly, provide your views.
Thanks
Mala S

1 Solution

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype
0 Karma

mala_splunk_91
Explorer

Thanks Someson, It is working.
And pls tell me how can i show csv event in table format.

0 Karma

somesoni2
Revered Legend

Do you want to store lines starting with "sep-1..." with separate sourcetype (no csv lines) and CSV lines in different sourcetype (split)?

0 Karma

mala_splunk_91
Explorer

Yes, I want to split lines starting with "sept-12.." and csv line and store CSV lines in different sourcetype as CSV, so that i can view data in table format in UI.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...