Getting Data In

Extract some lines of an event from a CSV file and index in separate sourcetype OR Index

mala_splunk_91
Explorer

Hi guys,

Please provide your input on the below scenario.
I have some events like below. Here , I want to extract some part of event which is in CSV format and that is starting after "#" till the end of an event and store them in separate new index/sourcetype, either by using props/transforms conf OR using query.

I have questions like
1)Is there any way to split/extract some part of an event and store it in separate index/sourcetype?
2)How can I extract only CSV event part and display/View it in table format in Splunk?

Final result I need is:
Extract CSV format events separately from the below events and display it in table format OR store in lookup file.(Simply, to make it human readable).

sample.log:

sep-12 02:45:56 This message is received from printer,something like this as a eveent.
sep-12 02:46:56 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xyz,5768,city1,fail,0,0,
mno,7898,city3,done,0,0,
.
.
.
.
.
tno,7459,cityx,done,0,0,
sep-1:3 01:45:56 This message is received from printer,something like this as a event.
sep-1:3 02:05:52 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xez,5718,city1,fail,0,0,
kno,7878,city3,done,0,0,
.
.
.
.
.
mno,1459,cityx,done,0,0,

Kindly, provide your views.
Thanks
Mala S

1 Solution

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype
0 Karma

mala_splunk_91
Explorer

Thanks Someson, It is working.
And pls tell me how can i show csv event in table format.

0 Karma

somesoni2
Revered Legend

Do you want to store lines starting with "sep-1..." with separate sourcetype (no csv lines) and CSV lines in different sourcetype (split)?

0 Karma

mala_splunk_91
Explorer

Yes, I want to split lines starting with "sept-12.." and csv line and store CSV lines in different sourcetype as CSV, so that i can view data in table format in UI.

0 Karma
Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...