Getting Data In

Extract some lines of an event from a CSV file and index in separate sourcetype OR Index

mala_splunk_91
Explorer

Hi guys,

Please provide your input on the below scenario.
I have some events like below. Here , I want to extract some part of event which is in CSV format and that is starting after "#" till the end of an event and store them in separate new index/sourcetype, either by using props/transforms conf OR using query.

I have questions like
1)Is there any way to split/extract some part of an event and store it in separate index/sourcetype?
2)How can I extract only CSV event part and display/View it in table format in Splunk?

Final result I need is:
Extract CSV format events separately from the below events and display it in table format OR store in lookup file.(Simply, to make it human readable).

sample.log:

sep-12 02:45:56 This message is received from printer,something like this as a eveent.
sep-12 02:46:56 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xyz,5768,city1,fail,0,0,
mno,7898,city3,done,0,0,
.
.
.
.
.
tno,7459,cityx,done,0,0,
sep-1:3 01:45:56 This message is received from printer,something like this as a event.
sep-1:3 02:05:52 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xez,5718,city1,fail,0,0,
kno,7878,city3,done,0,0,
.
.
.
.
.
mno,1459,cityx,done,0,0,

Kindly, provide your views.
Thanks
Mala S

1 Solution

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype
0 Karma

mala_splunk_91
Explorer

Thanks Someson, It is working.
And pls tell me how can i show csv event in table format.

0 Karma

somesoni2
Revered Legend

Do you want to store lines starting with "sep-1..." with separate sourcetype (no csv lines) and CSV lines in different sourcetype (split)?

0 Karma

mala_splunk_91
Explorer

Yes, I want to split lines starting with "sept-12.." and csv line and store CSV lines in different sourcetype as CSV, so that i can view data in table format in UI.

0 Karma
Get Updates on the Splunk Community!

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...