Getting Data In

How to extract multiple fields using the Extract property in props.conf?

greggz
Communicator

So I'm trying to extract multiple fields using the Extract property in props.conf
The source file looks like

my.prop.1=1
my.prop.2=2
my.prop.3=3
my.prop.4=4

And I want what EACH prop becomes a field. HENCE I did not used a prefix in the regex.

[my_sourcetype]
LINE_BREAKER = ((?!))
DATETIME_CONFIG=CURRENT
TRUNCATE=100000
KV_MODE = none
EXTRACT-watt_grouping = (?m)(.+?)=(.*)

I already used multiple variations of this regex and nothing seems to work. Is there anything blocking the field extraction ?

EDIT:
At search time I provided the conrresponding pipe and it seems to extract quite nicely: | extract pairdelim=",", kvdelim="=", auto=f, limit=500, mv_add=t .. Anyone knows how to make this work in transforms.conf ? I tried it with DELIMS = "\n","=" but it breaks in a specific property prop.256 = ?'- \#&@^\!%*\:$./\;,~+=)(|}{][><` .. It extracts the previous 255 props quite nicely though

0 Karma

masonmorales
Influencer

Try this instead:

props.conf

[my_sourcetype]
LINE_BREAKER = ((?!))
DATETIME_CONFIG=CURRENT
TRUNCATE=100000
KV_MODE = none
REPORT-my_fields = multi_extract

transforms.conf

[multi_extract]
REGEX = (?m)(.+?)=(.*)
FORMAT = $1::$2
MV_ADD = true
0 Karma

greggz
Communicator

@masonmorales can you provide support to your own answer based on the feedback I gave you ?

0 Karma

masonmorales
Influencer

Are there always only 4 lines? If so you could just use 4 extract statements that are more specific.

0 Karma

greggz
Communicator

No.. they're above 400. This is the reason I don't use KV_MODE=AUTO because it has a 100 limit max extractions. The can be alphanumerical aswell or have only symbols

0 Karma

greggz
Communicator

This doesnt help.. It just clumps them all together. I want each line to become a new field with a given value.

0 Karma

greggz
Communicator

It just makes my.prop.1 = "1 my.prop.2 = 2 my.prop.3 = 3 ..."

0 Karma

greggz
Communicator

Thanks, let me try it out

0 Karma

somesoni2
Revered Legend

Your source, is it one line per event or all those properties appear in same event? What are the fields that are automatically extracted by Splunk?

0 Karma

greggz
Communicator

None, because I made it KV_MODE=none. No, All lines come in the same event. I purposedly made it so with the LINE_BREAKER = ((?!))

0 Karma

greggz
Communicator

@somesoni2 ♦ Do you know how this should be done ?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Manage Edge Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Edge Processor, onboard one instance, deploy a pipeline ...

Splunk Brings Proof of Control to Digital Asset Transfers

Digital assets now sit inside the core of financial services. HSBC launched its Tokenized Deposit Service in ...

Manage Ingest Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Ingest Processor pipeline, deploy it to the ...