Knowledge Management

how do I remove the single quotes from my key value pairs,how to remove single quotes from a key value pair

jyothiadobe
New Member

I have a logs like below and this is not a JSON logs, indexing through HEC.

Key1='value1' Key2='value'

how do I remove this single quotes from value?

Regards,
Thippesh

0 Karma
1 Solution

koshyk
Super Champion

you can do in few ways
1. At indextime, completely remove this before indexing. May not be good if you want to keep "pure" source data
2. replace all single quotes in _raw message using rex | rex mode=sed "y/\=\'/\=/"| rex mode=sed "y/\'\s/\s/"
3. Do individual replacement of key-value | eval key1=replace(key1,"\'"."")

View solution in original post

0 Karma

woodcock
Esteemed Legend

Assuming that search-time is OK, like this:

|makeresults | eval "'foo'" = "'bar'"

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| foreach "'*'" [rex mode=sed field="<<FIELD>>" "s/^'// s/'$//"]
| rename "'*'" AS "*"
0 Karma

jyothiadobe
New Member

Hi woodcock,

Thanks for the answer, would like to know how I can fix this in index time so that I no need to do this for each search.

0 Karma

woodcock
Esteemed Legend

In order to do that, you MUST show us a sample event. Why do OPs so rarely share sample events?

0 Karma

koshyk
Super Champion

you can do in few ways
1. At indextime, completely remove this before indexing. May not be good if you want to keep "pure" source data
2. replace all single quotes in _raw message using rex | rex mode=sed "y/\=\'/\=/"| rex mode=sed "y/\'\s/\s/"
3. Do individual replacement of key-value | eval key1=replace(key1,"\'"."")

0 Karma

jyothiadobe
New Member

Hi koshyk,

Thanks for the answer, would like to know how I can fix this in index time so that I no need to do this for each search.

0 Karma

koshyk
Super Champion

If you want in indextime, you can do in two ways

  1. Just using props.conf

    [myQuoteSourcetype]
    SEDCMD-singleQuotedData = s/(\w+)=\'([^\']+)/\1=\2/g

Please try and test your regex properly

  1. Using props and Transforms (In case if you want to re-use in multiple sourcetypes)

In props.conf

[myQuoteSourcetype]
TRANSFORMS-removeSingleQuotedKeys = removeSingleQuotedKeys

In transforms.conf

[removeSingleQuotedKeys]
REGEX = (\w+)=(?:\'?)([^\']+)(?:\'?)\s
FORMAT = $1:$2\s
DEST_KEY = _raw

I've tested sample data Regex I've put into below link:
https://regex101.com/r/4caTBy/1

But of course, you need to fine tune your regex

(Please upvote/accept, if it worked. Cheers)

0 Karma

jotne
Builder

Take care.  Example above fails for key name like "my-key"  since \w does not include -

Also it fails for last KV, since you expect a \s and end of line, so \s? would be better.

([a-zA-Z0-9-_]+)=(?:\'?)([^\']+)(?:\'?)\s?

 

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...