Knowledge Management

how do I remove the single quotes from my key value pairs,how to remove single quotes from a key value pair

jyothiadobe
New Member

I have a logs like below and this is not a JSON logs, indexing through HEC.

Key1='value1' Key2='value'

how do I remove this single quotes from value?

Regards,
Thippesh

0 Karma
1 Solution

koshyk
Super Champion

you can do in few ways
1. At indextime, completely remove this before indexing. May not be good if you want to keep "pure" source data
2. replace all single quotes in _raw message using rex | rex mode=sed "y/\=\'/\=/"| rex mode=sed "y/\'\s/\s/"
3. Do individual replacement of key-value | eval key1=replace(key1,"\'"."")

View solution in original post

0 Karma

woodcock
Esteemed Legend

Assuming that search-time is OK, like this:

|makeresults | eval "'foo'" = "'bar'"

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| foreach "'*'" [rex mode=sed field="<<FIELD>>" "s/^'// s/'$//"]
| rename "'*'" AS "*"
0 Karma

jyothiadobe
New Member

Hi woodcock,

Thanks for the answer, would like to know how I can fix this in index time so that I no need to do this for each search.

0 Karma

woodcock
Esteemed Legend

In order to do that, you MUST show us a sample event. Why do OPs so rarely share sample events?

0 Karma

koshyk
Super Champion

you can do in few ways
1. At indextime, completely remove this before indexing. May not be good if you want to keep "pure" source data
2. replace all single quotes in _raw message using rex | rex mode=sed "y/\=\'/\=/"| rex mode=sed "y/\'\s/\s/"
3. Do individual replacement of key-value | eval key1=replace(key1,"\'"."")

0 Karma

jyothiadobe
New Member

Hi koshyk,

Thanks for the answer, would like to know how I can fix this in index time so that I no need to do this for each search.

0 Karma

koshyk
Super Champion

If you want in indextime, you can do in two ways

  1. Just using props.conf

    [myQuoteSourcetype]
    SEDCMD-singleQuotedData = s/(\w+)=\'([^\']+)/\1=\2/g

Please try and test your regex properly

  1. Using props and Transforms (In case if you want to re-use in multiple sourcetypes)

In props.conf

[myQuoteSourcetype]
TRANSFORMS-removeSingleQuotedKeys = removeSingleQuotedKeys

In transforms.conf

[removeSingleQuotedKeys]
REGEX = (\w+)=(?:\'?)([^\']+)(?:\'?)\s
FORMAT = $1:$2\s
DEST_KEY = _raw

I've tested sample data Regex I've put into below link:
https://regex101.com/r/4caTBy/1

But of course, you need to fine tune your regex

(Please upvote/accept, if it worked. Cheers)

0 Karma

jotne
Builder

Take care.  Example above fails for key name like "my-key"  since \w does not include -

Also it fails for last KV, since you expect a \s and end of line, so \s? would be better.

([a-zA-Z0-9-_]+)=(?:\'?)([^\']+)(?:\'?)\s?

 

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...