topic Re: Help with Regex in Splunk Enterprise

Help with Regex

ak9092 — Mon, 20 Sep 2021 20:40:30 GMT

Hello,

I want to remove all the back slashes and double quotes from following fields -

conn=\"pass\""

ip=\"10.23.22.1\""

I am trying to extract with EVAL-conn = replace(conn,"\\\\(.),"") and EVAL-ip= replace(ip,"\\\\(.),"") in my props.conf but it is not removing the last double quotes and give me following results -

conn=pass"

ip=10.23.22.1"

Results I want : conn=pass & ip=10.23.22.1

Can someone please help/guide me with this extraction.

Thanks in Advance

Re: Help with Regex

PickleRick — Thu, 16 Sep 2021 20:30:19 GMT

Are those your literal evals? Because they seem to be syntactically bad.

EVAL-conn = replace(conn,"\\\\","") EVAL-ip= replace(ip,"\\\\","")

That looks more correct.

I don't quite follow what you want to do with the loc though.

Re: Help with Regex

ak9092 — Fri, 17 Sep 2021 08:26:01 GMT

@PickleRick thank you for your response.

The EVAL statements shared by you and the one which I have shared above both are working fine only if the field is like conn="\success\" , but not working if there is an additional double quotes at the end like conn=\"pass\"". So in both; the results looks like -

Conn=pass", what I want is; conn=pass

Re: Help with Regex

PickleRick — Fri, 17 Sep 2021 13:42:06 GMT

OK. But the evals only remove backslashes. So you end up with
conn="pass""

I suppose (I don't know your sourcetype and the rest of props/transforms) that it's getting interpreted as key-value and the quotes are not getting removed by any evals but are dropped as part of K/V parsing. You'd need another eval to drop the quotes.

Re: Help with Regex

ak9092 — Mon, 20 Sep 2021 18:09:04 GMT

@PickleRick The eval which I am using removes the double quotes too, but its just not removing the additional quotes which is coming in some events.

In below run anywhere search for conn2 field you will see one double quote remaining at the end, which I don't want.

|makeresults

|eval conn="\\\"08/24/2021\\\"\""

|eval conn2=replace(conn,"\\\\(.)","")

Also there is no such any specific extraction in my props.conf and it roughly looks something like below

[mysourcetype]

EVAL-conn = replace(conn,"\\\\(.),"")

With following regex, I am able to replace all the backslashes and double quotes the way I want but I am unable to make it work with Splunk replace command.

https://regex101.com/r/glIfKz/1

@niketn , @ITWhisperer, @bowesmana

Thanks in advance.

Re: Help with Regex

ashvinpandey — Mon, 20 Sep 2021 18:37:44 GMT

@ak9092 Use the below query in your search:

| regex field=conn "\\"(?P<conn>\w+)" | regex field=ip "\\"(?P<conn>\d+.\d+.\d+.\d+)"

Also, If this reply helps you, an upvote would be appreciated.

Re: Help with Regex

ak9092 — Mon, 20 Sep 2021 19:17:04 GMT

@ashvinpandey , thank you for your response.

I am trying to achieve this using props.conf.

Also the data in "conn" field can be a mix of IP , string and other special characters just like the "loc" field which you can see in the question.

Re: Help with Regex

PickleRick — Mon, 20 Sep 2021 19:36:23 GMT

Well... splunk sometimes guesses stuff. Sometimes too much.

Let me show you. I push an event to a HEC input on my home splunk server. The data is as follows:

{"event":"<13>1 2021-09-20T21:22:30.495495+02:00 test=\"test\"","host":"test1","index":"test1","sourcetype":"syslog"}

It gets ingested into the index

<13>1 2021-09-20T21:21:30.495495+02:00 test="test"

When I look into the event I see that the "test" field is being parsed out because it's a key-value pair so Splunk automaticaly recognizes it.

To make things even more interesting, in case of plain syslog sourcetype it gets the value from between the quotes even if you have additional unbalanced quote at the end (as in your example). So I suppose your K/V pairs might also be being parsed automaticaly.

Re: Help with Regex

ak9092 — Mon, 20 Sep 2021 20:32:22 GMT

@PickleRick Yes , there are some events in which it is extracting exactly the way you mentioned.

But then there are some event which are coming as a K/V pairs inside another field,

So the extraction which I am doing is for the field, where i am getting data like -

field1="field2=\"some_string_value\", field3=\"some_path_value\", field4=\"again_some_value\", ... fieldN=\"valueN\""

So now from field1 to field4 everything is getting formatted perfectly fine using EVAL-field = replace(field,"\\\\(.),"")

The issue comes in "fieldN" where an extra double quotes is added to close the quotes which is opened at field1.

Now the problem is my fieldN can be any any field and can contain multiple back slashes , double quotes and any string, so need some robust solution here which will work fine on any kind of field value.

If somehow I can make the below regex work with replace command , then i guess it should solve my problem, but unfortunately its giving me error with something like - "|ev...{snipped} {errorcontext = ce(conn1,[\\"]+],"")}'.

https://regex101.com/r/glIfKz/1

|makeresults | eval conn1="\\\"08/24/2021\\\"\"" |eval conn=replace(conn1,[\\"]+],"")

Re: Help with Regex

bowesmana — Mon, 20 Sep 2021 22:20:31 GMT

You are creating a calculated field right through props.conf, so rather than use replace why not use left and right trim and trim off backslash and quotes from the ends, e.g.

ip=ltrim(rtrim(ip,"\\\""),"\\\"")

which trims off the backslash \\ and quotes \" from both ends of the string.

You can also use replace like this

conn=replace(conn,"[\\\\\"]","")

which will replace all backslash and quotes in the entire string, regardless of where they are.

Re: Help with Regex

ak9092 — Tue, 21 Sep 2021 19:18:01 GMT

Just the replace i was looking for. Thanks Much!! @bowesmana