How to encode a URL for a Hipchat notification alert action if there is no urlencode() function?

[floriancoulmier]

Hi all,

I have a Splunk alert configured to send Hipchat notifications. My goal is to have a link in the search to go on a dashboard prefilled with elements from the alert.

Here is the search of my alert:

index=myindex earliest=-2m@m latest=@m | where match(Subject, "(..........)") | stats count by Subject | where count > 100

Where the index contains "kind of" email logs and Subject is an email Subject.
In the alert configuration, I send a Hipchat notification using the "Subject" field :

<p><strong>Alert:  Possible spam detected</strong> - <a href="https://mysplunkurl.com/en-US/app/myapp/myform?form.Field=Subject&form.Value=$result.Subject$>View details</a></p>
<p> $result.count$ ex. / Subject: $result.Subject$</p>

However, the "Subject" field can contains special characters, such as "'[]()%&...
In that case, the link might be broken, or the browser might trigger an alert because it thinks I have a SQL injection or XSS attack.

So, I would like to encode the URL in a way that it will be opened by a browser 100% of the time. I saw there is a urldecode() function, but no urlencode()
Using replace() for each special character does not feel to be the right solution.

How do you (or would you) handle that?

Thank you

[dmarling]

I ran into this issue of doing url encoding to utf-8 in Splunk as well and I did not have the luxury of getting a custom search command installed. I have instead taken the work that @woodcock started in his answer and I have taken it to its logical conclusion:

| foreach *
    [ rex field="<<FIELD>>" mode=sed "s:%:%25:g s:\t:%09:g s:\n:%0A:g s: :%20:g s:\ :%C2%A0:g s:\!:%21:g s:\":%22:g s:\#:%23:g s:\$:%24:g s:\&:%26:g s:\':%27:g s:\(:%28:g s:\):%29:g s:\*:%2A:g s:\+:%2B:g s:\,:%2C:g s:\-:%2D:g s:\.:%2E:g s:\/:%2F:g s:\::%3A:g s:\;:%3B:g s:\<:%3C:g s:\=:%3D:g s:\>:%3E:g s:\?:%3F:g s:\@:%40:g s:\[:%5B:g s:\\\:%5C:g s:\]:%5D:g s:\^:%5E:g s:\_:%5F:g s:\`:%60:g s:\`:%E2%82%AC:g s:\{:%7B:g s:\|:%7C:g s:\}:%7D:g s:\~:%7E:g s:\:%9D:g s:\¡:%C2%A1:g s:\¢:%C2%A2:g s:\£:%C2%A3:g s:\¤:%C2%A4:g s:\¥:%C2%A5:g s:\¦:%C2%A6:g s:\§:%C2%A7:g s:\¨:%C2%A8:g s:\©:%C2%A9:g s:\ª:%C2%AA:g s:\«:%C2%AB:g s:\¬:%C2%AC:g s:\®:%C2%AE:g s:\¯:%C2%AF:g s:\°:%C2%B0:g s:\±:%C2%B1:g s:\²:%C2%B2:g s:\³:%C2%B3:g s:\´:%C2%B4:g s:\µ:%C2%B5:g s:\¶:%C2%B6:g s:\·:%C2%B7:g s:\¸:%C2%B8:g s:\¹:%C2%B9:g s:\º:%C2%BA:g s:\»:%C2%BB:g s:\¼:%C2%BC:g s:\½:%C2%BD:g s:\¾:%C2%BE:g s:\¿:%C2%BF:g s:\À:%C3%80:g s:\Á:%C3%81:g s:\Â:%C3%82:g s:\Ã:%C3%83:g s:\Ä:%C3%84:g s:\Å:%C3%85:g s:\Æ:%C3%86:g s:\Ç:%C3%87:g s:\È:%C3%88:g s:\É:%C3%89:g s:\Ê:%C3%8A:g s:\Ë:%C3%8B:g s:\Ì:%C3%8C:g s:\Í:%C3%8D:g s:\Î:%C3%8E:g s:\Ï:%C3%8F:g s:\Ð:%C3%90:g s:\Ñ:%C3%91:g s:\Ò:%C3%92:g s:\Ó:%C3%93:g s:\Ô:%C3%94:g s:\Õ:%C3%95:g s:\Ö:%C3%96:g s:\×:%C3%97:g s:\Ø:%C3%98:g s:\Ù:%C3%99:g s:\Ú:%C3%9A:g s:\Û:%C3%9B:g s:\Ü:%C3%9C:g s:\Ý:%C3%9D:g s:\Þ:%C3%9E:g s:\ß:%C3%9F:g s:\à:%C3%A0:g s:\á:%C3%A1:g s:\â:%C3%A2:g s:\ã:%C3%A3:g s:\ä:%C3%A4:g s:\å:%C3%A5:g s:\æ:%C3%A6:g s:\ç:%C3%A7:g s:\è:%C3%A8:g s:\é:%C3%A9:g s:\ê:%C3%AA:g s:\ë:%C3%AB:g s:\ì:%C3%AC:g s:\í:%C3%AD:g s:\î:%C3%AE:g s:\ï:%C3%AF:g s:\ð:%C3%B0:g s:\ñ:%C3%B1:g s:\ò:%C3%B2:g s:\ó:%C3%B3:g s:\ô:%C3%B4:g s:\õ:%C3%B5:g s:\ö:%C3%B6:g s:\÷:%C3%B7:g s:\ø:%C3%B8:g s:\ù:%C3%B9:g s:\ú:%C3%BA:g s:\û:%C3%BB:g s:\ü:%C3%BC:g s:\ý:%C3%BD:g s:\þ:%C3%BE:g s:\ÿ:%C3%BF:g s:\Œ:%C5%92:g s:\œ:%C5%93:g s:\Š:%C5%A0:g s:\š:%C5%A1:g s:\Ÿ:%C5%B8:g s:\Ž:%C5%BD:g s:\ž:%C5%BE:g s:\ƒ:%C6%92:g s:\ˆ:%CB%86:g s:\˜:%CB%9C:g s:\–:%E2%80%93:g s:\—:%E2%80%94:g s:\‘:%E2%80%98:g s:\’:%E2%80%99:g s:\‚:%E2%80%9A:g s:\“:%E2%80%9C:g s:\”:%E2%80%9D:g s:\„:%E2%80%9E:g s:\†:%E2%80%A0:g s:\‡:%E2%80%A1:g s:\•:%E2%80%A2:g s:\…:%E2%80%A6:g s:\‰:%E2%80%B0:g s:\‹:%E2%80%B9:g s:\›:%E2%80:g s:\™:%E2%84:g"]

I have tons of things escaped unnecessarily in the regular expression as I took a url encode reference table, dumped it into Splunk and then used it to generate this search for me and I didn't want to really figure out which special regular expression characters required escaping:

| makeresults count=1
| eval data="
%   %25 %25
    %20 %20
!   %21 %21
\"  %22 %22
#   %23 %23
$   %24 %24
&   %26 %26
'   %27 %27
(   %28 %28
)   %29 %29
*   %2A %2A
+   %2B %2B
,   %2C %2C
-   %2D %2D
.   %2E %2E
/   %2F %2F
0   %30 %30
1   %31 %31
2   %32 %32
3   %33 %33
4   %34 %34
5   %35 %35
6   %36 %36
7   %37 %37
8   %38 %38
9   %39 %39
:   %3A %3A
;   %3B %3B
<   %3C %3C
=   %3D %3D
>   %3E %3E
?   %3F %3F
@   %40 %40
A   %41 %41
B   %42 %42
C   %43 %43
D   %44 %44
E   %45 %45
F   %46 %46
G   %47 %47
H   %48 %48
I   %49 %49
J   %4A %4A
K   %4B %4B
L   %4C %4C
M   %4D %4D
N   %4E %4E
O   %4F %4F
P   %50 %50
Q   %51 %51
R   %52 %52
S   %53 %53
T   %54 %54
U   %55 %55
V   %56 %56
W   %57 %57
X   %58 %58
Y   %59 %59
Z   %5A %5A
[   %5B %5B
\   %5C %5C
]   %5D %5D
^   %5E %5E
_   %5F %5F
`   %60 %60
a   %61 %61
b   %62 %62
c   %63 %63
d   %64 %64
e   %65 %65
f   %66 %66
g   %67 %67
h   %68 %68
i   %69 %69
j   %6A %6A
k   %6B %6B
l   %6C %6C
m   %6D %6D
n   %6E %6E
o   %6F %6F
p   %70 %70
q   %71 %71
r   %72 %72
s   %73 %73
t   %74 %74
u   %75 %75
v   %76 %76
w   %77 %77
x   %78 %78
y   %79 %79
z   %7A %7A
{   %7B %7B
|   %7C %7C
}   %7D %7D
~   %7E %7E
    %7F %7F
`   %80 %E2%82%AC
‚ %82 %E2%80%9A
ƒ  %83 %C6%92
„ %84 %E2%80%9E
… %85 %E2%80%A6
† %86 %E2%80%A0
‡ %87 %E2%80%A1
ˆ  %88 %CB%86
‰ %89 %E2%80%B0
Š  %8A %C5%A0
‹ %8B %E2%80%B9
Π %8C %C5%92
Ž  %8E %C5%BD
‘ %91 %E2%80%98
’ %92 %E2%80%99
“ %93 %E2%80%9C
” %94 %E2%80%9D
• %95 %E2%80%A2
– %96 %E2%80%93
— %97 %E2%80%94
˜  %98 %CB%9C
™ %99 %E2%84
š  %9A %C5%A1
› %9B %E2%80
œ  %9C %C5%93
  %9D %9D
ž  %9E %C5%BE
Ÿ  %9F %C5%B8
    %A0 %C2%A0
¡  %A1 %C2%A1
¢  %A2 %C2%A2
£  %A3 %C2%A3
¤  %A4 %C2%A4
¥  %A5 %C2%A5
¦  %A6 %C2%A6
§  %A7 %C2%A7
¨  %A8 %C2%A8
©  %A9 %C2%A9
ª  %AA %C2%AA
«  %AB %C2%AB
¬  %AC %C2%AC
®  %AE %C2%AE
¯  %AF %C2%AF
°  %B0 %C2%B0
±  %B1 %C2%B1
²  %B2 %C2%B2
³  %B3 %C2%B3
´  %B4 %C2%B4
µ  %B5 %C2%B5
¶  %B6 %C2%B6
·  %B7 %C2%B7
¸  %B8 %C2%B8
¹  %B9 %C2%B9
º  %BA %C2%BA
»  %BB %C2%BB
¼  %BC %C2%BC
½  %BD %C2%BD
¾  %BE %C2%BE
¿  %BF %C2%BF
À  %C0 %C3%80
Á  %C1 %C3%81
  %C2 %C3%82
à %C3 %C3%83
Ä  %C4 %C3%84
Å  %C5 %C3%85
Æ  %C6 %C3%86
Ç  %C7 %C3%87
È  %C8 %C3%88
É  %C9 %C3%89
Ê  %CA %C3%8A
Ë  %CB %C3%8B
Ì  %CC %C3%8C
Í  %CD %C3%8D
Î  %CE %C3%8E
Ï  %CF %C3%8F
Ð  %D0 %C3%90
Ñ  %D1 %C3%91
Ò  %D2 %C3%92
Ó  %D3 %C3%93
Ô  %D4 %C3%94
Õ  %D5 %C3%95
Ö  %D6 %C3%96
×  %D7 %C3%97
Ø  %D8 %C3%98
Ù  %D9 %C3%99
Ú  %DA %C3%9A
Û  %DB %C3%9B
Ü  %DC %C3%9C
Ý  %DD %C3%9D
Þ  %DE %C3%9E
ß  %DF %C3%9F
à  %E0 %C3%A0
á  %E1 %C3%A1
â  %E2 %C3%A2
ã  %E3 %C3%A3
ä  %E4 %C3%A4
å  %E5 %C3%A5
æ  %E6 %C3%A6
ç  %E7 %C3%A7
è  %E8 %C3%A8
é  %E9 %C3%A9
ê  %EA %C3%AA
ë  %EB %C3%AB
ì  %EC %C3%AC
í  %ED %C3%AD
î  %EE %C3%AE
ï  %EF %C3%AF
ð  %F0 %C3%B0
ñ  %F1 %C3%B1
ò  %F2 %C3%B2
ó  %F3 %C3%B3
ô  %F4 %C3%B4
õ  %F5 %C3%B5
ö  %F6 %C3%B6
÷  %F7 %C3%B7
ø  %F8 %C3%B8
ù  %F9 %C3%B9
ú  %FA %C3%BA
û  %FB %C3%BB
ü  %FC %C3%BC
ý  %FD %C3%BD
þ  %FE %C3%BE
ÿ  %FF %C3%BF"
| rex max_match=0 field=data "(?<data>[^\n]+)"
| mvexpand data
| rex field=data "^(?<Character>.*)\t(?<dontcare>.*)\t(?<UTF8>.*)$"
| table Character UTF8
| where NOT match(Character, "[A-Za-z0-9]")
| search Character=*
| eval rex="s:\\".Character.":".UTF8.":g"
| stats values(rex) as rex
| eval rex=mvjoin(rex, " ")

[jkat54]

I like it!

[woodcock]

You can also do this:

... | rex field=fieldToEncode mode=sed "s:%:%25:g s: :%20:g s:<:%3C:g s:>:%3E:g s:#:%23:g s:{:%7B:g s:}:%7D:g s:\|:%7C:g s:\\\:%5C:g s:\^:%5E:g s:~:%7E:g s:\[:%5B:g s:\]:%5D:g s:\`:%60:g s:;:%3B:g s:/:%2F:g s:\?:%3F:g s/:/%3A/g s:@:%40:g s:=:%3D:g s:&:%26:g s:\$:%24:g s:\!:%21:g s:\*:%2A:g"

[woodcock]

It makes a nice macro.

[jkat54 · Accepted solution]

too late...

import urllib
import splunk.Intersplunk
import splunk.mining.dcutils as dcu
import traceback

# Setup logging/logger
logger = dcu.getLogger()

def encode(s):
  try:
    return urllib.quote_plus(s)

  except Exception, e:
    stack =  traceback.format_exc()
    splunk.Intersplunk.generateErrorResults(str(e))
    logger.error(str(e) + ". Traceback: " + str(stack))

def execute():
  try:
    # get the keywords and options passed to this command
    keywords, options = splunk.Intersplunk.getKeywordsAndOptions()

    # get the previous search results
    results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

    # if no keywords, send error results through
    if len(keywords) == 0:
      results = []
      results.append({"error":"syntax: urlencode <field_1> <field_2> <field_n> ..."})
      results.append({"error":"example: urlencode punct"})

    # else encode the fields provided
    if len(keywords) >= 1:
     for keyword in keywords:
       for result in results: 
        result[keyword] = encode(result[keyword])

    # return the results 
    results.sort()
    splunk.Intersplunk.outputResults(results)

  except Exception, e:
    stack =  traceback.format_exc()
    splunk.Intersplunk.generateErrorResults(str(e))
    logger.error(str(e) + ". Traceback: " + str(stack))

if __name__ == '__main__':
    execute()

To use this... save it as urlencode.py, put it in your app's bin directory, and then create a commands.conf in your app's local directory and add the following:

[urlencode]
filename = urlencode.py

Then restart, and you should be able to use it in your search like this:

... | urlencode  <field1> <field2> <field_n>  | table field1 field2 field_n

[gabriel_vasseur]

Do you have a version compatible with python3?

[jkat54]

as FYI, i've added this search command to my toolkit app for those of you who want to take the easy route to get this command: https://splunkbase.splunk.com/app/3265/

[woodcock]

Evidently this is no longer in that app and has been moved to the premium version here:
See https://splunkbase.splunk.com/app/3420

[jkat54]

The web tools app is not a premium app. It was for about a month or two but that was nearly a year ago.

[floriancoulmier]

Many thanks!

[jkat54]

You're very welcome. Feel free to come find me anytime you need custom search commands or apps. I take requests.

[jkat54]

If you're interested, i can write a urlencode command...