Splunk Search

How to encode a URL for a Hipchat notification alert action if there is no urlencode() function?

floriancoulmier
Engager

Hi all,

I have a Splunk alert configured to send Hipchat notifications. My goal is to have a link in the search to go on a dashboard prefilled with elements from the alert.

Here is the search of my alert:

index=myindex earliest=-2m@m latest=@m | where match(Subject, "(..........)") | stats count by Subject | where count > 100

Where the index contains "kind of" email logs and Subject is an email Subject.
In the alert configuration, I send a Hipchat notification using the "Subject" field :

<p><strong>Alert:  Possible spam detected</strong> - <a href="https://mysplunkurl.com/en-US/app/myapp/myform?form.Field=Subject&form.Value=$result.Subject$>View details</a></p>
<p> $result.count$ ex. / Subject: $result.Subject$</p>

However, the "Subject" field can contains special characters, such as "'[]()%&...
In that case, the link might be broken, or the browser might trigger an alert because it thinks I have a SQL injection or XSS attack.

So, I would like to encode the URL in a way that it will be opened by a browser 100% of the time. I saw there is a urldecode() function, but no urlencode()
Using replace() for each special character does not feel to be the right solution.

How do you (or would you) handle that?

Thank you

1 Solution

jkat54
SplunkTrust
SplunkTrust

too late...

import urllib
import splunk.Intersplunk
import splunk.mining.dcutils as dcu
import traceback

# Setup logging/logger
logger = dcu.getLogger()

def encode(s):
  try:
    return urllib.quote_plus(s)

  except Exception, e:
    stack =  traceback.format_exc()
    splunk.Intersplunk.generateErrorResults(str(e))
    logger.error(str(e) + ". Traceback: " + str(stack))

def execute():
  try:
    # get the keywords and options passed to this command
    keywords, options = splunk.Intersplunk.getKeywordsAndOptions()

    # get the previous search results
    results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

    # if no keywords, send error results through
    if len(keywords) == 0:
      results = []
      results.append({"error":"syntax: urlencode <field_1> <field_2> <field_n> ..."})
      results.append({"error":"example: urlencode punct"})

    # else encode the fields provided
    if len(keywords) >= 1:
     for keyword in keywords:
       for result in results: 
        result[keyword] = encode(result[keyword])

    # return the results 
    results.sort()
    splunk.Intersplunk.outputResults(results)

  except Exception, e:
    stack =  traceback.format_exc()
    splunk.Intersplunk.generateErrorResults(str(e))
    logger.error(str(e) + ". Traceback: " + str(stack))

if __name__ == '__main__':
    execute()

To use this... save it as urlencode.py, put it in your app's bin directory, and then create a commands.conf in your app's local directory and add the following:

[urlencode]
filename = urlencode.py

Then restart, and you should be able to use it in your search like this:

... | urlencode  <field1> <field2> <field_n>  | table field1 field2 field_n

alt text

View solution in original post

dmarling
Builder

I ran into this issue of doing url encoding to utf-8 in Splunk as well and I did not have the luxury of getting a custom search command installed. I have instead taken the work that @woodcock started in his answer and I have taken it to its logical conclusion:

| foreach *
    [ rex field="<<FIELD>>" mode=sed "s:%:%25:g s:\t:%09:g s:\n:%0A:g s: :%20:g s:\ :%C2%A0:g s:\!:%21:g s:\":%22:g s:\#:%23:g s:\$:%24:g s:\&:%26:g s:\':%27:g s:\(:%28:g s:\):%29:g s:\*:%2A:g s:\+:%2B:g s:\,:%2C:g s:\-:%2D:g s:\.:%2E:g s:\/:%2F:g s:\::%3A:g s:\;:%3B:g s:\<:%3C:g s:\=:%3D:g s:\>:%3E:g s:\?:%3F:g s:\@:%40:g s:\[:%5B:g s:\\\:%5C:g s:\]:%5D:g s:\^:%5E:g s:\_:%5F:g s:\`:%60:g s:\`:%E2%82%AC:g s:\{:%7B:g s:\|:%7C:g s:\}:%7D:g s:\~:%7E:g s:\:%9D:g s:\¡:%C2%A1:g s:\¢:%C2%A2:g s:\£:%C2%A3:g s:\¤:%C2%A4:g s:\¥:%C2%A5:g s:\¦:%C2%A6:g s:\§:%C2%A7:g s:\¨:%C2%A8:g s:\©:%C2%A9:g s:\ª:%C2%AA:g s:\«:%C2%AB:g s:\¬:%C2%AC:g s:\®:%C2%AE:g s:\¯:%C2%AF:g s:\°:%C2%B0:g s:\±:%C2%B1:g s:\²:%C2%B2:g s:\³:%C2%B3:g s:\´:%C2%B4:g s:\µ:%C2%B5:g s:\¶:%C2%B6:g s:\·:%C2%B7:g s:\¸:%C2%B8:g s:\¹:%C2%B9:g s:\º:%C2%BA:g s:\»:%C2%BB:g s:\¼:%C2%BC:g s:\½:%C2%BD:g s:\¾:%C2%BE:g s:\¿:%C2%BF:g s:\À:%C3%80:g s:\Á:%C3%81:g s:\Â:%C3%82:g s:\Ã:%C3%83:g s:\Ä:%C3%84:g s:\Å:%C3%85:g s:\Æ:%C3%86:g s:\Ç:%C3%87:g s:\È:%C3%88:g s:\É:%C3%89:g s:\Ê:%C3%8A:g s:\Ë:%C3%8B:g s:\Ì:%C3%8C:g s:\Í:%C3%8D:g s:\Î:%C3%8E:g s:\Ï:%C3%8F:g s:\Ð:%C3%90:g s:\Ñ:%C3%91:g s:\Ò:%C3%92:g s:\Ó:%C3%93:g s:\Ô:%C3%94:g s:\Õ:%C3%95:g s:\Ö:%C3%96:g s:\×:%C3%97:g s:\Ø:%C3%98:g s:\Ù:%C3%99:g s:\Ú:%C3%9A:g s:\Û:%C3%9B:g s:\Ü:%C3%9C:g s:\Ý:%C3%9D:g s:\Þ:%C3%9E:g s:\ß:%C3%9F:g s:\à:%C3%A0:g s:\á:%C3%A1:g s:\â:%C3%A2:g s:\ã:%C3%A3:g s:\ä:%C3%A4:g s:\å:%C3%A5:g s:\æ:%C3%A6:g s:\ç:%C3%A7:g s:\è:%C3%A8:g s:\é:%C3%A9:g s:\ê:%C3%AA:g s:\ë:%C3%AB:g s:\ì:%C3%AC:g s:\í:%C3%AD:g s:\î:%C3%AE:g s:\ï:%C3%AF:g s:\ð:%C3%B0:g s:\ñ:%C3%B1:g s:\ò:%C3%B2:g s:\ó:%C3%B3:g s:\ô:%C3%B4:g s:\õ:%C3%B5:g s:\ö:%C3%B6:g s:\÷:%C3%B7:g s:\ø:%C3%B8:g s:\ù:%C3%B9:g s:\ú:%C3%BA:g s:\û:%C3%BB:g s:\ü:%C3%BC:g s:\ý:%C3%BD:g s:\þ:%C3%BE:g s:\ÿ:%C3%BF:g s:\Œ:%C5%92:g s:\œ:%C5%93:g s:\Š:%C5%A0:g s:\š:%C5%A1:g s:\Ÿ:%C5%B8:g s:\Ž:%C5%BD:g s:\ž:%C5%BE:g s:\ƒ:%C6%92:g s:\ˆ:%CB%86:g s:\˜:%CB%9C:g s:\–:%E2%80%93:g s:\—:%E2%80%94:g s:\‘:%E2%80%98:g s:\’:%E2%80%99:g s:\‚:%E2%80%9A:g s:\“:%E2%80%9C:g s:\”:%E2%80%9D:g s:\„:%E2%80%9E:g s:\†:%E2%80%A0:g s:\‡:%E2%80%A1:g s:\•:%E2%80%A2:g s:\…:%E2%80%A6:g s:\‰:%E2%80%B0:g s:\‹:%E2%80%B9:g s:\›:%E2%80:g s:\™:%E2%84:g"]

I have tons of things escaped unnecessarily in the regular expression as I took a url encode reference table, dumped it into Splunk and then used it to generate this search for me and I didn't want to really figure out which special regular expression characters required escaping:

| makeresults count=1
| eval data="
%   %25 %25
    %20 %20
!   %21 %21
\"  %22 %22
#   %23 %23
$   %24 %24
&   %26 %26
'   %27 %27
(   %28 %28
)   %29 %29
*   %2A %2A
+   %2B %2B
,   %2C %2C
-   %2D %2D
.   %2E %2E
/   %2F %2F
0   %30 %30
1   %31 %31
2   %32 %32
3   %33 %33
4   %34 %34
5   %35 %35
6   %36 %36
7   %37 %37
8   %38 %38
9   %39 %39
:   %3A %3A
;   %3B %3B
<   %3C %3C
=   %3D %3D
>   %3E %3E
?   %3F %3F
@   %40 %40
A   %41 %41
B   %42 %42
C   %43 %43
D   %44 %44
E   %45 %45
F   %46 %46
G   %47 %47
H   %48 %48
I   %49 %49
J   %4A %4A
K   %4B %4B
L   %4C %4C
M   %4D %4D
N   %4E %4E
O   %4F %4F
P   %50 %50
Q   %51 %51
R   %52 %52
S   %53 %53
T   %54 %54
U   %55 %55
V   %56 %56
W   %57 %57
X   %58 %58
Y   %59 %59
Z   %5A %5A
[   %5B %5B
\   %5C %5C
]   %5D %5D
^   %5E %5E
_   %5F %5F
`   %60 %60
a   %61 %61
b   %62 %62
c   %63 %63
d   %64 %64
e   %65 %65
f   %66 %66
g   %67 %67
h   %68 %68
i   %69 %69
j   %6A %6A
k   %6B %6B
l   %6C %6C
m   %6D %6D
n   %6E %6E
o   %6F %6F
p   %70 %70
q   %71 %71
r   %72 %72
s   %73 %73
t   %74 %74
u   %75 %75
v   %76 %76
w   %77 %77
x   %78 %78
y   %79 %79
z   %7A %7A
{   %7B %7B
|   %7C %7C
}   %7D %7D
~   %7E %7E
    %7F %7F
`   %80 %E2%82%AC
‚ %82 %E2%80%9A
ƒ  %83 %C6%92
„ %84 %E2%80%9E
… %85 %E2%80%A6
† %86 %E2%80%A0
‡ %87 %E2%80%A1
ˆ  %88 %CB%86
‰ %89 %E2%80%B0
Š  %8A %C5%A0
‹ %8B %E2%80%B9
Π %8C %C5%92
Ž  %8E %C5%BD
‘ %91 %E2%80%98
’ %92 %E2%80%99
“ %93 %E2%80%9C
” %94 %E2%80%9D
• %95 %E2%80%A2
– %96 %E2%80%93
— %97 %E2%80%94
˜  %98 %CB%9C
™ %99 %E2%84
š  %9A %C5%A1
› %9B %E2%80
œ  %9C %C5%93
  %9D %9D
ž  %9E %C5%BE
Ÿ  %9F %C5%B8
    %A0 %C2%A0
¡  %A1 %C2%A1
¢  %A2 %C2%A2
£  %A3 %C2%A3
¤  %A4 %C2%A4
¥  %A5 %C2%A5
¦  %A6 %C2%A6
§  %A7 %C2%A7
¨  %A8 %C2%A8
©  %A9 %C2%A9
ª  %AA %C2%AA
«  %AB %C2%AB
¬  %AC %C2%AC
®  %AE %C2%AE
¯  %AF %C2%AF
°  %B0 %C2%B0
±  %B1 %C2%B1
²  %B2 %C2%B2
³  %B3 %C2%B3
´  %B4 %C2%B4
µ  %B5 %C2%B5
¶  %B6 %C2%B6
·  %B7 %C2%B7
¸  %B8 %C2%B8
¹  %B9 %C2%B9
º  %BA %C2%BA
»  %BB %C2%BB
¼  %BC %C2%BC
½  %BD %C2%BD
¾  %BE %C2%BE
¿  %BF %C2%BF
À  %C0 %C3%80
Á  %C1 %C3%81
  %C2 %C3%82
à %C3 %C3%83
Ä  %C4 %C3%84
Å  %C5 %C3%85
Æ  %C6 %C3%86
Ç  %C7 %C3%87
È  %C8 %C3%88
É  %C9 %C3%89
Ê  %CA %C3%8A
Ë  %CB %C3%8B
Ì  %CC %C3%8C
Í  %CD %C3%8D
Î  %CE %C3%8E
Ï  %CF %C3%8F
Ð  %D0 %C3%90
Ñ  %D1 %C3%91
Ò  %D2 %C3%92
Ó  %D3 %C3%93
Ô  %D4 %C3%94
Õ  %D5 %C3%95
Ö  %D6 %C3%96
×  %D7 %C3%97
Ø  %D8 %C3%98
Ù  %D9 %C3%99
Ú  %DA %C3%9A
Û  %DB %C3%9B
Ü  %DC %C3%9C
Ý  %DD %C3%9D
Þ  %DE %C3%9E
ß  %DF %C3%9F
à  %E0 %C3%A0
á  %E1 %C3%A1
â  %E2 %C3%A2
ã  %E3 %C3%A3
ä  %E4 %C3%A4
å  %E5 %C3%A5
æ  %E6 %C3%A6
ç  %E7 %C3%A7
è  %E8 %C3%A8
é  %E9 %C3%A9
ê  %EA %C3%AA
ë  %EB %C3%AB
ì  %EC %C3%AC
í  %ED %C3%AD
î  %EE %C3%AE
ï  %EF %C3%AF
ð  %F0 %C3%B0
ñ  %F1 %C3%B1
ò  %F2 %C3%B2
ó  %F3 %C3%B3
ô  %F4 %C3%B4
õ  %F5 %C3%B5
ö  %F6 %C3%B6
÷  %F7 %C3%B7
ø  %F8 %C3%B8
ù  %F9 %C3%B9
ú  %FA %C3%BA
û  %FB %C3%BB
ü  %FC %C3%BC
ý  %FD %C3%BD
þ  %FE %C3%BE
ÿ  %FF %C3%BF"
| rex max_match=0 field=data "(?<data>[^\n]+)"
| mvexpand data
| rex field=data "^(?<Character>.*)\t(?<dontcare>.*)\t(?<UTF8>.*)$"
| table Character UTF8
| where NOT match(Character, "[A-Za-z0-9]")
| search Character=*
| eval rex="s:\\".Character.":".UTF8.":g"
| stats values(rex) as rex
| eval rex=mvjoin(rex, " ")
If this comment/answer was helpful, please up vote it. Thank you.

jkat54
SplunkTrust
SplunkTrust

I like it!

0 Karma

woodcock
Esteemed Legend

You can also do this:

... | rex field=fieldToEncode mode=sed "s:%:%25:g s: :%20:g s:<:%3C:g s:>:%3E:g s:#:%23:g s:{:%7B:g s:}:%7D:g s:\|:%7C:g s:\\\:%5C:g s:\^:%5E:g s:~:%7E:g s:\[:%5B:g s:\]:%5D:g s:\`:%60:g s:;:%3B:g s:/:%2F:g s:\?:%3F:g s/:/%3A/g s:@:%40:g s:=:%3D:g s:&:%26:g s:\$:%24:g s:\!:%21:g s:\*:%2A:g"

woodcock
Esteemed Legend

It makes a nice macro.

0 Karma

jkat54
SplunkTrust
SplunkTrust

too late...

import urllib
import splunk.Intersplunk
import splunk.mining.dcutils as dcu
import traceback

# Setup logging/logger
logger = dcu.getLogger()

def encode(s):
  try:
    return urllib.quote_plus(s)

  except Exception, e:
    stack =  traceback.format_exc()
    splunk.Intersplunk.generateErrorResults(str(e))
    logger.error(str(e) + ". Traceback: " + str(stack))

def execute():
  try:
    # get the keywords and options passed to this command
    keywords, options = splunk.Intersplunk.getKeywordsAndOptions()

    # get the previous search results
    results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

    # if no keywords, send error results through
    if len(keywords) == 0:
      results = []
      results.append({"error":"syntax: urlencode <field_1> <field_2> <field_n> ..."})
      results.append({"error":"example: urlencode punct"})

    # else encode the fields provided
    if len(keywords) >= 1:
     for keyword in keywords:
       for result in results: 
        result[keyword] = encode(result[keyword])

    # return the results 
    results.sort()
    splunk.Intersplunk.outputResults(results)

  except Exception, e:
    stack =  traceback.format_exc()
    splunk.Intersplunk.generateErrorResults(str(e))
    logger.error(str(e) + ". Traceback: " + str(stack))

if __name__ == '__main__':
    execute()

To use this... save it as urlencode.py, put it in your app's bin directory, and then create a commands.conf in your app's local directory and add the following:

[urlencode]
filename = urlencode.py

Then restart, and you should be able to use it in your search like this:

... | urlencode  <field1> <field2> <field_n>  | table field1 field2 field_n

alt text

View solution in original post

jkat54
SplunkTrust
SplunkTrust

as FYI, i've added this search command to my toolkit app for those of you who want to take the easy route to get this command: https://splunkbase.splunk.com/app/3265/

woodcock
Esteemed Legend

Evidently this is no longer in that app and has been moved to the premium version here:
See https://splunkbase.splunk.com/app/3420

0 Karma

jkat54
SplunkTrust
SplunkTrust

The web tools app is not a premium app. It was for about a month or two but that was nearly a year ago.

0 Karma

floriancoulmier
Engager

Many thanks!

0 Karma

jkat54
SplunkTrust
SplunkTrust

You're very welcome. Feel free to come find me anytime you need custom search commands or apps. I take requests.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If you're interested, i can write a urlencode command...

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.