topic Re: How i can know splunk's method or function name inside splunk module if i need to write python script? in Splunk Dev

How i can know splunk's method or function name inside splunk module if i need to write python script?

claudius — Sat, 21 May 2011 07:44:00 GMT

python script like this for convers form decimal to ip

ipint = int(<input from splunk search>)
ip=""
for i in range(4):
ip1 = ""
for j in range(8):
# print ipint % 2
ip1=str(ipint % 2)+ip1
ipint = ipint >> 1
# print ip1
print ip1
ip = str(int(ip1,2)) + "." + ip
print ip.strip(".")

if i want to create command for use with splunk search!! What splunk module or method i need to use for python script.

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

hazekamp — Mon, 23 May 2011 21:19:30 GMT

claudis,

You could implement this using an external lookup or search command/script. I have provided links below which detail both methods along with sample search syntax.

Search w/ command: Search Scripts

<your search> | long2ip in=src_long out=src_ip

Search w/ lookup: External Lookups

<your search> | lookup long2ip src_long OUTPUT src_ip

This can also be implemented as a macro:

## macros.conf
[long2ip(2)]
args = long, ip
definition = eval $long$temp=$long$ | eval $ip$0=round($long$temp/16777216) | eval $long$temp=$long$temp-($ip$0*16777216) | eval $ip$1=round($long$temp/65536) | eval $long$temp=$long$temp-($ip$1*65536) | eval $ip$2=round($long$temp/256) | eval $ip$3=$long$temp-($ip$2*256) | strcat $ip$0 "." $ip$1 "." $ip$2 "." $ip$3 $ip$ | fields - $ip$0,$ip$1,$ip$2,$ip$3,$long$temp
iseval = 0

## Search
<your search> | `long2ip(src_long, src_ip)`

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

bwooden — Mon, 23 May 2011 21:33:45 GMT

Splunk allows the creation of custom search commands.

Alternatively, you can do this using the existing Splunk search language to solve this problem.

The below solution is intentionally verbose to show the different steps.

First, break the binary into 4 equal parts (each having a length of 8).
Next, convert each octet to its decimal form.
Finally, combine the results.

This example would take an ip_binary=11000000101010000000000000000001 and create an ip_decimal=192.168.0.1

...
| eval ip_binary=substr(ip_binary,1,8) + "." + substr(ip_binary,9,8) + "." + substr(ip_binary,17,8) + "." + substr(ip_binary,25,8) 
| makemv delim="." ip_binary 
| eval ip_decimal=tostring(tonumber(mvindex(ip_binary,0),2)) + "." + tostring(tonumber(mvindex(ip_binary,1),2)) + "." + tostring(tonumber(mvindex(ip_binary,2),2)) + "." + tostring(tonumber(mvindex(ip_binary,3),2))

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

Marinus — Tue, 24 May 2011 11:46:49 GMT

Here's an example script I write to do a reverse DNS lookup for an IP, it adds a new field dns_name. You can refactor it with your code.

search > * | long2ip field=ip

import splunk.Intersplunk
import sys
import socket 

def resolve(ip):
    name = 'unresolved'
    try:
        name = socket.gethostbyaddr(ip)[0] 
    except:
        pass
    return name

try:   
    # get the field 
    field = None

    if len(sys.argv) > 1:
            for a in sys.argv:
            if a.startswith("field="):
                        where = a.find('=')
                        field = a[where+1:len(a)].strip()

    if not field:
        results = splunk.Intersplunk.generateErrorResults("no field specified")
    else:
        results,unused1,unused2 = splunk.Intersplunk.getOrganizedResults()

        for result in results:
            result['dns_name'] = resolve(result[field])

        splunk.Intersplunk.outputResults(results)

except Exception, e:
    results = splunk.Intersplunk.generateErrorResults(str(e))

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

claudius — Thu, 26 May 2011 15:12:08 GMT

Great....too many ideas, thanks and thanks everyone for good answer.

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

data4734 — Tue, 04 Dec 2012 12:27:12 GMT

hazekamp shows using an external script named long2ip. Hazekamp, would you mind sharing the contents of long2ip.py so I can learn from it?

In trying and failing to reproduce long2ip.py by example, this is what I have done:

http://python-iptools.readthedocs.org/en/latest/#iptools.long2ip

I installed iptools and created a py file as simple as possible trying to follow the most simple examples that came with splunk.

long2ip.py returns the expected result at the command line if I comment out the splunk specific lines:

python long2ip.py
127.0.0.1

but when running from within splunk

index="passivecidr" decip="*"|fields + decip|long2ip

I get error code 1, instead of what I would expect, a 127.0.0.1 for every search result.

long2ip.py is located in $SPLUNK_HOME/etc/apps/search/bin dir and this is v5.0.1

In $SPLUNK_HOME/etc/apps/search/local/commands.conf

[long2ip]
filename = long2ip.py
streaming = true
retainsevents = true
overrides_timeorder = false


# Copyright (C) 2005-2012 Splunk Inc. All Rights Reserved.  Version 4.0
import sys,splunk.Intersplunk
import iptools
try:
    results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

    newresults = []
    newresults = iptools.long2ip(2130706433)

except:
    import traceback
    stack =  traceback.format_exc()
    results = splunk.Intersplunk.generateErrorResults("Error : Traceback: " + str(stack))

splunk.Intersplunk.outputResults(newresults)

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

hazekamp — Tue, 04 Dec 2012 16:23:05 GMT

Here is the script we use (this can be configured to run automatically via props.conf as well):

###### ip2long.py ######
'''
Copyright (C) 2005-2011 Splunk Inc. All Rights Reserved.
'''
import csv
import re
import sys

from ipMath import LongToIP, IPToLong

# set the maximum allowable CSV field size 
# 
# The default of the csv module is 128KB; upping to 10MB. See SPL-12117 for 
# the background on issues surrounding field sizes. 
# (this method is new in python 2.5) 
csv.field_size_limit(10485760)


if __name__ == '__main__':

    ipRex = re.compile('^(([0-1]\d{0,2}|2[0-4]\d|25[0-5]|\d{0,2})\.){3}([0-1]\d{0,2}|2[0-4]\d|25[0-5]|\d{0,2})$')

    ## Retrieve input
    inputResults = csv.reader(sys.stdin, lineterminator='\n')

    header = inputResults.next()

    ## Initialize output handlers
    csv.writer(sys.stdout, lineterminator='\n').writerow(header)
    output = csv.DictWriter(sys.stdout, header, lineterminator='\n', restval='')

    longKey = header.index('long')
    ipKey = header.index('ip')

    for inputResult in inputResults:
        ## Initialize and populate output lists/dictionary
        outputResult = {}

        if len(inputResult[longKey]) > 0:
            outputResult['long'] = inputResult[longKey]

            try:
                longVal = long(inputResult[longKey])

                if longVal >= 0 and longVal <= 4294967295:
                    outputResult['ip'] = LongToIP(longVal)   

            except:
                pass

        ## The reverse lookup
        else:
            outputResult['ip'] = inputResult[ipKey]

            if ipRex.match(inputResult[ipKey]):
                outputResult['long'] = IPToLong(inputResult[ipKey])

        output.writerow(outputResult)


###### ipmath.py ######
'''
Copyright (C) 2005-2011 Splunk Inc. All Rights Reserved.
'''

import heapq
import re

def LongToIP(value):
    '''Convert long to dotted ip address.'''
    if isinstance(value, int):
        value = long(value)
    if isinstance(value, long):
        return '%d.%d.%d.%d' % ((value >> 24) % 256, (value >> 16) % 256, (value >> 8) % 256, value % 256)
    return None

### Conversion functions.
### Note that conversions should perform validation and return None
### in the event of failure.
def IPToLong(value):
    '''Convert dotted ip address to long.'''
    if is_valid_ip(value):
        ip = map(long, value.split('.'))
        return 16777216 * ip[0] + 65536 * ip[1] + 256 * ip[2] + ip[3]
    return None

###### transforms.conf ######
[ip2long]
external_cmd  = ip2long.py
external_type = python
fields_list   = ip,long

This can be used to go from both ip->long and long->ip. For example:

[sourcetype]
LOOKUP-convert_ip_to_long = ip2long long as src_long OUTPUT ip as src_ip

[sourcetype]
LOOKUP-convert_long_to_ip = ip2long ip as src_ip OUTPUT long as src_long

This definition can also be used in "| lookup" in the search language as well

David

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

bwmetz — Tue, 25 Jul 2017 18:59:32 GMT

The macro answer shown first is good, but is prone to errors. Change the definition to first ensure your temp value is a number and then uses floor() instead of round() to calculate the dotted decimal values. Use of round() in the original solution will produce negative octet values in some cases. Use of floor() imitates integer division, e.g. int(); alas a function several other community answers confirm is not available in Splunk, much to our lament:

definition = eval $long$temp=tonumber($long$) | eval $ip$0=floor($long$temp/16777216) | eval $long$temp=$long$temp-($ip$0*16777216) | eval $ip$1=floor($long$temp/65536) | eval $long$temp=$long$temp-($ip$1*65536) | eval $ip$2=floor($long$temp/256) | eval $ip$3=$long$temp-($ip$2*256) | strcat $ip$0 "." $ip$1 "." $ip$2 "." $ip$3 $ip$ | fields - $ip$0,$ip$1,$ip$2,$ip$3,$long$temp

Re: How i can know splunk's method or function name inside splunk module if i need to write python script?

bwmetz — Tue, 25 Jul 2017 21:26:13 GMT

Amendment to my suggestion to protect against invalid values in large data sets causing "..." to come out of the macro. Adding a validation rule to the macro might be smarter, but more complex when you don't understand them.

eval $long$temp=tonumber($long$) | eval $ip$0=floor($long$temp/16777216) | eval $long$temp=$long$temp-($ip$0*16777216) | eval $ip$1=floor($long$temp/65536) | eval $long$temp=$long$temp-($ip$1*65536) | eval $ip$2=floor($long$temp/256) | eval $ip$3=$long$temp-($ip$2*256) | strcat $ip$0 "." $ip$1 "." $ip$2 "." $ip$3 $ip$ | eval $ip$=if($ip$="...","",$ip$) | fields - $ip$0,$ip$1,$ip$2,$ip$3,$long$temp