GoogleMaps app does work with latest MAXMIND databases?

[asleeis]

Hi,

The link in the README is no longer valid. When I went to get the latest full detail database, I found that GoogleMaps app doesn't work with it. For one, the format of the CSV is quoted comma separated. The database that is included with this download seems to be a semi-colon separated format. So, I went and did a mass search and replace for the newly downloaded CSV to match that. Then what I found is that the fields of data are different than they were with the included database.

Also, I'm not sure I understand how to update the database entirely. I updated the ip_group_city.csv with the download I got from here: http://ipinfodb.com/ip_database.php

I used the DB11 (full detail) CSV. I also noticed that there's country geo lookups under the "lookups" folder of the maps app. Should that also be updated somehow?

The results of my downloading the latest full detail CSV, changing commas to semi colons, was that my longitude has a ZIP code, ZIP code has something else (longitude I think), etc.

Anyway... is there an update that actually works with updated Geo data?

Thanks, -Alex

[ziegfried]

There will be a new version soon which will leverage the Maxmind GeoCity Light database instead of the one from ipinfodb.com.

[joberget]

Great news, will it support GeoIP City (the payable) version as well?

[oscargarcia]

Hi,

Can anybody post the patched geoip.py? I can't get to patch it properly..

Thanks

[joberget]

Hi, no worries it seems to be working now!

I solved the "buggy" interface by clearing web cache, now everything seems to be working great again.

The only problem I have now is that it sometimes mismatches the IP and what it plots on the map (Ip2loc says Russia but it plots Denmark for example), I use the latest DB11 map. I guess this is a "known issue" or? I would say that it works well 90% of the time. 🙂

[asleeis]

I did not place the code there. I only modified things minimally from what it was. That if statement was intended to identify the correct record in the file and return it. In all honesty, I have not looked at the code from a design perspective. I just did as little as I could to try and make it work. Keep in mind, I'm not the original developer, and only posted the "patch" due to request. I am also not a Python developer, so I was just winging it, having developed in many other languages before.

Even with the original version, I also saw problems when I tried it on 4.2. I've got no idea what the issue is. I've never coded a full app for Splunk before, so I have no idea where to even begin looking in the code. What's odd is that the bugs seem inconsistent. Sometimes I get the text numbers but not the color icons on the map. Definitely a little buggy. But sometimes, it works fine. heh.

[joberget]

Hi!

I found an error in this patch.

Shouldn't the following be in the def lookup(ip): and not def get_records(idx, n)?

• if ip > r[i][0] and ip < r[i][1]: return r[i][2]
• return r[-1][2]

I use Splunk 4.2, when I swapped the old csv file that comes with the app to the new DB11 from IP2Location my Google Maps app GUI looks really weird and buggy. Any idea what might be causing this? I only changed geoip.py and the csv file.

[asleeis · Accepted solution]

And in case you prefer a "patch" file...

--- orig/geoip.py   2011-03-16 15:11:21.000000000 -0700
+++ geoip.py    2011-03-16 16:13:32.000000000 -0700
@@ -4,7 +4,7 @@
 from string import atoi
 import socket, struct, csv

-ATTRIBUTES = ['countrycode', 'countryname', 'regioncode', 'regionname', 'city', 'zipcode', 'latitude', 'longitude']
+ATTRIBUTES = ['countrycode', 'countryname', 'region', 'city', 'latitude', 'longitude', 'zipcode', 'timezone']
 MAP_BLOCK_SIZE=64

 DB_FILE = os.path.join(os.environ['SPLUNK_HOME'], 'etc', 'apps', 'maps', 'lib', 'ip_group_city.csv')
@@ -51,7 +51,7 @@
        if not line: break
        i=i+1
        if i % MAP_BLOCK_SIZE == 0:
-           val = atoi(line.split(";")[0].strip('"'), 10)
+           val = atoi(line.split(",")[0].strip('"'), 10)
            k.append(val)
            v.append(pos)
    f = open(IDX_FILE, "wb")
@@ -75,8 +75,8 @@

 def read_record(l):
    if l:
-       row = [ x.strip('"') for x in l.strip().split(";") ]
-       return [atoi(row[0]), row[1:]]
+       row = [ x.strip('"') for x in l.strip().split(",") ]
+       return [atoi(row[0]), atoi(row[1]), row[2:]]

 def get_records(idx, n):
    v = idx_v[idx]
@@ -98,8 +98,8 @@
        r = get_records(i-1, MAP_BLOCK_SIZE+1)
        if r:
            for i in range(1, len(r)):
-               if r[i][0] > ip: return r[i-1][1]
-           return r[-1][1]
+               if ip > r[i][0] and ip < r[i][1]: return r[i][2]
+           return r[-1][2]

 def extract_ip(r):
    if '_raw' in r:
@@ -128,6 +128,6 @@
    if resvoled and resvoled[0] != 'RD':
        for i, f in enumerate(ATTRIBUTES):
            r["%s_%s" % (ip_field, f)] = resvoled[i]
-       r["_lat"] = resvoled[6]
-       r["_lng"] = resvoled[7]
+       r["_lat"] = resvoled[4]
+       r["_lng"] = resvoled[5]
    writer.writerow(r)

[asleeis]

It's not a huge amount of change, so I'll just post some diff's of the original files and my mods:

diff etc/apps/maps/bin/orig/geoip.py etc/apps/maps/bin/geoip.py 7c7
< ATTRIBUTES = ['countrycode', 'countryname', 'regioncode', 'regionname', 'city', 'zipcode', 'latitude', 'longitude']
---
> ATTRIBUTES = ['countrycode', 'countryname', 'region', 'city', 'latitude', 'longitude', 'zipcode', 'timezone']
54c54
<  val = atoi(line.split(";")[0].strip('"'), 10)
---
>  val = atoi(line.split(",")[0].strip('"'), 10)
78,79c78,79
<  row = [ x.strip('"') for x in l.strip().split(";") ]
<  return [atoi(row[0]), row[1:]]
---
>  row = [ x.strip('"') for x in l.strip().split(",") ]
>  return [atoi(row[0]), atoi(row[1]), row[2:]]
101,102c101,102
<  if r[i][0] > ip: return r[i-1][1]
<  return r[-1][1]
---
>  if ip > r[i][0] and ip < r[i][1]: return r[i][2]
>  return r[-1][2]
131,132c131,132
<  r["_lat"] = resvoled[6]
<  r["_lng"] = resvoled[7]
---
>  r["_lat"] = resvoled[4]
>  r["_lng"] = resvoled[5]

That should do it. 🙂 In addition to supporting CSV in the format used (i.e. commas), and supporting the current field order, I also modified it to use the IP start and stop values. The old format only provided start values, and assumed the next start being after the IP meant the previous one applied. Not a very deterministic approach. Now they are more specific, and this update takes advantage of that.

With this update, you should be able to just copy an updated file.

[gsellek]

I'm having the same issue here. Any chance you could post your change here 'unofficially'?

Thanks!

[asleeis]

Okay. This isn't really an answer, and I'm hoping that ziegfried (author) will read this. I couldn't quite find out how to contact an author aside from this. It would be nice if Splunk base supported a simple contact method.

After creating this question/issue, I dig more deeply into the code of geoip.py. I am not a Python developer, but having coded in many languages over the years, it was relatively easy to follow (give or take a couple syntax lookups). I created a modified version of geoip.py that will work with a proper CSV format (the version available uses semi-colon separated), and to work with the current structure of the full detail database (DB11, I think they call it) that is available free.

I'd love to contribute my update back (although wouldn't mind a quick review from the original author, since I may not have accounted for everything). Aside from just different fields, there's a start and end for the IP, so I adjusted the logic. I'm just not sure I did it correctly. heh. I still need to test things a little.

How can I go about contributing my update back?