Splunk Search

How to update geoip database for iplocation command?

echojacques
Builder

Hello,

I use Splunk's iplocation (not Maxmind or other) command extensively in our monitoring dashboards. Since this is Splunk's "built-in" geoip command, does the underlying geoip database get automatically updated or do I have to manually update it? If manual, how do I update it?

Thanks!

++ Edit: I also checked the iplocation documentation but it doesn't describe how to update the geoip database:
http://docs.splunk.com/Documentation/Splunk/6.0.1/SearchReference/Iplocation

+++ Edit 4/4/14: I re-installed Maxmind per a recent suggestion that Splunk's iplocation command uses Maxmind's geoip database. After re-installing Maxmind and restarting Splunk, I ran two identical searches with geoip lookups: one with Splunk's iplocation and the other with Maxmind's geoip command. The results were different, proving that Splunk's iplocation command does not use Maxmind's geoip database. This also explains why the iplocation command will continue to work even if you uninstall the Maxmind app (and delete all of the directories as well). Also, I found that the iplocation command produces much better results (more complete) than Maxmind's "lookup geoip.." command (doesn't have geoip info for many IPs in our logs, which is why I prefer to use Splunk's iplocation command). So this question is still open...

++++ Edit 4/14/14: I searched the /splunk for *geo* (not sure why I didn't do this earlier!) and found this file:

/splunk/share/ipv4.geodb

Does anyone know if this is Splunk's geoip database for the iplocation command? And if so, same question stands... how does it get updated?

Thanks!

Tags (2)
1 Solution

jbrodsky_splunk
Splunk Employee
Splunk Employee

echojacques, that IS the file we use. There is also a file for the iso3166 mappings in the same directory - the version of the .db file there does NOT contain them (it is based on the "lite" version of Maxmind) so these are separate. Unfortunately you can't just "update" that db file, but it should be updated with every maintenance release of Splunk - approximately every two months. Also in a future version of Splunk you can swap these files out for "paid" versions of Maxmind data - contact your local Splunk representation for more official detail on this.

View solution in original post

VatsalJagani
Motivator

This App seems to be really useful - https://splunkbase.splunk.com/app/5482/.

The App auto-updates the MaxMind database without going into the backend. It also allows you to run a search command on Splunk search to manually download and update latest database. 

0 Karma

awurster
Contributor

I recently discovered (maybe even by reading this question) that Splunk does not automatically update their GeoIP data. It lead me down an interesting search into IP location databases and how accurate they are and how much they cost, etc... I also discovered that MaxMind recently released a new "2.0" version of their free Location DB called GeoLite2.

Here's how I solve this problem in our deployment using cron and shell scripts. Note that the cron has a weird trick to catch the GeoLite DB updates on the first Tuesday of every month.

In Splunk user's crontab:

~$ crontab -l
# MaxMind DB Update for Splunk #
################################
SPLUNK_HOME=/opt/splunk    
OUTFILE=/tmp/cron.stdout.log

0 0 23 * * 3 [ $(date +\%d) -le 07 ] && $SPLUNK_HOME/bin/scripts/get_maxmind_db.sh >> $OUTFILE 2>&1

The shell script:

~$ cat $SPLUNK_HOME/bin/scripts/get_maxmind_db.sh
#!/bin/bash

# Author: Andrew Wurster
# Date: 13 Jan 2015

cd /opt/splunk/share

wget -O GeoLite2-City-Latest.mmdb.gz http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz || { echo 'Could not download MaxMind GeoIP DB, exiting.' ; exit 1; }
gunzip -f GeoLite2-City-Latest.mmdb.gz
chmod 644 GeoLite2-City-Latest.mmdb

In limits.conf

~$ cat etc/system/local/limits.conf 
[iplocation]
db_path = /opt/splunk/share/GeoLite2-City-Latest.mmdb
#db_path = /opt/splunk/share/GeoIP2-City.mmdb

This is valid for MaxMinds new 2.0 format, and so far has not shown me any changes in lookup performance, etc.

FWITW I believe Splunk should make it more obvious where / how they are distributing this file (and the update frequency). A way to update it from the GUI or somewhere else convenient would be nice.

arnauec
New Member

I have followed the procedure above but in my case I'm working with a distributed system.

How could I update the GeoLite2-City-latest.mmdb in each server automatically? For some reason my indexers are not fetching the updated database from the deployment server.

0 Karma

sgundeti
Path Finder
0 Karma

markhill1
Path Finder

Just a note on this, you will now need to use HTTPS in place of HTTP within the script.

0 Karma

koshyk
Super Champion

voted up @hortonew and @awurster
Thanks for saving a lot of work 🙂

0 Karma

hortonew
Builder

Andrew - thanks for this script. I just automated the deployment a little differently and wrote up a blog post about it.

http://blog.hortonew.com/splunk-automatically-update-geoip-database-across-environment

Rob2520
Communicator

@hortonew Do the update need to be done on just search head or both search head and indexer? I updated Maxmind Geolite2 on just my search head and restarted splunk. I'm asking this question even after seeing your blog because for one of the IP i see inaccurate results. Any suggestions is appreciated. Thanks!

0 Karma

markhill1
Path Finder

As far as my interactions with support have gone, it needs to be updated on the SH's and the IDX machines.

0 Karma

awurster
Contributor

@hortonew - legend.

0 Karma

Ovi
Path Finder

Andrew, this is awesome, works like a charm. Thank you!
One question that bothers me: when using Data Models - do the Data Models need to be updated after the GeoIP database update? I ask because I seem to get very strange results when searching against the data model vs. normal search (in normal search the results are accurate). Thanks

0 Karma

awurster
Contributor

hey sorry for the delay @Ovi - i didn't get a notice of your ping!

i really have no idea about data models and updating.. maybe a splunk guru can chime in here??

i was recently starting to explain how and why we did this for a colleague.. so maybe it's worth us turning this crontab into an app which would be more portable? not sure if that's even possible...? as for data models.. if there are any config tweaks etc, we could put it in there?

sylim_splunk
Splunk Employee
Splunk Employee

You can use your own premium MAXMIND data file using [iplocation] stanza in limits.conf

[iplocation]

db_path = path_to_the_MMDB

  • Location of GeoIP database in MMDB format
  • If not set, defaults to database included with splunk

sylim_splunk
Splunk Employee
Splunk Employee

Simpler way

  1. You can go to this site, http://dev.maxmind.com/geoip/geoip2/geolite2/
  2. Download GeoLite2-City.mmdb
  3. Replace it with $SPLUNK_HOME/share/GeoLite2-City.mmdb
  4. Restart splunk and prove it works as you expect.

jbrodsky_splunk
Splunk Employee
Splunk Employee

echojacques, that IS the file we use. There is also a file for the iso3166 mappings in the same directory - the version of the .db file there does NOT contain them (it is based on the "lite" version of Maxmind) so these are separate. Unfortunately you can't just "update" that db file, but it should be updated with every maintenance release of Splunk - approximately every two months. Also in a future version of Splunk you can swap these files out for "paid" versions of Maxmind data - contact your local Splunk representation for more official detail on this.

View solution in original post

echojacques
Builder

Thanks for the answer, I suspected that it was updated with every release, and you confirmed this.

Really appreciate the clarification and happy to know that I haven't been slacking by not updating this file... 🙂

awurster
Contributor

you're not really "slacking" per se - versus other splunk users - but since the mappings change fairly regularly in some parts of the world.. you'd potentially be using months-old, if not years-old data.

0 Karma

somesoni2
Revered Legend

echojacques
Builder

Thanks but I don't think so. The link answers how to update the maxmind database, if you are using maxmind for geoip. I've uninstalled maxmind since I decided to only use Splunk's built-in iplocation command... so I don't have the maxmind directory, etc. that this link refers to.

0 Karma

cam343
Path Finder

I also would be interested in the answer to this question...

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!