How to update geoip database for iplocation command?

[echojacques]

Hello,

I use Splunk's iplocation (not Maxmind or other) command extensively in our monitoring dashboards. Since this is Splunk's "built-in" geoip command, does the underlying geoip database get automatically updated or do I have to manually update it? If manual, how do I update it?

Thanks!

++ Edit: I also checked the iplocation documentation but it doesn't describe how to update the geoip database:
http://docs.splunk.com/Documentation/Splunk/6.0.1/SearchReference/Iplocation

+++ Edit 4/4/14: I re-installed Maxmind per a recent suggestion that Splunk's iplocation command uses Maxmind's geoip database. After re-installing Maxmind and restarting Splunk, I ran two identical searches with geoip lookups: one with Splunk's iplocation and the other with Maxmind's geoip command. The results were different, proving that Splunk's iplocation command does not use Maxmind's geoip database. This also explains why the iplocation command will continue to work even if you uninstall the Maxmind app (and delete all of the directories as well). Also, I found that the iplocation command produces much better results (more complete) than Maxmind's "lookup geoip.." command (doesn't have geoip info for many IPs in our logs, which is why I prefer to use Splunk's iplocation command). So this question is still open...

++++ Edit 4/14/14: I searched the /splunk for *geo* (not sure why I didn't do this earlier!) and found this file:

/splunk/share/ipv4.geodb

Does anyone know if this is Splunk's geoip database for the iplocation command? And if so, same question stands... how does it get updated?

Thanks!

[VatsalJagani]

This App seems to be really useful - https://splunkbase.splunk.com/app/5482/.

The App auto-updates the MaxMind database without going into the backend. It also allows you to run a search command on Splunk search to manually download and update latest database. 

[awurster]

I recently discovered (maybe even by reading this question) that Splunk does not automatically update their GeoIP data. It lead me down an interesting search into IP location databases and how accurate they are and how much they cost, etc... I also discovered that MaxMind recently released a new "2.0" version of their free Location DB called GeoLite2.

Here's how I solve this problem in our deployment using cron and shell scripts. Note that the cron has a weird trick to catch the GeoLite DB updates on the first Tuesday of every month.

In Splunk user's crontab:

~$ crontab -l
# MaxMind DB Update for Splunk #
################################
SPLUNK_HOME=/opt/splunk    
OUTFILE=/tmp/cron.stdout.log

0 0 23 * * 3 [ $(date +\%d) -le 07 ] && $SPLUNK_HOME/bin/scripts/get_maxmind_db.sh >> $OUTFILE 2>&1

The shell script:

~$ cat $SPLUNK_HOME/bin/scripts/get_maxmind_db.sh
#!/bin/bash

# Author: Andrew Wurster
# Date: 13 Jan 2015

cd /opt/splunk/share

wget -O GeoLite2-City-Latest.mmdb.gz http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz || { echo 'Could not download MaxMind GeoIP DB, exiting.' ; exit 1; }
gunzip -f GeoLite2-City-Latest.mmdb.gz
chmod 644 GeoLite2-City-Latest.mmdb

In limits.conf

~$ cat etc/system/local/limits.conf 
[iplocation]
db_path = /opt/splunk/share/GeoLite2-City-Latest.mmdb
#db_path = /opt/splunk/share/GeoIP2-City.mmdb

This is valid for MaxMinds new 2.0 format, and so far has not shown me any changes in lookup performance, etc.

FWITW I believe Splunk should make it more obvious where / how they are distributing this file (and the update frequency). A way to update it from the GUI or somewhere else convenient would be nice.

[arnauec]

I have followed the procedure above but in my case I'm working with a distributed system.

How could I update the GeoLite2-City-latest.mmdb in each server automatically? For some reason my indexers are not fetching the updated database from the deployment server.

[sgundeti]

here is the doc, how to update in distributed deployments.
https://docs.splunk.com/Documentation/Splunk/7.2.5/SearchReference/Iplocation#The_MMDB_file_and_dist...

[markhill1]

Just a note on this, you will now need to use HTTPS in place of HTTP within the script.

[koshyk]

voted up @hortonew and @awurster
Thanks for saving a lot of work 🙂

[hortonew]

Andrew - thanks for this script. I just automated the deployment a little differently and wrote up a blog post about it.

http://blog.hortonew.com/splunk-automatically-update-geoip-database-across-environment

[Rob2520]

@hortonew Do the update need to be done on just search head or both search head and indexer? I updated Maxmind Geolite2 on just my search head and restarted splunk. I'm asking this question even after seeing your blog because for one of the IP i see inaccurate results. Any suggestions is appreciated. Thanks!

[markhill1]

As far as my interactions with support have gone, it needs to be updated on the SH's and the IDX machines.

[awurster]

@hortonew - legend.

[Ovi]

Andrew, this is awesome, works like a charm. Thank you!
One question that bothers me: when using Data Models - do the Data Models need to be updated after the GeoIP database update? I ask because I seem to get very strange results when searching against the data model vs. normal search (in normal search the results are accurate). Thanks

[awurster]

hey sorry for the delay @Ovi - i didn't get a notice of your ping!

i really have no idea about data models and updating.. maybe a splunk guru can chime in here??

i was recently starting to explain how and why we did this for a colleague.. so maybe it's worth us turning this crontab into an app which would be more portable? not sure if that's even possible...? as for data models.. if there are any config tweaks etc, we could put it in there?

[sylim_splunk]

You can use your own premium MAXMIND data file using [iplocation] stanza in limits.conf

[iplocation]

db_path = path_to_the_MMDB

• Location of GeoIP database in MMDB format
• If not set, defaults to database included with splunk

[sylim_splunk]

Simpler way

1. You can go to this site, http://dev.maxmind.com/geoip/geoip2/geolite2/
   
2. Download GeoLite2-City.mmdb
3. Replace it with $SPLUNK_HOME/share/GeoLite2-City.mmdb
4. Restart splunk and prove it works as you expect.

[jbrodsky_splunk · Accepted solution]

echojacques, that IS the file we use. There is also a file for the iso3166 mappings in the same directory - the version of the .db file there does NOT contain them (it is based on the "lite" version of Maxmind) so these are separate. Unfortunately you can't just "update" that db file, but it should be updated with every maintenance release of Splunk - approximately every two months. Also in a future version of Splunk you can swap these files out for "paid" versions of Maxmind data - contact your local Splunk representation for more official detail on this.

[echojacques]

Thanks for the answer, I suspected that it was updated with every release, and you confirmed this.

Really appreciate the clarification and happy to know that I haven't been slacking by not updating this file... 🙂

[awurster]

you're not really "slacking" per se - versus other splunk users - but since the mappings change fairly regularly in some parts of the world.. you'd potentially be using months-old, if not years-old data.

[somesoni2]

Looking for this?

http://answers.splunk.com/answers/48844/how-to-update-the-geoip-database-for-the-geo-location-lookup...

[echojacques]

Thanks but I don't think so. The link answers how to update the maxmind database, if you are using maxmind for geoip. I've uninstalled maxmind since I decided to only use Splunk's built-in iplocation command... so I don't have the maxmind directory, etc. that this link refers to.

[cam343]

I also would be interested in the answer to this question...