Problem with Geospatial lookup and geom command

[badrinath_itrs]

Hi All,

Posting this question, as I am new to Geospatial lookup and trying to configure it as per Michael Porath's blog (http://blogs.splunk.com/2015/10/01/use-custom-polygons-in-your-choropleth-maps/) .

I have received a shape file, which I have converted to KML format using the steps mentioned in it, but when I run the geom command it gives me a bunch of error message as below and splunkd crashes. Here is the few output from crash log.

04-11-2016 11:19:36.534 ERROR spatial:PointInPolygonIndex - wrote 18933 grid keys
    04-11-2016 11:19:44.880 INFO  spatial:Util - PIP index built successfully
    04-11-2016 11:19:44.889 INFO  SearchOperator:Geom - did not find clipped geometry in cache for featureCollection=bambi_aus_regions_old
    04-11-2016 11:19:44.889 INFO  SearchOperator:Geom - generalization=1.000000
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - Num Keys in ray.key: 16923
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - Num Keys in seg.key: 4331904
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - Num Keys in grid.key: 18933
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - memory mapping existing file: ray.key

04-11-2016 11:19:44.890 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/app/optier/splunk/etc/apps/BambiAdmin/lookups/bambi_aus_regions_old/ray.key in  0.001000 sec
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - memory mapping existing file: ray.val
    04-11-2016 11:19:44.890 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/app/optier/splunk/etc/apps/BambiAdmin/lookups/bambi_aus_regions_old/ray.val in  0.001000 sec
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - memory mapping existing file: seg.key
    04-11-2016 11:19:44.890 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/app/optier/splunk/etc/apps/BambiAdmin/lookups/bambi_aus_regions_old/seg.key in  0.001000 sec
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - memory mapping existing file: seg.val
    04-11-2016 11:19:44.890 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/app/optier/splunk/etc/apps/BambiAdmin/lookups/bambi_aus_regions_old/seg.val in  0.001000 sec
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - memory mapping existing file: grid.key
    04-11-2016 11:19:44.890 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/app/optier/splunk/etc/apps/BambiAdmin/lookups/bambi_aus_regions_old/grid.key in  0.001000 sec
    04-11-2016 11:19:44.890 INFO  spatial:PointInPolygonIndex - memory mapping existing file: grid.val
    04-11-2016 11:19:44.890 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/app/optier/splunk/etc/apps/BambiAdmin/lookups/bambi_aus_regions_old/grid.val in  0.001000 sec
    04-11-2016 11:19:44.892 WARN  spatial:PointInPolygonIndex - requested tile was at y=-92 but returned tile as at -93

In one of my local environment the geom command does produce an output in stats option but in Choropleth map section I get an error as "problem memmapping".

I tried to break the kml file into small piece and then it works.

Can you advice what needs to be performed to fix this issue .

My KML file is around 76MB and the KMZ format it comes to 20 MB. Is there a limit on the size of KML file which splunkd can process.

[woodcock]

We found that the problem was not really a bug, but an incompatibility of our KMZ file. It took WAY to long for us to figure this out and this is one area where the normally outstandingly excellent Splunk documentation is very lacking. We never would have figured it out without this blog post, so start here: https://www.splunk.com/en_us/blog/tips-and-tricks/use-custom-polygons-in-your-choropleth-maps.html
Here are the main points of confusion.

1: A `KMZ` file is simply a bundle of files, at least one of which is a `KML` file, which may, or may not be named that way (often, it is named `doc.html`).  Along with the `KML` file may be other files such as image bitmaps which can be used as a canvas on which to draw the shapes that are in your `KML` file.  We never did get this part of it working and we resorted to using the standard Splunk mapping with carefully selected tiles (more on that later).  To keep it simple, I recommend that you avoid creating a `KMZ` unless you need other files beyond the `KML` file.  Even so, start with just the `KML` file and get the basics working first.  You will probably find that you are happy enough without a custom canvas.
2: It might be best to simply start completely over, rather than try to make sense of the `KMZ` file that you were given, especially for educational purposes.  Consider using the `Shapester - Geo Shape Editor` app on `Splunkbase` (https://splunkbase.splunk.com/app/2893/) to build some shapes into a `KML` file that definitely should be Splunk-geo-compatible. 
3: Once uploaded as a `Lookup File`, you still will not be able to use your `KML` without also creating an accompanying `Lookup Definition`.  Go to `Settings` -> `Lookups` -> `Lookup Definitions` -> `New` and select `Geospatial` under `Type`, then select your `KML` file.  If it is named `MyShapes.KML` then I suggest that you name your `Lookup Definition` the same, but drop the `.kml` and use `MyShapes`.  From here on out, you will *only* use the `Lookup Definition`, never the `Lookup File`.

Now that you have a compatible KML file, you need to make it work. Start with | inputlookup MyShapes to see if you can access the featureId and geom fields inside of you KML. If it is built in such a way that Splunk can use it, you should see many lines returned on the Statistics tab. Then click on the Visualization tab and you should be able to see the results on a map but you must do ALL of the following:

1: Select the `Choropleth Map` visualization.
2: Keep `zooming` and `centering` your view until it is positioned over the location of the shapes in your file.
3: If your shapes are small, you will find that the default maps do not allow enough `zoom` to see them; to fix this....
4: Click on the `Format` tool (the `paint brush` icon) and go to the `Tiles` section.
5: Look at the comment that says `The URL to use for requesting tiles, ex: http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png` and grab the `http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png` text and paste it into the `URL` field.  Instantly you should have infinite `zoom` detail.  Really, this is probably the `secret magic` that you lacked.  This is not clearly documented anywhere and we discovered it by accident playing around.

It really helps to take a look at the Choropleth Map Color Modes example with San Francisco Neighborhoods in the Map Elements area of the Splunk Dashboard Examples app on Splunkbase (https://splunkbase.splunk.com/app/1603/). It shows you how to do everything EXCEPT for the magical #5 step. Although the recommended tile set is really good, there are many, MANY, options out there so be sure to try a variety. Here are some alternative tile sets that render instantly in Splunk:

https://wiki.openstreetmap.org/wiki/Tile_servers
OpenStreetMaps: http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png
Wikipedia: https://maps.wikimedia.org/osm-intl/{z}/{x}/{y}.png
OpenCycleMap: http://tile.thunderforest.com/cycle/{z}/{x}/{y}.png
Humanitarian Style: http://a.tile.openstreetmap.fr/hot/{z}/{x}/{y}.png
Hike and Bike: https://tiles.wmflabs.org/hikebike/{z}/{x}/{y}.png

[badrinath_itrs · Accepted solution]

Hi All,

It found out to be a problem with SPLUNK and as per SPLUNK Support this will be fixed as part of major release in September.

Will Wait till it gets resolved !!