Dashboards & Visualizations

Error When Creating Choropleth From Custom KMZ File

jihape
Path Finder

Hi,

I'm trying to use a custom kmz file in Splunk (6.5.3), but I keep getting a "problem memmapping" error.

I'm using a KMZ file from google maps - It can be found at https://www.google.com/maps/d/viewer?mid=1Dv1JRTGmzlm83tBv7tb8vQcOQXY&ll=-34.166789639385996%2C143.7... (click the three dots next to the magnifying glass)

When I run this search command | stats count | eval featureId="Sydney" | eval count=10000 | geom geo_lhds I get the memmapping error. Using | stats count | eval featureId="California" | eval count=10000 | geom geo_us_states works as expected.

Here is some of the output from the search.log
05-19-2017 15:19:17.001 INFO SearchOperator:Geom - loading pipIndex named geo_lhds
05-19-2017 15:19:17.002 INFO spatial:Util - could not resolve placemark_specifier, using default path
05-19-2017 15:19:17.002 INFO spatial:Util - fullPath to spatial polygons data file: /opt/splunk/etc/users/admin/search/lookups/geo_lhd_3.kmz
05-19-2017 15:19:17.002 INFO spatial:Util - using disk index: /opt/splunk/etc/users/admin/search/lookups/geo_lhds
05-19-2017 15:19:17.002 INFO ScopedSTLock - Waiting to acquire lock for path=/opt/splunk/etc/users/admin/search/lookups/geo_lhds
05-19-2017 15:19:17.002 INFO ScopedSTLock - Lock acquired for path=/opt/splunk/etc/users/admin/search/lookups/geo_lhds 0
05-19-2017 15:19:17.002 INFO SearchOperator:Geom - did not find clipped geometry in cache for featureCollection=geo_lhds
05-19-2017 15:19:17.002 INFO SearchOperator:Geom - generalization=1.000000
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - Num Keys in ray.key: 1
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - Num Keys in seg.key: 0
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - Num Keys in grid.key: 0
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: ray.key
05-19-2017 15:19:17.002 INFO spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/users/admin/search/lookups/geo_lhds/ray.key in 0.001000 sec
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: ray.val
05-19-2017 15:19:17.002 INFO spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/users/admin/search/lookups/geo_lhds/ray.val in 0.001000 sec
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: seg.key
05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file seg.key
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: seg.val
05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file seg.val
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: grid.key
05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file grid.key
05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: grid.val
05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file grid.val
05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - memmap failed! no valid keys and values
05-19-2017 15:19:17.002 INFO UserManager - Unwound user context: admin -> NULL
05-19-2017 15:19:17.002 ERROR DispatchThread - problem memmapping
05-19-2017 15:19:17.003 INFO UserManager - Setting user context: admin
05-19-2017 15:19:17.003 INFO UserManager - Done setting user context: NULL -> admin
05-19-2017 15:19:17.003 INFO UserManager - Unwound user context: admin -> NULL
05-19-2017 15:19:17.003 INFO DispatchManager - DispatchManager::dispatchHasFinished(id='1495171156.1629', username='admin')
05-19-2017 15:19:17.004 INFO SearchOperator:Geom - deleting PointInPolygonIndex cache for: geo_lhds
05-19-2017 15:19:17.004 INFO UserManager - Unwound user context: admin -> NULL
05-19-2017 15:19:17.005 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: problem memmapping

I suspect there is something wrong with the KMZ/KML file, but what I don't know. The only difference I can see is that the KML file in the KMZ file I downloaded from Google has spaces as indents and there no indents in the geo_us_states.kmz file. I've tried to remove those but it didn't help either.

Any help appreciated!

0 Karma
1 Solution

Gibbo87
Explorer

In the mean time I found a sort of workaround.
Apparently the problem is that the XPath expression used to find the XML element containing the featureID is not working. You can let Splunk use the default expression which is /kml/Document/Folder/Placemark/name, the documentation omits the /kml/Document/Folderpart.

I had to restructure my KML adding the Folder element which was not there. My KML now looks like this.

<kml ... >
<Document>
    <Folder>
        <name>ita_regions</name>
    <Placemark>
        <name>Abruzzo</name>
                .....

Apparently The folder name is mapped to the featureCollection field and the Placemark name is mapped to featureId and both have to be there. Here is how the lookup looks like in Splunk after the fix.

alt text

Hope this helps.

View solution in original post

0 Karma

Gibbo87
Explorer

In the mean time I found a sort of workaround.
Apparently the problem is that the XPath expression used to find the XML element containing the featureID is not working. You can let Splunk use the default expression which is /kml/Document/Folder/Placemark/name, the documentation omits the /kml/Document/Folderpart.

I had to restructure my KML adding the Folder element which was not there. My KML now looks like this.

<kml ... >
<Document>
    <Folder>
        <name>ita_regions</name>
    <Placemark>
        <name>Abruzzo</name>
                .....

Apparently The folder name is mapped to the featureCollection field and the Placemark name is mapped to featureId and both have to be there. Here is how the lookup looks like in Splunk after the fix.

alt text

Hope this helps.

0 Karma

jihape
Path Finder

I have tried to clean the KML and restructure as well and adding the full XPATH to the "featureId" field and none of it worked for me with the KMZ file I have.

I'll accept your answer as there are probably other people that need to restructure the KML for it to work.

0 Karma

Gibbo87
Explorer

I just incurred in the exact same problem using maps from this website. http://www.gadm.org/download

07-06-2017 13:53:11.944 INFO  SearchOperator:Geom - in getType _reporting is 0
07-06-2017 13:53:11.944 INFO  SearchOperator:Geom - in getType _reporting is 0
07-06-2017 13:53:11.944 INFO  SearchOperator:Geom - loading pipIndex named regioni2
07-06-2017 13:53:11.946 INFO  spatial:Util - get user-defined xpath /Placemark/name
07-06-2017 13:53:11.946 INFO  spatial:Util - fullPath to spatial polygons data file: /opt/splunk/etc/apps/search/lookups/ITA_regioni.kmz
07-06-2017 13:53:11.946 INFO  spatial:Util - using disk index: /opt/splunk/etc/apps/search/lookups/regioni2
07-06-2017 13:53:11.946 INFO  ScopedSTLock - Waiting to acquire lock for path=/opt/splunk/etc/apps/search/lookups/regioni2
07-06-2017 13:53:11.946 INFO  ScopedSTLock - Lock acquired for path=/opt/splunk/etc/apps/search/lookups/regioni2 0
07-06-2017 13:53:11.946 INFO  SearchOperator:Geom - did not find clipped geometry in cache for featureCollection=regioni2
07-06-2017 13:53:11.946 INFO  SearchOperator:Geom - generalization=1
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - Num Keys in ray.key: 1
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - Num Keys in seg.key: 0
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - Num Keys in grid.key: 0
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: ray.key
07-06-2017 13:53:11.947 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/apps/search/lookups/regioni2/ray.key in  0.001 sec
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: ray.val
07-06-2017 13:53:11.947 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/apps/search/lookups/regioni2/ray.val in  0.001 sec
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: seg.key
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file seg.key
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: seg.val
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file seg.val
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: grid.key
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file grid.key
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: grid.val
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file grid.val
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - memmap failed! no valid keys and values
07-06-2017 13:53:11.947 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.947 ERROR DispatchThread - problem memmapping
07-06-2017 13:53:11.948 INFO  UserManager - Setting user context: admin
07-06-2017 13:53:11.948 INFO  UserManager - Done setting user context: NULL -> admin
07-06-2017 13:53:11.948 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.948 INFO  DispatchManager - DispatchManager::dispatchHasFinished(id='1499341991.130', username='admin')
07-06-2017 13:53:11.956 INFO  SearchOperator:Geom - deleting PointInPolygonIndex cache for:  regioni2
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.958 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: problem memmapping
0 Karma

jihape
Path Finder

I have a support ticket open. Hopefully they figure it out soon.

0 Karma

Gibbo87
Explorer

I forgot to mention that I'm on version 6.6.2

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...