topic Re: Error When Creating Choropleth From Custom KMZ File in Dashboards & Visualizations

Error When Creating Choropleth From Custom KMZ File

jihape — Tue, 29 Sep 2020 14:07:26 GMT

Hi,

I'm trying to use a custom kmz file in Splunk (6.5.3), but I keep getting a "problem memmapping" error.

I'm using a KMZ file from google maps - It can be found at https://www.google.com/maps/d/viewer?mid=1Dv1JRTGmzlm83tBv7tb8vQcOQXY&ll=-34.166789639385996%2C143.70463756987533&z=6 (click the three dots next to the magnifying glass)

Here is some of the output from the search.log
05-19-2017 15:19:17.001 INFO SearchOperator:Geom - loading pipIndex named geo_lhds 05-19-2017 15:19:17.002 INFO spatial:Util - could not resolve placemark_specifier, using default path 05-19-2017 15:19:17.002 INFO spatial:Util - fullPath to spatial polygons data file: /opt/splunk/etc/users/admin/search/lookups/geo_lhd_3.kmz 05-19-2017 15:19:17.002 INFO spatial:Util - using disk index: /opt/splunk/etc/users/admin/search/lookups/geo_lhds 05-19-2017 15:19:17.002 INFO ScopedSTLock - Waiting to acquire lock for path=/opt/splunk/etc/users/admin/search/lookups/geo_lhds 05-19-2017 15:19:17.002 INFO ScopedSTLock - Lock acquired for path=/opt/splunk/etc/users/admin/search/lookups/geo_lhds 0 05-19-2017 15:19:17.002 INFO SearchOperator:Geom - did not find clipped geometry in cache for featureCollection=geo_lhds 05-19-2017 15:19:17.002 INFO SearchOperator:Geom - generalization=1.000000 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - Num Keys in ray.key: 1 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - Num Keys in seg.key: 0 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - Num Keys in grid.key: 0 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: ray.key 05-19-2017 15:19:17.002 INFO spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/users/admin/search/lookups/geo_lhds/ray.key in 0.001000 sec 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: ray.val 05-19-2017 15:19:17.002 INFO spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/users/admin/search/lookups/geo_lhds/ray.val in 0.001000 sec 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: seg.key 05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument 05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file seg.key 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: seg.val 05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument 05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file seg.val 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: grid.key 05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument 05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file grid.key 05-19-2017 15:19:17.002 INFO spatial:PointInPolygonIndex - memory mapping existing file: grid.val 05-19-2017 15:19:17.002 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument 05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - can not read index file grid.val 05-19-2017 15:19:17.002 FATAL spatial:PointInPolygonIndex - memmap failed! no valid keys and values 05-19-2017 15:19:17.002 INFO UserManager - Unwound user context: admin -> NULL 05-19-2017 15:19:17.002 ERROR DispatchThread - problem memmapping 05-19-2017 15:19:17.003 INFO UserManager - Setting user context: admin 05-19-2017 15:19:17.003 INFO UserManager - Done setting user context: NULL -> admin 05-19-2017 15:19:17.003 INFO UserManager - Unwound user context: admin -> NULL 05-19-2017 15:19:17.003 INFO DispatchManager - DispatchManager::dispatchHasFinished(id='1495171156.1629', username='admin') 05-19-2017 15:19:17.004 INFO SearchOperator:Geom - deleting PointInPolygonIndex cache for: geo_lhds 05-19-2017 15:19:17.004 INFO UserManager - Unwound user context: admin -> NULL 05-19-2017 15:19:17.005 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: problem memmapping

I suspect there is something wrong with the KMZ/KML file, but what I don't know. The only difference I can see is that the KML file in the KMZ file I downloaded from Google has spaces as indents and there no indents in the geo_us_states.kmz file. I've tried to remove those but it didn't help either.

Any help appreciated!

Re: Error When Creating Choropleth From Custom KMZ File

Gibbo87 — Thu, 06 Jul 2017 12:04:59 GMT

I just incurred in the exact same problem using maps from this website. http://www.gadm.org/download

07-06-2017 13:53:11.944 INFO  SearchOperator:Geom - in getType _reporting is 0
07-06-2017 13:53:11.944 INFO  SearchOperator:Geom - in getType _reporting is 0
07-06-2017 13:53:11.944 INFO  SearchOperator:Geom - loading pipIndex named regioni2
07-06-2017 13:53:11.946 INFO  spatial:Util - get user-defined xpath /Placemark/name
07-06-2017 13:53:11.946 INFO  spatial:Util - fullPath to spatial polygons data file: /opt/splunk/etc/apps/search/lookups/ITA_regioni.kmz
07-06-2017 13:53:11.946 INFO  spatial:Util - using disk index: /opt/splunk/etc/apps/search/lookups/regioni2
07-06-2017 13:53:11.946 INFO  ScopedSTLock - Waiting to acquire lock for path=/opt/splunk/etc/apps/search/lookups/regioni2
07-06-2017 13:53:11.946 INFO  ScopedSTLock - Lock acquired for path=/opt/splunk/etc/apps/search/lookups/regioni2 0
07-06-2017 13:53:11.946 INFO  SearchOperator:Geom - did not find clipped geometry in cache for featureCollection=regioni2
07-06-2017 13:53:11.946 INFO  SearchOperator:Geom - generalization=1
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - Num Keys in ray.key: 1
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - Num Keys in seg.key: 0
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - Num Keys in grid.key: 0
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: ray.key
07-06-2017 13:53:11.947 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/apps/search/lookups/regioni2/ray.key in  0.001 sec
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: ray.val
07-06-2017 13:53:11.947 INFO  spatial:timing:PointInPolygonIndex - memory mapped /opt/splunk/etc/apps/search/lookups/regioni2/ray.val in  0.001 sec
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: seg.key
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file seg.key
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: seg.val
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file seg.val
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: grid.key
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file grid.key
07-06-2017 13:53:11.947 INFO  spatial:PointInPolygonIndex - memory mapping existing file: grid.val
07-06-2017 13:53:11.947 ERROR spatial:PointInPolygonIndex - Memmap failed, file len 0 : Invalid argument
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - can not read index file grid.val
07-06-2017 13:53:11.947 FATAL spatial:PointInPolygonIndex - memmap failed! no valid keys and values
07-06-2017 13:53:11.947 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.947 ERROR DispatchThread - problem memmapping
07-06-2017 13:53:11.948 INFO  UserManager - Setting user context: admin
07-06-2017 13:53:11.948 INFO  UserManager - Done setting user context: NULL -> admin
07-06-2017 13:53:11.948 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.948 INFO  DispatchManager - DispatchManager::dispatchHasFinished(id='1499341991.130', username='admin')
07-06-2017 13:53:11.956 INFO  SearchOperator:Geom - deleting PointInPolygonIndex cache for:  regioni2
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.957 INFO  UserManager - Unwound user context: admin -> NULL
07-06-2017 13:53:11.958 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: problem memmapping

Re: Error When Creating Choropleth From Custom KMZ File

jihape — Thu, 06 Jul 2017 23:06:57 GMT

I have a support ticket open. Hopefully they figure it out soon.

Re: Error When Creating Choropleth From Custom KMZ File

Gibbo87 — Fri, 07 Jul 2017 06:42:24 GMT

I forgot to mention that I'm on version 6.6.2

Re: Error When Creating Choropleth From Custom KMZ File

Gibbo87 — Fri, 07 Jul 2017 06:54:52 GMT

In the mean time I found a sort of workaround.
Apparently the problem is that the XPath expression used to find the XML element containing the featureID is not working. You can let Splunk use the default expression which is /kml/Document/Folder/Placemark/name, the documentation omits the /kml/Document/Folderpart.

I had to restructure my KML adding the Folder element which was not there. My KML now looks like this.

<kml ... >
<Document>
    <Folder>
        <name>ita_regions</name>
    <Placemark>
        <name>Abruzzo</name>
                .....

Apparently The folder name is mapped to the featureCollection field and the Placemark name is mapped to featureId and both have to be there. Here is how the lookup looks like in Splunk after the fix.

Hope this helps.

Re: Error When Creating Choropleth From Custom KMZ File

jihape — Wed, 12 Jul 2017 02:06:32 GMT

I have tried to clean the KML and restructure as well and adding the full XPATH to the "featureId" field and none of it worked for me with the KMZ file I have.

I'll accept your answer as there are probably other people that need to restructure the KML for it to work.