Optimising Mappt for Viewing Large Vector Datasets

Working with spatial data sometimes has the user wrangling vector features with tens, perhaps hundreds, of thousands of polygon or polyline edges.

While the memory capacity of modern mobile devices is more than capable of handling such large datasets, which can have file sizes in the tens of megabytes even after compression, the devices can struggle to display such a large volume of data on-screen.

The end result is an app that is sluggish to use at best, if you’re lucky enough to avoid the “App Not Responding” alerts from the Android OS, with it’s helpful offer to terminate the app for you.



So, Mappt was having trouble viewing large datasets.  What to do?

The Thinker

We needed a solution, so I turned into a statue and pondered.

For Mappt, rather than attempt to define and enforce an arbitrary restriction on file sizes when loading data, which in itself is a difficult stance to decide (how much is too much?), we instead applied some good ol’ performance optimisation techniques.

When you have tens of thousands of polygon or polyline edges on-screen, your device becomes bogged down as it tries to draw and redraw each feature as you scroll the map around – there is simply too much work for your poor device to do.  So, let’s give it less work!

As Mappt is tasked with drawing more and more complicated shapes, it keeps track of how much work it is doing.  If it decides it is getting overloaded, it will reduce the fidelity of the shapes it is being asked to draw.  It does this in a manner so as to have the least impact possible on the shapes themselves.

For example, consider the following shape.

Let's just pretend it's the future plan of my free-range Quokka farm.

Let’s just pretend it’s the future plan of my free-range Quokka farm.

In terms of the points that make up the shape, it is rather plain on the right side, while somewhat busier on the left side. Now, consider that we want to “make it less work” for the device to draw this shape, which we will achieve by removing one of the points. This, itself, requires some thought, as we want to maintain the general “look” of the shape as much as possible.

If we remove the worst possible point, being one of the more prominent ones, we end up with this abomination:

The architect of my farm is not going to be happy.

However, if we remove the best possible point, we barely see any change, as an inconspicuous point in the lower-left of the shape has been removed:

He'll be none the wiser.

If you can’t see which point has been removed, that’s good!

We can see that it is important to remove the “best”, or least prominent, points first. Essentially, we want to remove points that will have the least impact on the overall shape. We can then continue point-by-point in this fashion until we have reduced the fidelity of the shape to a comfortable degree.

The number of points removed is generally determined by a “level of detail” percentage, with 100% meaning no points are removed and shapes retain full fidelity, or a value such as 50% meaning half of the points in the shape have been removed. Mappt determines this “level of detail” factor by measuring the current workload, i.e. it only drops fidelity when, as as much as, is needed.

Because nothing ever comes easy, we do eventually run into an understandable problem in that if we reduce the fidelity of a shape so much, it eventually becomes unrecognisable as seen in the following image – the original shape has been reduced to roughly 50% fidelity, becoming almost unrecognisable as around half of the points have been removed.  However, it still sort-of almost kinda has the same overall shape, if you squint.

The architect of my farm is not going to be happy.  Or maybe he will.  We're yet to find a good measure of architect happiness.

Again, the architect is not going to be happy. Or maybe he will. We’ve yet to find a good measure of architect happiness.

We’re very aware that this isn’t a “free” optimisation, as more optimisation means less accuracy.  To that end, we are keeping our earholes directed at the community of Mappt users to see if the optimisations are being too aggressive.  Should we receive feedback that Mappt is messing with the shape of… shapes too much, we will fine-tune the heuristics, or may even allow the user to choose the level of fidelity they are happy with – balancing app responsiveness against shape fidelity.

So, if you happen to see Mappt ruining your shapes, be sure to let us know!  Together, we can free the shapes from their aggressive fidelity overlords!