I’m incredibly excited to share a project that’s easily the thing I’ve most enjoyed working on this year. We call it ‘Portolan’, and there’s a great introduction to the whole thing on the CNG blog, published last week. I’d like to share here some of my perspective on it - how it came to be, some of the things we’re trying to do with it, and why I think it has such huge potential. I suspect I have far more than a single post’s worth of stuff to say on it, so it’s likely going to be a frequent topic in the next few months.
Finding spatial data infrastructure
The roots of Portolan for me go back more than 20 years, back to when I first heard about the concept of ‘spatial data infrastructure’. I had been leading development of GeoServer for a couple years, but wanted an international adventure and a friend had previously done a Fulbright scholarship and suggested I apply for one. He told me that the key to getting accepted was to find someone in the country who was interested in what you were working on and willing to write a letter to Fulbright saying they’d support you. I loved working on GeoServer, so wanted to figure out a project that would build on it. I was never actually trained in GIS, but GeoServer was a part of that world, so I started trying to network on ‘GIS stuff’. But it quickly became clear to me that I had no idea how to ‘do GIS’ - I’d never opened a desktop GIS and wouldn’t be any help on projects that were actually GIS. Then I found the term ‘Spatial Data Infrastructure’ and quickly realized that working on that was a much better match for what I wanted to do - GeoServer makes spatial data available for wider use, and I wanted to help enable that.
I ended up in Zambia, looking at the potential for Open Source Software for Spatial Data Infrastructure in developing countries. I can’t say my project was in any way successful - I was hoping to stand up a GeoServer with a bunch of Zambia data to demonstrate the power of spatial information and prove that open source software could work in that context. I didn’t come anywhere close to pulling that off, or even sharing any spatial information. But I did write my first paper, for the Global Spatial Data Infrastructure conference in Cairo (GSDI-8).
I unfortunately can’t find the original paper or slides, but Google does seem to be aware of it (though the OSGeo Library unfortunately seems to not actually store the original files):
So my project was a complete failure, but I did get more spacious time to think about spatial data infrastructures, along with how ideas from ‘Web 2.0’ (the cool new trend at the time) fit in. I wrote and talked about it, giving a keynote on the ideas at GSDI-10. And a couple years later I met Stuart Gill, one of my all-time favorite collaborators, who was able to run with those ideas and build a collaborative set of funders (see this great case study for more on it) so we could turn those ideas into reality with a project named GeoNode.
It took the core OGC standards support of GeoServer and combined it with a much easier-to-use web interface built on Django and OpenLayers, so you could have a full geospatial portal, with the ability to create maps and manage geospatial data. The GFDRR / World Bank did an incredible job of bringing it to a number of countries, and it was really satisfying to see a Zambian GeoNode a few years after I’d spent my time there failing to implement any meaningful SDI. I’ve long abandoned GeoNode, but it’s awesome to see that it is still thriving.
Ten years away from vector
After I stopped working on GeoNode I took about ten years of not working on Spatial Data Infrastructure, or really any vector data. I joined Planet, and I had a good time learning all about raster, which I’d previously only been interested in as a backdrop for vector. Planet did not have quite the open source culture that I was used to, and I have a strong desire to share most of my intellectual work openly, but I ended up finding a good outlet for that with open standards. After years of being frustrated with OGC specs being a bit disconnected from the reality of implementation I guided Cloud-Optimized GeoTIFF, SpatioTemporal Asset Catalog and GeoParquet formats to be ‘implementation first’ and completely developer-driven. The spec was the side-effect of an ecosystem of software working together and evolved with the software, with a big focus on making the core as small as possible for maximum flexibility and adoptability.
While I worked on the types of geospatial information I knew well we also had Howard Butler adopt the ‘cloud-optimized’ playbook for point cloud data, and Ryan Abernathey built up the use of Zarr for big climate data and other multi-dimensional data. Together this covered most every traditional geospatial format with a cloud-native equivalent.
Towards a cloud-native SDI
After GeoParquet I knew I didn’t want to work on yet another format. And I started thinking about this old ‘Spatial Data Infrastructure’ idea again. While GeoServer and GeoNode were nice tools for sharing data they were still a ton of work for the data publisher to run. Most organizations producing geospatial data were not experts in running web services, so practically the majority of data in Spatial Data Infrastructures was from national agencies that did have sufficient expertise to run services. But with all the formats covered by cloud-native equivalents I could see a world where you could swap a GeoServer or GeoNode for a bunch of files on a cloud bucket, and at least match the user experience of interacting with an SDI. But it would be way more scalable, much cheaper to operate, and we likely could beat what most data portals offer.
I gave a talk called ‘Towards a Cloud Native Spatial Data Infrastructure’ (slides / youtube) at FOSS4G-NA 2023 to articulate this vision and hopefully rally governments to adopt it. The early rise of AI made it even more clear to me that all the past SDI initiatives had fundamentally fallen far short of the vision to ‘enable the discovery and use of geospatial information by users and for purposes other than those it was created’ (Kuhn 2005) or to deliver ‘spatially related data from many different sources to the widest possible group of potential users at affordable costs’ (Groot & McLaughlin 2000).
The talk included this gif:
If we were living up to the SDI vision this should be an easy question to answer! Or the example Jen Marcus (another of the people I’ve most enjoyed working with) and I would give in the early days of Fields of The World (FTW) - ‘How many fields are there in Kansas?’ If SDIs had been successful then this should be trivially easy for an LLM to answer - they sucked up all the world’s information and so of course geospatial information should be there too. But I see the lack of that information as a failure of our industry - we should have done better and actually made foundational geospatial information open and accessible for the frontier labs to gobble it up.
A QGIS plugin and geoparquet-io
Around then I started to get back into coding thanks to the power of AI coding tools, releasing a QGIS plugin that could download a subset of GeoParquet, to help demonstrate that the experience of interacting with a remote file on a cloud-bucket is actually better than the Web Feature Service / OGC Features API experience I envisioned replacing. It’s easier to connect to and far faster, it’s integrated directly in your desktop GIS, and you can then convert it into any file format you want. The experience proves that GeoParquet doesn’t just replace a Shapefile or GeoPackage, it replaces a GeoServer or ArcGIS Server.
But one issue with supporting that use case is that with GeoParquet we embraced the flexibility of Parquet, so though you could put up a file that would replace a WFS it was also easy to put up a file that wasn’t well-optimized and wouldn’t actually compare favorably. I wrote up Best Practices for Distributing GeoParquet to capture the consensus of what a ‘good’ geoparquet looks like, for the publishing use case. And that in turn inspired me to start geoparquet-io (aka gpio), to have one-line CLI commands and python calls that make GeoParquets that follow those best practices, and to make spatial partitioning of GeoParquets much easier. At the time there were good DuckDB recipes to do it, but nothing that let you just say something like “gpio partition kdtree input.parquet output/”. I also wanted to enable gpio partition admin input.parquet output/ to make it incredibly easy for anyone to make The Admin-partitioned GeoParquet Distribution that I’d been exploring.
Finding the collaborators
That project managed to bring Nissim Lebovits, the newest on my list of ‘all-time favorite collaborators’, into my orbit. He started out making PRs to gpio, and then quickly went beyond being just a contributor and became a true co-maintainer, taking geoparquet-io far beyond what it would have been with just me, and inspiring me to finish many of the features I’d been thinking about. At some point last year we got to talking and it turned out he was using gpio as part of a Fulbright in Argentina. He was ‘exploring how cities in the Global South can leverage open data and emerging technologies to adapt to climate change without having access to enormous resources or expensive infrastructure.’ (from his blog post).
It was awesome to learn that twenty years after my failed attempt in Zambia he was on the same scholarship program, pursuing more or less the same idea. He was a hundred times more successful than I was, and the difference was that the CNG community has built some incredible tools and data in the last few years. Nissim was able to make real-world impact by leveraging data on Source Coop to update flood exposure maps (and he even contributed to the global CNG data commons, putting up Microsoft ML Roads as GeoParquet & PMTiles). Read his post on it, or watch his interview, which came out after I started this post.
We were both inspired by the idea of making it much easier for less resourced governments to leverage cloud-native geo to ‘easily share open data in a low-cost, accessible, sovereign, reliable and user-friendly manner’ as I put in the first description of the project. As Nissim transitioned from the Fulbright he started working on the tools to enable easy publishing of whole catalogs of cloud-native data. Radiant Earth then managed to find some funding to enable him to continue to build it out. He’s been the driving force behind the entire project: setting the direction, organizing the community and leading our meetings, converting data, building the website and all the messaging, and orchestrating Claude across ~15 different repositories. It’s been incredible to see the total output he’s been able to create - without him Portolan would still just be an idea. This first release is not a prototype: it’s a complete ecosystem of tools that can be used today to create and use Portolan catalogs.
When Portolan was still mostly a concept I also started chatting about the ideas with Javier de la Torre from CARTO. He had been seeing a big desire from governments for sovereignty, and also had a similar vision of cloud-native files fully implementing SDI. He immediately started prototyping lots of advanced concepts including Iceberg catalogs (which we all decided weren’t in scope for the first release, but have huge potential) and cross-jurisdiction agent workflows. He’s since oriented CARTO to build the first commercial offering that’s a part of our Portolan ecosystem. He brought Youssef Harby and Cayetano Benavent in to work on it, and they’ve both made huge contributions in the last couple of months. And then Matthias Mohr has also joined the community, providing expert STAC guidance and bringing a higher degree of rigor to our specifications. So already it is a true open source community with diverse contributions from several different organizations.
Portolan today
I’m quite proud of what this small group of people has been able to build. I believe it demonstrates without any doubt that the Cloud-Native Spatial Data Infrastructure I talked about a few years ago isn’t just possible - it’s actually an upgrade from what exists today.
The above is the first Portolan catalog I built, porting a subset of the amazing PDOK - the national open data portal for the Netherlands. There are no servers in the demo at all, which means the cost to serve it up is significantly less than it costs to keep a WMS/WFS up and running - you just pay for data storage (pennies per gigabyte) and ‘egress’ (how much data is transferred). And it will seamlessly scale to any load, which is important in the age of AI when open data servers are getting hammered with agent requests. I believe the user experience is much better - you can instantly see the data, try out different styles and explore the full dataset. With Portolan we also include all the information on the columns right there, and the Portolan Browser will also display what’s actually in the data rows:
The data will already work with a wide variety of tools - anything that can understand STAC will make sense of it, and any tool reading GeoParquet or COGs can make use of the data.
And then the ‘killer feature’ is that following the Portolan spec makes it so your data can be fully understood by agents. I’ve been frankly blown away by how well this works, and in the coming weeks we’ll all share more details on the Portolan Blog. You can ask questions like ‘How many tulip fields are in the Netherlands’ or have a conversation about what crops are most used in different provinces:
The tulip answer, for the record, is that Noord-Holland holds about 13,800 hectares across 1,526 parcels, and together with Flevoland accounts for over three quarters of all Dutch tulip growing. These are done by the agents querying the actual data with DuckDB, so it’s not going to be a hallucinated / non-deterministic answer - the LLM just needs to get the SQL right. And every Portolan dataset has an agents.md that guides the agent to the right SQL.
So all we need to do to enable the answer to ‘how many fields are there in Kansas’ is to point an agent at a Portolan dataset that has the fields in Kansas (FTW is getting there, the numbers are likely not quite accurate, but as new versions of FTW are released they will get closer).
None of which is to say it is finished, so let’s cover some of the places we still need work. The spec for Portolan is still 0.2, so we expect a number of changes, including potential breaking ones, in our path to 1.0. So the project for now is best suited to early adopters, but we’re optimistic that we’ll be iterating quickly and start to serve a wider variety of users. We’ve got a working roadmap that highlights some of the holes, including that normative Zarr and COPC support isn’t there yet. Searching across a whole large catalog, let alone across the whole registry, still needs collection-level stac-geoparquet and the tooling to go with it. Mirrors do not yet sync from their live upstream sources, so they go stale. And access-controlled and non-public data is a v1.0 item, not something you can do today.
Each of those is a place you can jump in and help out. If you work on Zarr or point clouds, the conformance profiles for them are wide open - we want to get the recommendations right and ensure all the tooling is there. If catalog-scale search is your thing, that is the single biggest gap between what Portolan does now and what the registry needs to become. If you publish data, the syncing problem is one you already understand better than we do. We’re still a small community of contributors, so if you’re able to show up and help out it can make a big difference.
Portolan’s potential
So my more ‘modest’ hope for Portolan is that it greatly accelerates the adoption of CNG formats by the geospatial industry, and the standard way to share geospatial information becomes to put a GeoParquet, COG, COPC or Zarr in a bucket and include the appropriate catalog.json/readme/agents.md file. It’s a simpler, more scalable, more cost-effective architecture. And I think the ‘hook’ of AI-enabling existing datasets has the potential to drive that adoption, so much of the coming months will go into refining the messaging, tutorials, value propositions and case studies.
But the bigger goal I have for Portolan is to realize the vision of all the Spatial Data Infrastructure dreams of the past - to finally make the world’s fundamental geospatial information available and accessible to all. I believe this is now a tractable problem that can be solved, by leveraging AI in a couple of key ways.
The first is to use AI tools to create Portolan catalogs. The process of creating a Portolan version of a single open dataset is a few minutes, and converting an entire open data portal is no more than a few hours - depending on the number of layers and how refined the output is. We provide a number of agent skills for Portolan, and one of the big goals over the next few months is to continue to iterate on the skills by creating many catalogs, to get to the point where a single prompt can create a high quality catalog (at least if the source data is well documented). But it’s already quite good, usually creating a variety of styles on its own, generating meaningful thumbnails, documenting the columns, etc. While the end goal is for providers themselves to publish Portolan catalogs we can also just convert existing data into ‘mirrors’ that then become part of the overall Portolan network.
And then the second is to use AI to query that network - the Portolan registry. This is also going to take refinement as we get to hundreds of catalogs, as we’ll need to guide agents to the right data. But I see a future where the Portolan registry + AI provides a conversational interface to all the world’s geospatial information. And where that frictionless experience leads to even more pull for governments and NGOs to publish their data openly. While GIS people love maps, most of the world just wants answers. For a while people will need to know to include the Portolan registry in their questions, but I believe if we can show enough value from it then the default experience of any AI tool will include pulling relevant geospatial information into any question.
Twenty years ago I went to Zambia to show that open source software could stand up a spatial data infrastructure node, and came home with nothing to show for it but a conference paper. With Portolan I believe we’re about to usher in an era where anyone can stand up an SDI node in a few hours. And if enough of us stand up those nodes we’ll truly achieve those original SDI goals and make spatial information a part of every major decision that affects our lives.
If anything in here intrigues you, please join us. Start by using the Portolan skills to convert some data you think people should use — if your government has good open data that’s a good place to start, or grab an open issue if you don’t have ideas — then publish it to the registry and help us improve the overall ecosystem.
I’ll explore these topics and more in coming posts.