Planet Plone

This is where developers and integrators write about Plone, and is your best source for news and developments from the community.

March 23, 2017

Reinout van Rees: Fossgis: open source for emergencies - Marco Lechner

by Reinout van Rees at 2017-03-23T14:28:00Z

(One of my summaries of a talk at the 2017 fossgis conference).

He works for the Bundesamtes fuer Strahlenschutz, basically the government agency that was started after Chernobil to protect against and to measure radioactivity. The software system they use/build is called IMIS.

IMIS consists of three parts:

  • Measurements (automatic + mobile measurements + laboratory results).
  • Prediction system. Including documentation (managed in Plone, a python CMS system).
  • Decision support. Help support the government layers that have to make the decisions.

They have a simple map at

The current core of the system is proprietary. They are dependent on one single firm. The system is heavily customized for their usage.

They need a new system because geographical analysis keeps getting more important and because there are new requirements coming out of the government. The current program cannot handle that.

What they want is a new system that is as simple as possible; that uses standards for geographical exchange; they don't want to be dependent on a single firm anymore. So:

  • Use open standards, so OGC. But also a specific world-wide nuclear info protocol.
  • Use existing open source software. OSGEO.
  • If we need something special, can we change/extend existing open source software?
  • If not, then it is OK to create our their software. Under an open source license.

They use open source companies to help them, including training their employees. And helping getting these employees used to modern software development (jenkins, docker, etc.)

If you use an open source strategy, what do you need to do to make it fair?

  • Your own developments should also be open source!
  • You need your own test and build infrastructure. (For instance Jenkins)
  • You need to make it easy to start working with what you made: documentation, docker, buildout (!), etc.

(Personal note: I didn't expect to hear 'buildout' at this open source GIS conference. I've helped quite a bit with that particular piece of python software :-) )

Reinout van Rees: Fossgis: sewer cadastre with qgis - jörg Höttges

by Reinout van Rees at 2017-03-23T10:24:00Z

(One of my summaries of a talk at the 2017 fossgis conference).

With engineer firms from the Aachen region they created qkan. Qkan is:

  • A data structure.
  • Plugins for Qgis.
  • Direct access. Not a specific application with restricted access, but unrestricted access from within Qgis. (He noticed lots of interest among the engineers to learn qgis during the project!)

It has been designed for the needs of the engineers that have to work with the data. You first import the data from the local sewer database. Qkan converts the data to what it needs. Then you can do simulations in a separate package. The results of the simulation will be visualized by Qkan in qgis. Afterwards you probably have to make some corrections to the data and give corrections back to the original database. Often you have to go look at the actual sewers to make sure the database is correct. Output is often a map with the sewer system.

Some functionality: import sewer data (in various formats). Simulate water levels. Draw graphs of the water levels in a sewer. Support database-level check ("an end node cannot occur halfway a sewer").

They took care to make the database schema simple. The source sewer database is always very complex because it has to hold lots of metadata. The engineer that has to work with it needs a much simpler schema in order to be productive. Qkan does this.

They used qgis, spatialite, postgis, python and qt (for forms). An important note: they used as many postgis functionality as possible instead of the geographical functions from qgis: the reason is that postgis (and even spatialite) is often much quicker.

With qgis, python and the "qt designer", you can make lots of handy forms. But you can always go back to the database that's underneath it.

The code is at

by Reinout van Rees at 2017-03-23T09:55:00Z

(One of my summaries of a talk at the 2017 fossgis conference).

The conference started with a quick introduction on several open source programs.

Openlayers 3 - Marc Jansen

Marc works on both openlayers and GeoExt. Openlayers is a javascript library with lots and lots of features.

To see what it can do, look at the 161 examples on the website :-) It works with both vector layers and raster layers.

Openlayers is a quite mature project, the first version is from 2006. It changed a lot to keep up with the state of the art. But they did take care to keep everything backwards compatible. Upgrading from 2.0 to 2.2 should have been relatively easy. The 4.0.0 version came out last month.


  • Allows many different data sources and layer types.
  • Has build-in interaction and controls.
  • Is very actively developed.
  • Is well documented and has lots of examples.

The aim is to be easy to start with, but also to allow full control of your map and all sorts of customization.

Geoserver - Marc Jansen

(Again Marc: someone was sick...)

Geoserver is a java-based server for geographical data. It support lots of OGC standards (WMS, WFS, WPS, etc). Flexible, extensible, well documented. "Geoserver is a glorious example that you can write very performant software in java".

Geoserver can connect to many different data sources and make those sources available as map data.

If you're a government agency, you're required to make INSPIRE metadata available for your maps: geoserver can help you with that.

A big advantage of geoserver: it has a browser-based interface for configuring it. You can do 99% of your configuration work in the browser. For maintaining: there is monitoring to keep an eye on it.

Something to look at: the importer plugin. With it you get a REST API to upload shapes, for instance.

The latest version also supports LDAP groups. LDAP was already supported, but group membership not yet.

Mapproxy - Dominik Helle

Dominik is one of the MapProxy developers. Mapproxy is a WMS cache and tile cache. The original goal was to make maps quicker by caching maps.

Some possible sources: WMS, WMTS, tiles (google/bing/etc), MapServer. The output can be WMS, WMS-C, WMTS, TMS, KML. So the input could be google maps and the output WMS. One of their customers combines the output of five different regional organisations into one WMS layer...

The maps that mapproxy returns can be stored on a local disk in order to improve performance. They way they store it allows mapproxy to support intermediary zoom levels instead of fixed ones.

The cache can be in various formats: MBTiles, sqlite, couchdb, riak, arcgis compact cache, redis, s3. The cache is efficient by combining layers and by omitting unneeded data (empty tiles).

You can pre-fill the cache ("seeding").

Some other possibilities, apart from caching:

  • A nice feature: clipping. You can limit a source map to a specific area.
  • Reprojecting from one coordinate system to another. Very handy if someone else doesn't want to support the coordinate system that you need.
  • WMS feature info: you can just pass it on to the backend system, but you can also intercept and change it.
  • Protection of your layers. Password protection. Protect specific layers. Only allow specific areas. Etcetera.

QGis - Thomas Schüttenberg

QGis is an opern source gis platform. Desktop, server, browser, mobile. And it is a library. It runs on osx, linux, windows, android. The base is the QT ui library, hence the name.

Qgis contains almost everything you'd expect from a GIS packages. You can extend it with plugins.

Qgis is a very, very active project. Almost 1 million lines of code. 30.000+ github commits. 332 developers have worked on it, in the last 12 months 104.

Support via documentation, mailinglists and . In case you're wondering about the names of the releases: they come from the towns where the twice-a-year project meeting takes place :-)

Since december 2016, there's an official (legal) association.

QGis 3 will have lots of changes: QT 5 and python 3.

Mapbender 3 - Astrid Emde

Mapbender is library to build webgis applications. Ideally, you don't need to write any code yourself, but you configure it instead in your browser. It also supports mobile usage.

You can try it at . Examples are at .

You can choose a layout and fill in and configure the various parts. Layers you want to show: add sources. You can configure security/access with roles.

An example component: a search form for addresses that looks up addresses with sql or a web service. Such a search form can be a popup or you can put it in the sidebar, for instance. CSS can be customized.

PostNAS - Astrid Emde, Jelto Buurman

The postnas project is a solution for importing ALKIS data, a data exchange format for the german cadastre (Deutsch: Kataster).

PostNAS is an extension of the GDAL library for the "NAS" vector data format. (NAS = normalisierte Austausch Schnittstelle, "normalized exchange format"). This way, you can use all of the gdal functionality with the cadastre data. But that's not the only thing: there's also a qgis plugin. There is configuration and conversion scripts for postgis, mapbender, mapserver, etc.

They needed postprocessing/conversion scripts to get useful database tables out of the original data, tables that are usable for showing in QGis, for instance.

So... basically a complete open source environment for working with the cadastre data!

Photo explanation: just a nice unrelated picture from the recent beautiful 'on traxs' model railway exibition (see video )

by Reinout van Rees at 2017-03-23T09:55:00Z

(One of my summaries of a talk at the 2017 fossgis conference).

He wanted to make a map for a local run. He wanted a nice map with the route and the infrastructure (start, end, parking, etc). Instead of the usual not-quite-readable city plan with a simple line on top. With qgis and openstreetmap he should be able to make something better!

A quick try with QGis, combined with the standard openstreetmap base map, already looked quite nice, but he wanted to do more customizations on the map colors. So he needed to download the openstreetmap data. That turned into quite a challenge. He tried two plugins:

  • OSMDownloader: easy selection, quick download. Drawback: too many objects as you cannot filter. The attribute table is hard to read.
  • QuickOSM: key/value selection, quick. Drawback: you need a bit of experience with the tool, as it is easy to forget key/values.

He then landed on . The user interface is very friendly. There is a wizard to get common cases done. And you can browse the available tags.

With the data downloaded with overpass-turbo, he could easily adjust colors and get a much nicer map out of it.

You can get it to work, but it takes a lot of custom work.

Some useful links:

Photo explanation: just a nice unrelated picture from the recent beautiful 'on traxs' model railway exibition (see video )

Jazkarta Blog: What Makes a Project Expensive?

by Sally Kleinfeldt at 2017-03-23T01:34:29Z

When clients are contemplating a new web development project, the question we most often hear is “How much will it cost?” The answer is usually “That depends.” Many factors can make a project more or less expensive. Here are a few to keep in mind, courtesy of the Jazkarta team (especially David Glick.)

  • How important is it to have a polished visual design? The more polished and refined, the more time and money will be spent.
  • Does everything need to be perfectly customized for your organization? If you are open to compromises that allow things to work in ways that are simpler to build you could reduce costs.
  • Is the functionality public-facing or internal? This can play into the previous two questions. In general it is easier to make compromises and keep expenses down on an internal site.
  • Does the project focus on a single feature area or a bunch of loosely connected areas? How much variation is there within a feature? How standardized are your processes? More features and variations lead to more work and more expense. Standardizing on one way of doing things might save on both development budget and staff time.
  • Is the functionality managed by a single part of the organization or does it require buy-in across the organization? It can be difficult to get a diverse group of stakeholders to agree, and their diverse opinions might cause the scope of the project to increase. Try to prevent surprise stakeholders with a lot of organizational clout from emerging late in the project and announcing significant “critical” changes. This can wreak havoc with budgets and demoralize project staff.
  • Does the project require coordination between multiple teams? This will take time and is another path to potential scope creep.
  • Can the project be built on one major platform, or does it require integrating multiple platforms? Each integration will increase expense.
  • How much traffic is expected? Traffic determines how much effort needs to be put into scaling and optimizing the site.
  • Are you building a tool tailor-made for your organization, or something that you aim to release for use by other organizations? Generalizing features often takes more time and makes writing automated tests more critical.
  • Does the project require migrating data from an old system? This will add to costs in proportion to the complexity of the data.
  • If the project involves updates to an existing system, how well-maintained is it? Poorly maintained systems are more difficult to work with.

Big web development projects are daunting and it can be hard to figure out your budget and stick to it. The two best tools for controlling costs are:

  1. A thorough discovery process with your technology partners, to figure out what exactly you need to build and how you’re going to build it.
  2. A flexible implementation process that lets you change your mind when you discover problems or opportunities – an “agile” as opposed to a “waterfall” process.

If you’d like to hear more and you’re coming to the Nonprofit Technology Conference in Washington DC this week, come hear Karen Uffelman (of Percolator Consulting), Loren Drummond (of Washington Trails Association) and me talk about a big project we did together to develop a volunteer management system. The session is called For the Love of Volunteers! How Do You Choose the Right Technology to Help Manage Them?

Hope to see you there!

Tagged: 17ntc, cost, discovery, web development

February 26, 2017

Asko Soukka: Tile based layouts and ESI on Plone

by Asko Soukka at 2017-02-26T20:27:57Z

Plone's Blocks: Grid based layouts is an old manifest (originally dated back to 2008 or 2009) about simplifying Plone's (ME)TAL-macros, content providers (portlets and viewlets) based layout machinery with a composition of independently rendered static HTML layouts and dynamic content tiles. The initial implementation of the manifest was completed already by 2011 with the first versions of plone.tiles and It was supposed to be a core part of Plone 5, but continues to be delayed because the failure of the Plone Deco project. Sad.

Because of the separation of content and composition, the new approach introduced new options for the actual page rendering process: It was no longer necessary to render a complete Plone page at once, but each page could be composed of multiple independently rendered and cached content fragment. Of course, the complete rendering could still be done on Plone at once like before, but it also became possible to compose the page from its fragments with ESI(Edge Side Includes) in a compatible caching proxy, or with JavaScript (e.g. with pat-inject) in an end users browser. Both of these approaches providing parallel rendering of page fragments, while each of those fragment could be cached independently simply by using the standard HTTP caching headers. Which is great.

So, what could all this mean in practice? Thanks to tile based layouts, Varnish and ESI, we are now able to cache every cacheable part of our pages also for logged-in users, resulting in noticeably better performance and user experience.

(And yes, this approach may look already outdated, when compared various front-end based composition options of headless CMS era, but still solves real issues with the current Plone with server-rendered HTML.)

Blocks rendering process revisited

To really understand the goals of tile based layouts, let's revisit the once revolutionary page composition process implemented in

In the simplest form of rendering, a Plone page could still render a complete HTML document as before:

<!DOCTYPE html>
<!-- ... -->

But instead of always rendering everything, with tile based layouts it become possible to speed up the main rendering of the page by delegating the rendering of dynamic parts of the page to separate independent renderers, called tiles:

<!DOCTYPE html>
<!-- ... -->
<div data-tile="TILE_URL" />
<!-- ... -->

The page rendering output could include as many plaholder elements with data-tile-attribute as required, and expect something later to replace those elements with the contents defined by their URL values. This something is still Plone by default, but it could also be a middleware (like ESI in caching proxy) or JavaScript in a browser.

The main benefits from decoupling rendering and composition like this include (with either ESI or JavaScript -based composition) include:

  1. Experiential speed-up, because the main page may be already partially or completely visible in the browser while the delegated parts are still being rendered
  2. real speed-up, because the delegated parts may be rendered in parallel
  3. real speed-up, because the delegated parts may be cached separately with optimized HTTP caching headeaders.

It's crucial that the value of data-tile attribute is full absolute or relative URL, and that the target address can be rendered independently from the main page. These assumptions would make the composition logically independent from then underlying server technology. It's even possible to compose page from snippets rendered by multiple different services.

In addition to the data-tile-composition, provides an additional composition to separate content area page design (content layout) from its surroundings (site layout).

To use this additional site layout composition, a page rendering must define the URL of the used site layout and the panels (slots) it fills into that layout by using the additional data-attributes data-layout and data-panel as in the following example:

<!DOCTYPE html>
<body data-layout="LAYOUT_URL">
<div data-panel="PANEL_ID">
<!-- ... -->
<div data-tile="TILE_URL" />
<!-- ... -->

Together, these attributes instruct the composition as follows: Please, load a site layout at LAYOUT_URL and render it with its panel named PANEL_ID filled with childrens of this tag.

So, if the site layout in question would look like:

<!DOCTYPE html>
<!-- ... -->
<div data-panel="PANEL_ID">
<!-- ... -->
<!-- ... -->

The main rendering of the page would look like:

<!DOCTYPE html>
<!-- ... -->
<!-- ... -->
<div data-tile="TILE_URL" />
<!-- ... -->
<!-- ... -->

Obviously, the site layout could define multiple panels, and the content layout could fill anything from none to all of them.

Currently, this so called panel merge is always done by Plone with transform code in, but technically this could also be done e.g. in a WSGI middleware, releasing Plone worker threads even more earlier than is currently possible with just ESI or browser side composition of tiles.

Caching ESI tiles for logged-in users

ESI (Edge Side Includes) is an old proposal (mainly by Akamai) for an XML namespace to describe HTML page composition from multiple separate resources. A quite minimal subset of the language is implemented also in Varnish, a popular and recommended caching proxy also in Plone setups.

Using and enabling ESI with plone.tiles, and Varnishis well documented in those packages' READMEs. Yet, something we discovered only very recently was, how to use ESI to safely cache tiles for logged-in users.

Of course, by default, Plone never caches anything for logged-in-users. At first, declares all responses private, unless they should be visible for anonymous users. And then, the recommended Varnish configuration skips caches whenever Plone session cookie is present in request. So, by default, we are protected from both sides. (And that's great design to protect us from our own mistakes!)

The first step to support caching for logged-in users is to allow Varnish (in default.vcl) to do cache lookup for ESI tiles:

sub vcl_recv {
# ...
if (req.esi_level > 0) {
set req.http.X-Esi-Level = req.esi_level;
return (hash);
} else {
unset req.http.X-Esi-Level;
# ....

Of course, this would allow lookup for only completely public tiles, because only those could be cached by Varnish by default. That's why, in the example above, we also manage a completely new header X-Esi-Level, and we make sure it's only available when Varnish is doing its internal subrequests for ESI-tiles.

With that extra header in place, we can instruct Varnish to hash responses to ESI-subrequests separately from responses to main requests. In other words, we split Varnish cache into public and private areas. While public cache remains accessible for anyone knowning just the cached URL, the private one is only accessible for Varnish itself, when it's doing ESI subrequests:

sub vcl_hash {
if ( {
} else {
if (req.http.X-Esi-Level) {
return (lookup);

Now we are almost ready to patch let Plone to allow caching of restricted tiles. But only tiles. Because of X-Esi-Level would only be set for Varnish's internal subrequests for tiles, all other requests would be handled as before. This patch is done by monkeypatching an utility method in allow public Cache-Control-header for otherwise restricted tiles when the trusted X-Esi-Level-header is in place:

def visibleToRole(published, role, permission='View'):
request = getRequest()
if request.getHeader('X-Esi-Level'):
return True
return role in rolesForPermissionOn(permission, published)

Please, don't do this, unless you really know and test what you are doing!

Because something crucial is still missing: Even with the private cache for ESI tiles, Varnish will still cache tiles just by URL. And that would mean that, by default, all users would get whatever version of the tile was cached first.

To really make it safe to cache restricted tiles for logged-in users, we must ensure that cacheable tiles should have unique URL for users with different roles. We fixed this by implementing a custom transform, which hashes roles of the current user (with time based salt) into an extra query string parameter for each tile URL in the rendered page. The result: users with same set of roles share the same cache of tiles. Fast.

February 23, 2017

Jazkarta Blog: Two Jazkarta Projects Competing for Digital Humanities Award

by Sally Kleinfeldt at 2017-02-23T22:06:20Z

Byzantine Seals Exhibit

The annual Digital Humanities Awards allow the public to recognize outstanding digital humanities resources on the web. This year not one but two Jazkarta projects are nominated in the “Best Use of Digital Humanities for Public Engagement” category!

  • Pleiades, a gazetteer of ancient places that recently won the Archaeological Institute of America’s 2017 Award for Outstanding Work in Digital Archaeology. It lets enthusiasts use, create, and share geographic information about the ancient world – associating locations with ancient places, names and scholarly references.
  • Dumbarton Oaks’ Online Catalog of Byzantine Seals, which allows users to browse photographs and detailed information about the 17,000+ seals in the collection. Tools include a faceted search, bibliography, and “save my favorites” feature, as well as two informative online exhibitions.

Jazkarta helped develop the software that powers these sites – in both cases, custom add-ons for the Plone open source content management system. But the real work was done by the Pleiades and Byzantine Seals teams who conceived and designed these projects and did the heavy lifting of data entry and curation.

You can cast your vote for either of these projects – or for the National Endowment for the Humanities’ Office of Digital Humanities – by visiting and completing the voting form linked at the bottom before midnight this Saturday, 25 February.

Tagged: ancient world, award, cms, digital humanities, dumbarton oaks, isaw, pleiades, Plone

February 13, 2017 Towards Plone 6


A report from the Alpine City Sprint in Innsbruck

February 06, 2017 Push coverage report out of an Gitlab CI Runner


For our Plone/Python projects we often generate coverage reports as HTML sites, this posts show how you can push this report out of the Gitlab CI Runner.

January 31, 2017

Affinitic: No more JS / CSS browser cache refresh pain

by Laurent Lasudry at 2017-01-31T13:42:47Z

We often have the following problem :

  1. a CSS or a Javascript file is changed in a package
  2. we make a release (or not) and put it to production
  3. some visitors still gets old CSS and Javascript versions (except if by chance we remembered to reinstall package / re-save portal_css and portal_javascripts configs) and need to flush their browser cache

As we do not have great memory and as our customer’s Helpdesk was tired of doing remote cache flush, we wanted to do something (automatic) about it.
Plone adds cache keys as suffixes for CSS and JS files, as for example base-cachekey6247.css.
This cache key doesn’t change, even after an instance restart. This is why browsers can keep the “old” versions cached.

To avoid that (and without touching to Apache or whatever), we wrote a little script that exectutes at the end of the instance startup process.It forces the “cook” of resources which generates a new cache key so that browser cannot cache CSS and JS anymore !

This script can of course be improved, but it fits our needs.

[Please note that this code is written for Plone 3.]
First, we subscribe to IDatabaseOpenedWithRootEvent (too bad we couldn’t use IProcessStarting because we have no context to get Plone sites) :


Then, our handler gets the work done :

# -*- coding: utf-8 -*-
from Products.CMFCore.utils import getToolByName
from import getInformationFromEvent
import transaction
from our.package import logger
def clear_cache(event):
    Force cache key renewal for CSS and JS on all Plone sites at startup.
    This avoids any caching of changed CSS or JS.
    db, connection, root, root_folder = getInformationFromEvent(event)
    app = root_folder
    for obj_tuple in app.items():
        (obj_name, obj) = obj_tuple
        if obj.get('portal_type') == 'Plone Site':
            p_css = getToolByName(obj, 'portal_css')
            p_js = getToolByName(obj, 'portal_javascripts')
  'Cache key changed for CSS/JS on Plone %s' % obj_name)

And voilà, no more browser cache flush hassle for our customer and their Helpdesk :-)

If you have any thoughts on this, or think that this should be packaged in a collective egg (useful for you ?), feel free to comment.

January 30, 2017

Jazkarta Blog: Pleiades Wins an Award and Gets an Upgrade

by Sally Kleinfeldt at 2017-01-30T20:18:39Z

Via Appia entry on Pleiades

Here at Jazkarta we’ve been working with the Institute for the Study of the Ancient World (ISAW) for the past year on a project funded by the National Endowment for the Humanities to upgrade and improve Pleiades, a gazetteer of ancient places that is free and open to the public. Thus it was very gratifying to learn that Pleiades is the 2017 recipient of the Archaeological Institute of America’s Award for Outstanding Work in Digital Archaeology. Congratulations to the Pleiades team, headed by ISAW’s Tom Elliott!

Pleiades is the most comprehensive geospatial dataset for antiquity available today, giving scholars, students, and enthusiasts the ability to use, create, and share geographic information about the ancient world. It was developed between 2006 and 2008 on version 3 of the open source Plone content management system. Pleiades allows logged in users to define geographic locations and associate them with ancient places, names and scholarly references. The system remained in place from 2008 to 2016 without significant upgrades to the core Plone stack, despite the addition of a number of custom features. Over that time, over 35,000 places were added to the system – and performance degraded significantly as the content expanded.

Our most important NEH deliverable was improving site performance, which we accomplished through an upgrade from Plone 3 to Plone 4.3 and the elimination of performance bottlenecks identified with the help of New Relic monitoring. As of last September we had reduced the average page load time from 8.48 seconds before the upgrade to 2.1 seconds after. This 400% speed-up is even more impressive than it sounds because bots (search engine indexers and third ­party data users) were severely restricted during the pre-­upgrade measurements, and all restrictions were lifted after the upgrade.

Performance improvement was just the start. Here are some of the other changes we’ve made to the site, which you can read more about in the August NEH performance report.

  • Process improvements that streamline content creation, review, and publication
  • UI improvements that facilitate the use of Pleiades data
  • Improved citation and bibliography management through integration with the Zotero bibliography management system
  • Enhanced Linked Open Data capabilities
  • Comprehensive daily exports
  • Bulk imports of Pleiades content

Because of Jazkarta’s high level of expertise in Plone and related technologies, we were able to deliver the Plone upgrade and related performance improvements 6 months ahead of schedule. This left more time for feature improvements than were originally envisioned. As Tom Elliott put it, “our investment in Jazkarta is paying dividends.”

Tagged: award, bibliographies, cms, digital humanities, isaw, performance, pleiades, Plone

Mikko Ohtamaa: Simple loop parallelization in Python

by Mikko Ohtamaa at 2017-01-30T11:04:57Z

Sometimes you are programming a loop to run over tasks that could be easily parallelized. Usual suspects include loads that wait IO like calls to third party API services.

Since Python 3.2, there have been easy tool for this kind of jobs. concurrent.futures standard library module provides thread and multiprocess pools for executing tasks parallel. For older Python versions, a backport library exists.

Consider a loop that waits RPC traffic and the RPC has a wide enough pipe to handle multiple calls simultaneously:

def import_all(contract: Contract, fname: str):
    """Import all entries from a given CSV file."""
    for row in read_csv(fname):
        # This functions performs multiple RPC calls
        # with wait between calls
        import_invoicing_address(contract, row)

You can create a thread pool that runs tasks on N worker threads. Tasks are wrapped in futures that call the worker function. Each thread keeps consuming tasks from the queue until all of work is done.

import concurrent.futures
def import_all_pooled(contract: Contract, fname: str, workers=32):
    """Parallerized CSV import."""
    # Run the futures within this thread pool
    with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
        # Stream incoming data and build futures.
        # The execution of futures beings right away and the executor
        # does not wait the loop to be completed.
        futures = [executor.submit(import_invoicing_address, contract, row) for row in read_csv(fname)]
        # This print may be slightly delayed, as futures start executing as soon as the pool begins to fill,
        # eating your CPU time
        print("Executing total", len(futures), "jobs")
        # Wait the executor to complete each future, give 180 seconds for each job
        for idx, future in enumerate(concurrent.futures.as_completed(futures, timeout=180.0)):
            res = future.result()  # This will also raise any exceptions
            print("Processed job", idx, "result", res)

If the work is not CPU intensive then Python’s infamous Global Interpreter Global will not become an issue either.

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

January 25, 2017

Martijn Faassen: Seven Years: A Very Personal History of the Web

by Martijn Faassen at 2017-01-25T22:55:02Z


Humans are storytellers. As anyone who knows me can confirm, I definitely enjoy the activity of telling stories. We process and communicate by telling stories. So let me tell you all a story about my life as a web developer the last 7 years. It may just be self-reflection, but hopefully it's also useful for some others. Perhaps we can see my story as a highly personal history of the web.

I always say that what pulls me to software development most is creativity. I am creative; I can't help myself. I enjoy thinking about creativity. So this is also going to be about my creative trajectory over the last 7 years.

Why 7 years? Because in early 2010 I decided to withdraw myself from a software development community I had been involved in for about 12 years previously, and I took new paths after that. It is now early 2017; time to take stock.

Letting go of an intense involvement with a software development project, up to the point where it became part of my identity, was difficult. Perhaps that's a geeky thing to say, but so it is. I needed to process it.

Life after Zope

Zope was a web framework before that concept existed. The modern web framework only materialized sometime in the year 2005, but Zope had been ahead of its time. I was involved with the Zope community from when it was first released, in 1998. I learned a lot from Zope and its community. I made many valued connections with people that last until this day.

Zope helped shape who I am and how I think, especially as a web developer. In 2013 I wrote a retrospective that went into the history of Zope and my involvement with it.

But I did not just process it by writing blog posts. I also processed it creatively.


So I am a web developer. Back in 2010 I saw some people argue that the time of the web framework had passed. Instead, developers should just gather together a collection of building blocks and hook it up to a server using a standard API (WSGI in the case of Python). This would provide more flexibility than a framework ever could.

In a "X considered Y" style post I argued that web frameworks should be considered useful. Not that many people needed convincing, but hey, why not?

I wrote:

The burden of assembling and integrating best of breed components can be shared: that's what the developers of a framework do. And if you base your work on a pre-assembled framework, it's likely to be less work for you to upgrade, because the framework developers will have taken care that the components in the framework work together in newer versions. There is also a larger chance that people will write extensions and documentation for this same assembly, and that is very likely something you may benefit from in the future.

I follow the ReactJS community. The React community definitely is a community that lets you self-assemble a framework out of many parts. This gives that ecosystem flexibility and encourages creativity -- new approaches can be tried and adopted quickly. I like it a lot.

But figuring out how to actually start a React-based project had become a major effort. To get a good development platform, you needed to learn not only about React but also about a host of packaging and build tools: CommonJS and Webpack and npm and Babel and so on. That's quite intimidating and plain work.

So some React developers realized this and created create-react-app which makes it easy to start a working example, with minimal boilerplate code, and with suggestions on how to expand from there. It's a build framework for React that gathers good software in one place and makes it easy to use. It demonstrates how frameworks can make life easier for developers. It even goes a step further and allows you to opt out of the framework once you need more control. Now that's an interesting idea!

Client-side JS as Servant to the Server UI

So frameworks are useful. And in late 2010 I had an idea for a new one. But before I go into it, I will go on a digression on the role of client-side JavaScript on the web.

This is how almost all JavaScript development used to be done and how it's still done today in many cases: the server framework generates the HTML for the UI, and handles all the details of UI interaction in request/response cycles. But sometimes this is not be enough. More dynamic behavior on the client-side is needed. You then write a little bit of JavaScript to do it, but only when absolutely necessary.

This paradigm makes JavaScript the ugly stepsister of whatever the server server programming language is used; a minor servant of Python, Ruby, Java, PHP or whatever. The framework is on the server. JavaScript is this annoying thing you have to use; a limited, broken language. As a web developer you spend as little as time possible writing it.

In short, in this paradigm JavaScript is the servant to the server, which is in charge of the UI and does the HTML generation.

But JavaScript had been gathering strength. The notion of HTTP-based APIs had attracted wide attention through REST. The term AJAX had been coined by 2005. Browsers had become a lot more capable. To exploit this more and more JavaScript needed to be written.

jQuery was first released in 2006. jQuery provided better APIs over sucky browser APIs, and hid incompatibilities between them. Its core concept is the selector: you select things in the web page so you can implement your dynamic behavior. Selectors fit the server-dominant paradigm very well.

Client-side JS as Master of the UI

By 2010, the notion of single-page web application (SPA) was in the air. SPAs promised more powerful and responsive UIs than server-side development could accomplish. The backend is a HTTP API.

This is a paradigm shift: the server framework lets go of its control of the UI entirely, and client-side JavaScript becomes its new master. It encourages a strong separation between UI on the client and business logic on the server. This brings a big benefit: the unit of UI reuse is on the client, not spread between client and server. This makes reuse of UI components a lot easier.

By 2010, I had played with client-side template languages already. I was about to build a few large web applications, and I wanted them to be dynamic and single page. But client-side JavaScript could easily become a mess. I wanted something that would help organize client-side code better. I wanted a higher-level framework.

The idea

So there we finally get to my idea: to create a client side web framework, by bringing over concepts from server frameworks to the client, and to see what happens to them. Cool things happen! We started with templates and then moved to MVC. We created a notion of components you could compose together. We created a client-side form library based on that. In 2011 we released this as Obviel.

For a little while in 2010, early 2011, I thought I was the only one with this cool idea of a client-side web framework. It turns out that I was not: it was a good idea, and so many people had the same idea at about the same time. Even before we released Obviel, I started to hear about Backbone. Ember and Angular soon followed.

I continued working on Obviel for some time. I created a template language with a built-in i18n system for it, and a client-side router. Almost nobody seemed to care.

In 2011 and 2012 we built a lot of stuff with Obviel. In the beginning of 2013 those projects were done. Obviel didn't get any traction in the wider community. It was a project I learned a lot from, so I don't regret it. I can claim deep experience in the area of client-side development.


I went to my first JS conference in September of 2013. I had originally submitted a talk about Obviel to it, but it wasn't accepted. Everybody was promoting their shiny new client-side framework by that time.

So was Facebook. Pete Hunt gave a talk about React. This was in fact only the second time React had been introduced to a wider audience. Apparently it went over a lot better than the first time. It certainly made an impression on me: there were some fascinating new ideas in it. The React community became ferment with fascinating new ideas. At the conference I talked to people about another idea I'd had: a client framework that helps coordinate client/server communication; maybe sort of like a database, with transactions that commit UI state to the server? Nobody seemed to know of any at the time. Uh oh. If nobody has the idea at the same time, then it might be a bad one?

Then from the React community came Flux and Redux and Relay and Mobx. I let go of Obviel and started to use React. There is a little irony there: my move client-side frameworks had started with templates, but React actually let go of them.

The Server in Modern Client-side times

In early 2013 I read an interesting blog post which prompted me to write Modern Client-Side Times, in which I considered the changed role of the server web framework if it was to be the servant to JavaScript instead of its master.

I wrote a list of what tasks remain for the server framework:

What remains is still significant, however:

  • serving up JSON on URLs with hyperlinks to other URLs
  • processing POSTs of JSON content (this may include parts of form validation)
  • traversal or routing to content
  • integrating with a database (SQLAlchemy in this case)
  • authentication - who are you?
  • authorization - who can access this URL?
  • serve up an initial web page pulling in a whole bunch of static resources (JS, CSS)

I also wrote:

Much of what was useful to a server side web framework is still useful. The main thing that changes is that what goes over the wire from server to client isn't rendered HTML anymore. This is a major change that affects everything, but much does stay the same nonetheless.

I didn't know at the time of writing that I would be working on just such a server web framework very soon.

On the Morepath

In 2013 I put some smaller pieces I had been playing with for a while together and created Morepath, a server Python web framework. I gave an over-long keynote at PyCON DE that year to describe the creative processes that had gone behind it. I gave a more focused talk at EuroPython 2014 that I think works better as an introduction.

I announced Morepath on my blog:

For a while now I've been working on Morepath. I thought I'd say a bit about it here.

Morepath is a Python web micro-framework with super powers. It looks much like your average Python micro-framework, but it packs some seriously power beneath the hood.

One of the surprises of Morepath was the discovery that a web framework that tries to be good at being a REST web server actually works very well as a server web framework as well. That does make sense in retrospect: Morepath is good at letting you build REST services, therefore it needs to be good at HTTP, and any HTTP application benefits from that, no matter whether they render their UI on the client or the server. Still, it was only in early 2015 that Morepath gained official support for server templates.

2014 was full with Morepath development. I announced it at EuroPython. It slowed down a little in 2015, then picked up speed again in 2016.

I'm proud that Morepath is micro in implementation, small in its learning surface, but macro in power. The size of Morepath is another surprise: Morepath itself is currently a little over 2000 lines of Python code, but it does a lot, helped by the powerful Reg (<400 lines) and Dectate (<750 lines) libraries. Morepath offers composable, overridable, extensible applications, an extensible configuration system, an extensible view dispatch system, automated link generation, built-in powerful permission rule system, and lots more. Morepath is like Flask, but with a nuclear fusion generator inside. Seriously. Take a look.

The Future?

Over the last few years Morepath has become a true open source project; we have a small team of core contributors now. And in late 2016 Morepath started to gain a bit more attention in the wider world. I hope that continues. Users that turn into contributors are invaluable for an open source project.

There was a mention of Morepath in an Infoworld article, I was interviewed about it for Podcast__init__, and was also interviewed about it for an upcoming episode of Talk Python to Me.

Ironically I've been writing some Django code lately. I'm new to Django (sort of). I have been reintroduced to the paradigm I started to leave behind 7 years ago. With standard Django, the server rules and JavaScript is this adjunct that you use when you have to. The paradigm works, and for some projects it may be the best approach, but it's definitely not my preferred way to work anymore. But I get to help with architecture and teach a bit so I'll happily take the Django on board.

The Django management UI is cool. It makes me want to implement the equivalent for Morepath with PonyORM and React and Mobx. Or something. Want to help?

I've been itching to do something significant on the client-side again. It's been a little while since I got to do React. I enjoyed attending React Europe 2015 and React Europe 2016. I played with React Native for a bit last year. I want to work with that stuff again.

The space where client and server interacts is fertile with creative potential. That's what I've found with Obviel and React on the client, and with Morepath on the server side. While GraphQL replaces the REST paradigm that Morepath is based around (oops!), I'd enjoy working with it too.

Where might the web be going? I like to think that by being creative I sometimes get to briefly peek into its near future. I hope I can continue to be creative for the next 7 years, as I really enjoy it.

I'm a freelancer, so the clients I work for in part shape my creative future. Hint. Let me know if you have something interesting for me to work on. Fix failing parallel running browser tests with Gitlab & Robotframework


One of our project is organized in sprints where all developer work on the same code at the same time. We use one Gitlab CI server with a simple shell executer and had random failing builds with Robotframwork & Firefox.

January 16, 2017

eGenix: Python Meeting Düsseldorf - 2017-01-18


The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.


Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:

18.01.2017, 18:00 Uhr
Raum 1, 2.OG im Bürgerhaus Stadtteilzentrum Bilk
Düsseldorfer Arcaden, Bachstr. 145, 40217 Düsseldorf


Bereits angemeldete Vorträge

Charlie Clark
        "Kurze Einführung in openpyxl und Pandas"

Jochen Wersdörfer

Marc-Andre Lemburg
        "Optimierung in Python mit PuLP"

Weitere Vorträge können gerne noch angemeldet werden. Bei Interesse, bitte unter melden.

Startzeit und Ort

Wir treffen uns um 18:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden.

Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.

Über dem Eingang steht ein großes "Schwimm’ in Bilk" Logo. Hinter der Tür direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.

>>> Eingang in Google Street View


Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.

Einen guten Überblick über die Vorträge bietet unser PyDDF YouTube-Kanal, auf dem wir Videos der Vorträge nach den Meetings veröffentlichen.

Veranstaltet wird das Meeting von der GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf:


Das Python Meeting Düsseldorf nutzt eine Mischung aus (Lightning) Talks und offener Diskussion.

Vorträge können vorher angemeldet werden, oder auch spontan während des Treffens eingebracht werden. Ein Beamer mit XGA Auflösung steht zur Verfügung.

(Lightning) Talk Anmeldung bitte formlos per EMail an


Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.

Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.

Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.


Da wir nur für ca. 20 Personen Sitzplätze haben, möchten wir bitten, sich per EMail anzumelden. Damit wird keine Verpflichtung eingegangen. Es erleichtert uns allerdings die Planung.

Meeting Anmeldung bitte formlos per EMail an

Weitere Informationen

Weitere Informationen finden Sie auf der Webseite des Meetings:


Viel Spaß !

Marc-Andre Lemburg,