Planet Plone

This is where developers and integrators write about Plone, and is your best source for news and developments from the community.

October 25, 2016

Mikko Ohtamaa: Deform 2.0

by Mikko Ohtamaa at 2016-10-25T23:22:39Z

Deform 2.0 has been released.

Deform is a Python form library for generating HTML forms on the server side. Date and time picking widgets, rich text editors, forms with dynamically added and removed items and a few other complex use cases are supported out of the box.

Deform integrates with the Pyramid web framework and several other web frameworks. Deform comes with Chameleon templates and Bootstrap 3 styling. Under the hood, Colander schemas are used for serialization and validation. The Peppercorn library maps HTTP form submissions to nested structure.

Although Deform uses Chameleon templates internally, you can embed rendered Deform forms into any template language.

1. Use cases

Deform is ideal for complex server-side generated forms. Potential use cases include:

2. Installation

Install using pip and Python package installation best practices:

pip install deform

3. Example

See all widget examples. Below is a sample form loop using the Pyramid web framework.

Example code:

"""Self-contained Deform demo example."""
from __future__ import print_function
from pyramid.config import Configurator
from pyramid.session import UnencryptedCookieSessionFactoryConfig
from pyramid.httpexceptions import HTTPFound
import colander
import deform
class ExampleSchema(deform.schema.CSRFSchema):
    name = colander.SchemaNode(
    age = colander.SchemaNode(
        description="Your age in years")
def mini_example(request):
    """Sample Deform form with validation."""
    schema = ExampleSchema().bind(request=request)
    # Create a styled button with some extra Bootstrap 3 CSS classes
    process_btn = deform.form.Button(name='process', title="Process")
    form = deform.form.Form(schema, buttons=(process_btn,))
    # User submitted this form
    if request.method == "POST":
        if 'process' in request.POST:
                appstruct = form.validate(request.POST.items())
                # Save form data from appstruct
                print("Your name:", appstruct["name"])
                print("Your age:", appstruct["age"])
                # Thank user and take him/her to the next page
                request.session.flash('Thank you for the submission.')
                # Redirect to the page shows after succesful form submission
                return HTTPFound("/")
            except deform.exception.ValidationFailure as e:
                # Render a form version where errors are visible next to the fields,
                # and the submitted values are posted back
                rendered_form = e.render()
        # Render a form with initial default values
        rendered_form = form.render()
    return {
        # This is just rendered HTML in a string
        # and can be embedded in any template language
        "rendered_form": rendered_form,
def main(global_config, **settings):
    """pserve entry point"""
    session_factory = UnencryptedCookieSessionFactoryConfig('seekrit!')
    config = Configurator(settings=settings, session_factory=session_factory)
    config.add_static_view('static_deform', 'deform:static')
    config.add_route('mini_example', path='/')
    config.add_view(mini_example, route_name="mini_example", renderer="templates/")
    return config.make_wsgi_app()

This example is in deformdemo repository. Run the example with pserve:

pserve mini.ini --reload

4. Status

This library is actively developed and maintained. Deform 2.x branch has been used in production on several sites for more than two years. Automatic test suite has 100% Python code coverage and 500+ tests.

5. Projects using Deform

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

October 24, 2016

David "Pigeonflight" Bain: Characteristics of a sustainable buildout

by David Bain at 2016-10-24T13:12:00Z

Buildout is a software build system used to manage development and deployment setups, especially in Python. When done well, Buildout makes everything very repeatable. Widely used in Plone projects but not exclusive to such projects, I've used buildout with Pyramid, Tryton and even for Sphinx based documentation projects. I've also seen buildouts on github for Django, Flask and Odoo based projects. 

What I expect of a proper build system

  • Easy to switch between development and production
  • Easy management of packages and package versions
  • Easy to share with other developers
  • A workflow as close to what developers expect

Here's a breakdown of how we do our buildouts to ensure that these goals are met.

The anatomy of a sustainable buildout

There are a few things that I add to my buildout to help to keep things sane. The profiles and templates folder and a requirements.txt file. I discuss their usage below.

Profiles folder

The profiles folder holds the configuration files used by the buildout. Generally the ones that we link to directly are dev.cfg and prod.cfg. These then link to the supporting configuration profiles.

Templates folder

Templates are stored in the templates folder, these templates are used to dynamically generate configuration settings. Some of them include variables which are configured in the .cfg files found in the profiles folder.

requirements.txt to define build tools

I recently added the requirements.txt file to make the workflow closer to a more typical Python development workflow, for example most Flask developers expect such a file. Since Buildout is your build tool, we define it as a dependency in the requirements.txt so that a user can easily install it.

Getting started is a matter of running the well known "pip install -r requirements.txt" command. This installs the buildout command.

buildout.cfg is not stored by git

With this approach the buildout.cfg file is not stored in the repository, so it is added to the .gitignore file. The process of kick starting things requires that a user copies the buildout template from the templates folder into the root of their buildout.


Here's a quick example with a real project, the Python Jamaica website. I've added comments to describe each step of the process. I only assume that you are familiar with virtualenv and pip.

 # clone the site, enter the newly cloned directory
git clone
# create a buildout.cfg file (buildout expects this to be the default config file)
# create a local virtual environment then install the requirements using that environment's pip
cp templates/ buildout.cfg
virtualenv venv
venv/bin/pip install -r requirement.txt
# bootstrap the buildout
venv/bin/buildout bootstrap
# initiate a build
One gotcha to look out for, after bootstrapping a local command bin/buidout is used to run the actual build (not to be confused with venv/bin/buildout).

Thanks to Maik Derstappen for pointing me towards "buildout bootstrap" which replaces the older approach of using a special file.

Day to day usage

Once the buildout is installed here are some day to day tasks that a developer might find themselves doing.

Launching an instance

During development a standalone instance can be launched with the command:

bin/instance fg

This launches the reference site on port 8080.

Switching between development and production mode

Switching between development and production is as simple as changing a line in the buildout.cfg file.
Other available profiles are commented out.

extends =
#   profiles/prod.cfg
#   profiles/prodtest.cfg

Package management

The most common package management tasks include adding packages, removing packages, pinning the versions of packages. The buildout is configured to read the file. The big deal here is the settings associated with install_requires. Just look for that in the file and add the packages that are dependencies. (This is a standard python packaging convention learn more about install_requires).
Here's what it looks like in my buildout.

Additional Background

This is not original, while I've tweaked and hopefully improved things, I lifted the idea of a "profiles" folder from Redturtle
I've seen similar approaches implemented by others, for example Jarn calls their profiles folder "cfgs" and Starzel calls the profiles folder "linkto". I like the name "cfgs" it is most descriptive so I may adopt this naming approach in the future.

October 23, 2016

Gil Forcada: Plone on python 3 status

by gforcada at 2016-10-23T03:53:27Z

2020 is approaching fast, one day at a time, and besides being a nice catchy year, it will be also the year where Python 2.7 will no longer get any security updates.

What that means for Plone? We should hurry up and get a python 3 porting of the whole stack, yes, including Zope, ZTK, ZODB and all the tooling around (zc.buildout, etc etc).

Fortunately quite some tooling and Zope/ZTK/ZODB is already updated and there’s ongoing effort on porting the remaining parts.

The big elephant on the room blocking any porting effort of Plone to python 3 was RestrictedPython, a python distribution that, quoting itself: provides a restricted execution environment for Python.

Note the past on the previous sentence.

Since RestrictedPython is being worked, now it’s high  time for the other python distributions from Plone to be also made compatible with Python 3. Stay tuned for the Plone Conference 2016 sprint report!

Copy&pasting&adapting a set of scripts to track the progress of the porting for the Zope foundation github organization, results here, I made the same but tracking what Plone 5.1 (including our testing environment) looks like on Python 3:

The code is on collective, so feel free to update the package list.

During the Plone Conference 2016  there is quite some work put on either reducing the amount of dependencies, or updating our stack to use newer (already python 3 compatible) part of the underlying stack.

The clock is ticking and Plonistas all over the world are working hard on it!

October 22, 2016

Maurits van Rees: Ploneconf Sprint Report Saturday

by Maurits van Rees at 2016-10-22T21:33:06Z

    • Updating add-ons for Plone 4.3 and 5., FacultyStaffDirectory, contentrules mailtogroup, collective.cover.
    • Working on nicer listing of add-ons on Plone.
    • Plone 5 toolbar UI improvements, default icons if they are missing, changed manage portlets sidebar with sensible texts, edit all portlets.
    • plone.restapi. Long discussion about the framing, options listed as url, building basic Angular app, tutorial for search explaining how to setup al kinds of stuff.
    • RestrictedPython to Python 3: lot of work
    • Porting away several packages from ZopeTestCase. Looking for new server for Jenkins nodes. Removed lots of old upgrades from plone.appgrade.
    • Release team: working on signing PyPI uploads.
    • Review Plone documentation, looking how to use Sphinx in more sophisticated way, linking to the source if we mention classes or modules.
    • Documentation on Plone support channels is done.
    • Pyramid. Start using cookiecutter instead of pcreate to create a new project. Tutorials. Working on deform, colander, and demoes of that.
    • Rapido in Mosaic tiles.
    • Video and VR, Plone 5 support for, created template for 360 degrees viewer
    • Update Zope dependencies, working at fixing some breakage when using newer versions.
    • Resource handling, working on bug with legacy scripts, discussions.
    • eea.facetednavigation, fix issues during upgrade from version 9 to 10, fixed batching issue, working on final blocker for release.
    • Jasonic api for ZODB, working remotely with Jim.
    •, better visibility for selecting translations, inline svg.
    • Make icons great again with inline svg instead of fonts.
    • bobtemplates.plone theming improvements, separate the theming template from plone_addon, add a fat theme template for TTW, added fat theme buildout template, adding Rapido and other stuff, working on wrapper script.
    • Plone marketing, prepared more news items for, content changes, marketing ideas for headless CMS, Carol will be interviewing some of you.
    • Plone Cleanup, made it so that you can really get Products.CMFPlone without Archetypes, removed zope.formlib from GenericSetup which was the last one that was using it, zope.globalrequest, reviewing work.

October 21, 2016

Maurits van Rees: Lightning talks Friday

by Maurits van Rees at 2016-10-21T22:04:03Z

Paul Roeland: Plone Open Garden

Sorrento. Lovely spot in Italy. Annual event for past ten years or more: Plone Open Garden. What is it? It is a place that has Plone and Plonistas. It has a hotel as our central place, great food and drink, family friendly atmosphere for open discussions. Nice view. We want to focus on headless CMS. But not only tech, also how this affects our marketing and strategy. People with different skill sets are welcome to join.

When? Not fixed yet, but around 18 till 22 April 2017. Watch for one or more preparatory sprints over the world. Watch for discussion docs and roadmap. Please signal your attention early so we can make a great deal with the hotel.

Eric Bréhault and Philip Bauer: Plone futures

There are different possible futures for us. Valid, possible, and good. Several roads.

  1. CMS. We are targeting that market. It's what we do right now. Lots of plans like moving to Python 3.
  2. Products. Quaive (Intranet), CastleCMS. Targeting specialised audiences. Built on the same technology. Various approaches to UI.
  3. Headless CMS. Different market, like Contentful. Expose Plone as API for Javascript. Compared with comparable solutions, we are way better and we are open source. It is a different market with a lot of potential. We have a management application already in front of it: Plone 5. Unique! Plone 5 is the reference implementation of the UI.
  4. plone.server. All of the above, headless plus Python 3.

Thomas Schorr: Managing revisions in Plone

CMFEditions has been used for a long time for content revisions in Plone. Configured in the control panel. You can view old versions, or revert to them.

Several limitations and issues. History listing and statistics are calculated on the fly, which takes a very long time. If you delete a working copy, old revisions stay in the ZODB, they will not get deleted by anything through the web. Real life example, 50 GB data, out of which 34 GB was in revisions. That may not be common, they were editing several large documents daily.

We created collective.revisionmanager for this customer. Sorted listing of portal_historiesstorage. You can purge revisions or delete entire histories. It maintains a cache for statistics and history data. It has a control panel for the purge policy.

Timo Stollenwerk: Angular2 app

We made an example Angular2 blog app on top of Plone.

Fred van Dijk: From process with love

Talking about processes let's you end up with an empty room. There was a Planning and Organising Sprint in June this year in Amsterdam. Shouldn't we write down our processes? We have between ten and fifteen teams in Plone. Who knows what a PLIP is? Everyone. Who does the roadmap? Framework team? Release team? Roadmap team, is that existing?

How do people who are not here or are introverts, give feedback? How much time does volunteering cost? If we describe tasks, it makes it easier to give a task to someone else.

Release team only does releases to PyPI? What about press releases? News items, tweets, documentation? There is more process here.

Do we need a process team? Yet another team? I will start documenting some.

Hector Velarde: Brazil

Joke in Brazil: Brazil is a country of the future and it always will be. President Lula got lots of people out of poverty. Still big gap between rich and poor. Next president Dilma was impeached. Police used to be nice to protesters, but not anymore.

What has this got to do with Plone? We created a blog add-on, with payment system, to maintain freedom of speech.

David Bain: Gloss

Gloss helps with theming by adding classes. gl-menu, gl-drop, gl-frontend. Diazo makes xslt easy. Gloss is supposed to make Diazo easy.

Other David: Don't get pwnd

Use https! Get a certificate. Nag your sysadmin about it. Free at Commercial may be better for you.

Don't drop to http if the client tells you.

Only send cookies over https

Get a good score on ssllabs.

Ivan Teoh: Plomino 2.0

Plomino is a flexible and powerful application builder in the Plone UI. Version 2.0 is mainly to support Plone 5. Archetypes support has been removed. Small demo.

Annette Lewis: Empathetic designer

This is for anyone who needs to give deliverables to other person. Don't let others set your feelings. They will try to bring you down if they see it has an effect. Smile, turn up the corners of your mouth. The person in front of you is inclined to mirror this. Enjoy what you are doing, appreciate what you enjoy. Disassociate from 'toxic' workplaces or persons, you are separate.

Eric Wohnlich: ims.upload

In other solutions we were missing chunked uploads, resuming a failed upload when you retry. The jquery.upload library does support it, so we support it in ims.upload.

Not released or on github yet, I hope to do that soon.

Matthew Wilkes: Saving a start up money

Using Pyramid and Probability. I created a site for a startup that were trying to match cat owners and cat sitters. We needed to enable conversations by phone between the two groups, using Twilio. We made it so it in the end cost far less than otherwise. Lesson learned: don't guess, because we started with a solution that would have cost much more. See

Sally Kleinfeldt: CMFBibliographyAT

This was not Plone 5 ready. We had several options, but decided we may just want to store it outside of Plone. There is Plone plus Zotero. It would still be a lot of work, but if you are using it, please contact us and we'll see if we can pool resources.

Cris and Sally: Closing words

Thank you to the Microsoft NERD center for hosting us. Thanks Jazkarta and Wildcard for organising. Thanks to MIT Media Lab for provide us the Barton room for the keynote talks, especially Jen. Thanks to the training spaces: District Hall, Landmark Center, ZipCar. Thanks to Gold sponsors and SixFeetUp, and the sponsors at all the other levels. Thanks to our media partner evenios, especially Armin. Special thanks volunteers Doug Feeney and Michelle Esperanza. Thanks to T. Kim Nguyen for your time, effort, patience, generosity of spirit and just being you. Thanks to our fantastic trainers, our amazing speakers. Thanks to every last one of you who attended the conference.

Paul: "A roaring applause for you two, Cris and Sally!!!!"

Ramon and Victor: Special surprise announcement

16 to 22 October 2017: Plone Conference Barcelona. We will be at the technical university, they support us, they have 400 Plone sites. We want to involve the wider Python community of Barcelona, encouraging others to join, maybe a more general Python track.

Timo Stollenwerk: Sprint kick-off

The sprint starts tomorrow at 9 o' clock, in this room. This is a perfect time for beginners to join and start doing some work. You don't have to be a crack core developer, not at all. You are very welcome, we are very friendly and open people.

Possible sprint topics are on titanpad.

Maurits van Rees: Plone Foundation Annual General Meeting

by Maurits van Rees at 2016-10-21T20:10:32Z

Not only is the full current board here, but also our treasurer Jen Myers. Hurray!

Thank you to our membership committee, ambassadors, sprint organisers, creators of the new site, our keynote speaker at PyCon (Cris Ewing), speakers at other conferences, CMS Gardeners, Google Summer of Code students and mentors, our intellectual property and trademark watchers and relicensers, new Foundation members, sponsors both companies, universities and individuals. The general and financial reports have been approved.

The current board has these seven members:

  • Chrissy Wainwright
  • Paul Roeland
  • T. Kim Nguyen
  • Carol Ganz
  • Philip Bauer
  • Víctor Fernández de Alba
  • Alexander Loechel

And the new board:

  • all of the above.

Please step up to nominate yourself next year if you want to serve on the board. If there are more than seven, we will have an election.

The board has received a proposal for the next Plone conference, we have done due diligence and you will hear about it later today.

by Maurits van Rees at 2016-10-21T19:18:24Z

This talk is about psychoanalysis of a community, psychology of developers. What motivates 'us', the Plone community? How do we feel? We do technical stuff, but we are humans, so we have feelings.

Open source, free software: you work for free for people you don't know. That is how your father-in-law would think about it. He thinks you are either stupid or generous.

Jessie J: "It's not about the money, we just want to make the world dance."

We work because we need to eat, and belong, and have a shared ideal, participating in something.

Study: only 16 percent of employees would recommend their company's products. Disaster! Okay, this was in France, we love to be critical, even about things we don't care about.

100 percent of community members would recommend their product.

Two times:

  • Otium: free time, but not just leasure, also time for caring.
  • Negotium: the very opposite of negotium. Time for business.

In English: business and busyness. Being busy. Business is about not caring? That may be exaggerated.

Doesn't it happen to us? A client says something does not work, we don't care. Someone on the other side of the ocean says something does not work in our favorite project: we fix it even in the middle of the night.

Kant talks about minority and majority. Minor: dependent, unfree. Major: autonomous, a scholar among scholars. Developers are grown-ups. But software industry tends to reduce our responsibility. You cannot make developers responsible. Communities value their grown-up status. You do not face clients or a boss, but your co-developers. We discuss with equal people. We feel more committed in this kind of group than in a private company.

Writing software is a full process. You do everything. You are going to iterate over versions. This process induces parenthood. You spend time with the code. You want to fix it, want to feel proud about it, mainly if we show this code to someone. We want to be prepared for the future. In English, software 'runs'. In French it walks, it marches ('il marche'). My kids don't wake me up at night anymore, my 10 year old Plomino software project sometimes does.

Communities are adoption structures. They make sure a piece of software always has loving parents to raise it. It is not just a bunch of developers sharing code. It works like a family, and we welcome new members into the family.

Another concept: how do we build our identity? 'I' needs a 'We'. Gilbert Simondon says: "Becoming myself is a process, which is both individual and collective." Belonging to a group makes me feel more myself. You imitate others.

We want to be Plone. The community is more than the sum of its contributors. The fact that developers want to be a community, makes it a community.

Bernard Stiegler: transindividuation. Plone participates in our 'We' and each of our 'I' processes. We and I influence one another. We change I, and I change We.

Plone-the-thing is just the software itself. Code. Not perfect, lots of bugs. It is real.

Plone-the-concept is a moral ideal. We want it to be perfect, 100 percent tested, fully secure. It is the community's super-ego, in German an Über-Ich. The concept rises above the thing. And the concept influences the thing. The concept stays in the mind of the community, even as individual members may leave or enter.

What keeps Plone going? Love!

by Maurits van Rees at 2016-10-21T18:37:19Z

Datawire is a first-class Python shop. Python and microservices complement each other really well.

What is a microservice? Small service, self contained, narrow in scope. It is like a lego block. Simpler: it is a unit of business logic. It could send a mail, or fill out a row in a database, anything. You wire them together by combining them in a topology, like a ring, map, star, fully connected, line. Fully connected topology is what I a focusing on in this talk. Netflix five years ago had more than 500 microservices. Twitter too.

Developers are the ones who should define the topology. Business logic is distributed, rather than central. It can help when you need to scale up, integrate various systems, or have really high reliability. If one part of the topology fails, the rest should still remain functioning as much as possible.

Linear topology lends itself for scaling and optimising each part separately: ingest data (network heavy), have one source of truth for data (disk heavy), transform data (disk and cpu), present data (network and cpu).

Microservices are about service oriented development. Thinking about architecture up front can sound nice, but when this takes months without actually building anything, you are probably out of date before you get started. An architecture should think less up front, and do more experimentation, and enable developers.

You need to move away from DNS to service discovery. And from central load balancers to smart end points, to update routing tables in real time. A microservices client should be a smart endpoint that can do the routing, that knows how to discover working and available services.

A microservice is a node in an agile service topology. And it is service oriented development. Needed mindset: from architecture to experimentation.

See the company at and the microservices development kit at

Twitter: @TheBigLombowski

by Maurits van Rees at 2016-10-21T16:48:35Z

See slides at

Paul Everitt introduces the talk: The ZODB is still amazing after twenty years. Hierarchical object database including permissions, NoSQL, lots of things. On to Jim.

I am working one hundred percent on ZODB currently. Previously for Zope Corporation I could focus only part of the time on it, solving some problems we were having. Zope Corporation no longer exists. I was contracted by ZeroDB, who made this possible. ZeroDB had two products. Database that stores data encrypted at rest. Big-data analysis with hadoop. They decided to focus on their Hadoop-based product for now. I plan to offer ZODB support, consultancy, so get in contact if you need me.

Are any people here using ZODB based on NEO? No. NEO is doing some interesting things for highly durable storage. I bit more effort to setup. Poll: about half the people on the room use RelStorage, all use ZEO, a few use ZRS. I really recommend you to look at ZRS if you use ZEO. ZRS (Zope Replication Services) 1 was a nightmare, but version 2 is very good. We never made backups with repozo, we just replicated it.

ZEO version 4 used asyncore, by far the oldest async library in Python. It has lots of issues and is deprecated. I had a suspicion that maybe asyncore made ZEO slower. I rewrote most of ZEO to use asyncio instead, and cleaned the code up. In most cases there is performance improvement.

The ZODB API is synchronous. I have been using async libraries since say 1996. The API could change. Shane added a cool hack to ZServer to avoid waking up the event loop, which is a big performance win.

Transactions should be short. The longer the transaction, the higher the chance of a conflict. Connections are expensive resources, they take memory. If you have long-running work, try doing this asynchronously. But handing this off reliably is tricky.

Consider using content-aware load balancers, so you don't need all data in memory on all servers. They working set may not even fit in memory.

Might ZODB run with Javascript in a browser? Run ZODB in a web worker. Provide an async API to your UI code. This assumes that ZODB has been ported to Javascript, which should actually be doable. If someone wants to pay me for it... :-)

A challenge for some applications, is to get objects loaded fast, especially on startup. (You can often mitigate this using a ZEO client cache.) There were some problems with persistent caches, but they have been stable for a few years. But you can now prefetch items. You tell ZODB to prefetch some items, and then you can forget about the request and ZODB will meanwhile prefetch it for you, so it may be available later when you really need it. So the items are loaded asynchronously.

ZEO now has SSL. ZEO had authentication, but it made the code harder to understand. It is now out in favor of SSL. So you can restrict access to the ZODB.

ZeroDB stored the data encrypted, which meant the server could not do conflict resolution. So I added conflict resolution on the client. You can then work with real objects instead of just state. Solving conflicts in BTree splits would be easier then. It reduces processing time on the server. I would like to move conflict resolution up to the ZODB, instead of having it in ZEO.

Object-level locks. Currently ZEO locks the database for writes during the second phase of the commit process. In that phase it needs to wait for the clients to maybe do conflict resolution. Object-level locks could help here. I got it working, but it mostly did not give a performance win.

ZODB on the server is actually faster with PyPy.

ZeroDB did some interesting experiments. Split a database into multiple virtual databases, one per user, separate invalidations.

Unification of RelStorage, NEO, ZEO. NEO had some patches for ZODB and they are now merged, like a simpler implementation of multi version concurrency control. This is better for RelStorage as well. RelStorage is no longer a special case, and it has a new maintainer in Jason Maddon.

Inconsistency between ZEO clients. Scenario: add an object in one zeoclient, next request goes to second zeoclient and it potentially does not have the object yet during a very short timespan. There now is a new server-sync option to force a server round trip before each transaction. That is a cost, but maybe it should be the default.

What have I been doing after my work for ZeroDB. I worked on decent documentation, which lagged behind a long time. See You can help me improve it, by writing documentation, or also definitely by bugging me about documentation that you are missing.

FileStorage2. FileStorage worked out much better than I ever imagined. The main code has probably not changed in twenty years. It is a bit slow. With FileStorage2 we have better, separate packing, external garbage collection needed though, but that is better. Unneeded features are removed: versions and back-pointers. It uses multiple files, so with a pack you can split a file, write newly incoming transactions to the new part and pack the old part.

Byteserver is an alternative ZEO server implementation, written in the Rust language. Rust is very fast, faster than Go mostly. No Global Interpreter Lock like Python has. Byteserver includes a FileStorage2 implementation, new API between server and storage, built for speed rather than pluggability. Initial tests, from this morning, are promising, twice as fast as ZEO.

We used Zookeeper a lot, which helps keep track of which server are live and which have disappeared.

Future ZODB ideas:

  • more speed. I don't need speed to be the reason people use ZODB, but it should not be a barrier.
  • more documentation
  • OO conflict resolution
  • The ability to subscribe to object updates.
  • Integration with external indexes like Elastic Search, Solr. ZRS could be used for this: look at that stream of data and push the relevant parts to the external index.
  • Persistent pandas data frames
  • A 'jsonic' API, to be able to look at the data without having the classes. There are some zodb browsers already.
  • ZRS auto fail-over. At Zope Corp we probably only had one or two unexpected fail-overs in all those years.
  • Official Docker images would be good. But if that uses Python 3 then your client also needs to be Python 3.
  • ZEO authorization.
  • Persistent classes?
  • Other languages? Javascript, Ruby, Scala.

by Maurits van Rees at 2016-10-21T16:39:02Z

I work at Wildcard, mostly on highly secure websites. I am on the Plone Security Team.

CastleCMS is an opinionated version of Plone. It packages all kinds of things up for Plone that we have been doing for security.

It is not a fork and it won't ever be. We want to work with the Plone community and want to continue to innovate with Plone. It gives us a place to innovate. Performance is important for us, and we have integrations like CloudFlare for that. ElasticSearch indexes all your content, with an asynchronous implementation of the search api. We heavily use Redis, using it as a cache that is shared between clients or threads. ZRS (Zope Replication Services) to share the load over databases. A lot of ReactJs. z3c.unconfigure for some adapters. And sometimes just monkey patches if there is no better way.

Security is integrated 2-factor authentication. Too many login attempts lock you out. The root Zope user can only login at the root of Zope, not in Plone.

Adding content is done slightly differently, and everything is Mosaic. No display menu, no default pages. All videos are compiled to a web compatible format asynchronously when uploaded. We have a Map tile for OpenStreetmap. Focal point image tile, where you can indicate what the main focus point of an image is, which is then using during image cropping. Social media tiles. All tiles are integrated with patternslib. We have a preview for different screen sizes.

We have a plugin for ElasticSearch to let the search results order be impacted by Google Analytics and social media popularity, for anonymous users.

You can audit what users have been doing. You can login as a different user. We have additional integrations with Google Analytics, archiving of content in S3, sms support, twitter. Recycle bin. Celery integration for asynchronous tasks like video conversion or pushing large files or moving lots of documents.

What is missing? Diazo (you could use it for theming, but we don't, and you don't need it for moving html content around), portlets and viewlets (just use tiles, also possibly inheriting from parent pages), default pages and display menu (we use Mosaic).

When stuff is not open sourced yet, bug me about it.

Roadmap: chat (rocketchat, ask Sam Fords about it), Mosaic enhancement, built-in A/B-testing, continue to refine the UI, more rich tiles, on the long term use plone.server.

We use React because it is better for small parts of the page. If you create a whole app, Angular2 would be better I think. We can fight.

by Maurits van Rees at 2016-10-21T15:50:24Z

In Europe there is the CMS Garden project: combined marketing for open source CMSes. We are partners and learn from each other.

Is Plone secure? It depends. Core is pretty secure. But security of an installation is dependent upon maintenance: if you don't apply hotfixes, it is not secure.

You can look at number of hacked sites, but security is a process, not a state. You may get a zero day export today. Are you ready for it? Are there bugfix or hotfix release processes? How do you discover those?

OWASP has a top ten report on common vulnerabilities in web sites. Plone is handling them.

Study from BSI 2013: the vulnerabilities in Plone are in the core, mostly not the add-ons, which is different in other systems. So Plone actually protects the add-ons: you don't usually make a site insecure with an add-on. New BSI study this year, not yet published, raw number may seem not so good for Plone, but there was only one really important issue, they were looking at the fresh Plone 5.0, and most problems have meanwhile been fixed.

For most of the other CMSes you need a lot of add-ons to come to a comparable functionality as Plone, and that may be less secure: their add-ons have more problems. On my university I see hacks for wordpress and Typo3 sites every week, for Plone: none.

Plone has a different focus. It is good for intranets, and is not only a CMS, but a portal engine. Security is built in, with RestrictedPython, AccessControl. There is no SQL database, which means you avoid a whole category of problems. We have generators for add-ons, giving a secure base for adding features, so you don't make beginner's faults.

Plone's market share is not so large, so large botnets will mostly ignore us. That does not mean we are more secure, but it does help in practice. But we are used by several high value targets, like the FBI, which will normally get attacked first. Zope/Plone users are usually more aware of security.

Permissions and workflow are a real strength in Zope and Plone. An institute like BSI will give Plone at most a medium security level. Not high security, because admins can see all information. If you really would want this, you could actually build it with workflow.

In PHP, data and code are mixed, also for add-ons. In Plone, code is on the filesystem, and you cannot change it.

Sanitised input. Warning: don't use the structure keyword to display unfiltered user input. We do automatic csrf protection.

Plone does not enforce active bans of ip addresses, and security studies may complain about it missing out of the box, but you can simply use fail2ban in front of it. Use tools like that. And use good caching to avoid your site going down under an attack. There are ways outside of Plone, or any other CMS, that you can use.

The Joomla security team does a good job of communication, we could learn from that.

But other security teams often belong to one company. Often only bug fix releases, not security hotfixes. Bug fix releases may contain all kinds of small or large feature updates. Sometimes no security information is available, especially for add-ons, which is where most of the issues may be.

Never use a system 'as is'. Think about extra security you can apply in front of it. Spend fifteen minutes a day per system to maintain it.

If you have a strong security need, check out the Zope Replication Service to have a read-only front-end.

Audience: shameless promotion for Radio Free Asia. It is using Plone, and it is a constant target of attacks, and we have a clean record, no successful hacks.

by Maurits van Rees at 2016-10-21T15:03:08Z

It's nice to do a tech talk again, instead of always doing a keynote about the future of Plone. Sorry, I can't show you details, because I got laid off and there is a non disclosure agreement.

SQLAlchemy allows you to talk to SQL databases in a Pythonic way, getting Python objects as results.

Martijn Faassen created Traject, combining routing and traversal. Izhar Firdaus wrote collective.trajectory to do this in Plone. We use this to traverse to SQLAlchemy objects, by registering functions. We hooked the results up to as well, so they show up correctly in listings, like the standard tabular view.

A trajectory example project: It has add and edit forms which talk to the SQL database.

We had really a lot of tabular data, so it made sense to do this in SQL, not in a normal Plone portal type.

by Maurits van Rees at 2016-10-21T14:01:44Z

When I met the Plone world around 2008, I saw there were lots of small companies and individual consultants. It inspired me to quit my job and university. Then the economy collapsed. Coincidence?

Movie The Social Network, 2010. When I heard about this movie, I thought it was going to be horrible. A movie about Facebook, really? But it was really good writing. One line struck me especially: "We don't even know what it is yet." Often we will get a new technology, and in retrospect have no idea what it was yet. We invent something, but we don't even know what it is yet.

The web in Python, end of the nineties. It was not much, it did not look like clean code today. The web was young. The blink tag, anyone? These two not yet mature technologies met, and mingled.

What does Python offer? What sets it apart?

  • Reflection. Your program can look at itself, like in a mirror. Things like: list all functions of a class, ask the type, get an attribute by name. Not a lot of languages did that.
  • Object oriented. Not just object based, allowing you to split functionality over several objects. But sub classing, multiple inheritance. You get invited to write new methods for existing classes. (And it eats a whole level of indentation, but that is another story.)
  • Dynamic. You can wind up with code and objects that don't even exist in your raw source code. You can generate code on the fly, insert variables dynamically/magically. Such things might not make your code more readable though: code that does not even exist on the file system?
  • Simple. Einstein: "Things should be as simple as possible, but no simpler." Dan McKinley, book Choose Boring Technology: "Every company gets about three innovation tokens." You can do about three new things without falling over. Are you going to use MongoDB? You have just used up one of your innovation tokens. Only choose new exiting technology in strategic places. For the rest: just use MySQL, Postgres, Python. Boring, but stable.

Some Python web technologies: Django, Flask, Bottle, Pyramid, Morepath. Flask and Django are an order of magnitude more popular. I created this list, found the same list somewhere else. But some techs fitting in the middle were forgotten, like, web2py, CherryPy.

Flask and Django:

  • Views are plain functions
  • explicit registration, not reflection
  • your app can start small and simple

Today, Django stands as Python's default first framework. Flask actually has more github 'stars' for people who follow it. Yet there are far more Django conferences or meetings world wide. Django gives you a good start. Years later, you may get more opinionated, with good reasons, and choose your own ORM and a different, smaller framework. As a beginner, Django just helps you a lot.

And Django has a forms library. Wait, that may involve classes. You may need inheritance and introspection. But good programmers don't avoid complexity when it helps them. Then again, a form library is a bit like backing up a truck with multiple trailers. You have knobs to control how they are going, but can you do that without failure? And you are probably missing all kinds of knobs that the forms library has not thought about. But a forms library does automatic validation, refills forms for retry, does cross site scripting, which a new web developer probably does not even think about. So it is good for them to use it.

Flask has become the go-to second framework, for when you know what you are doing.

But will we keep writing Python for the web? We have competition. Javascript is in every browser. With Node.js you even have it on the server.

Javascript is a much cleaner story from the outside. Python has 2 versus 3, C Python versus PyPy. For Javascript on the server, everyone uses Node.js, which is the PyPy of Javascript. And for ES6 there is a cross compiler to ES5.

But, you say: Python has reflection, generators, iterators, classes, modules, and Javascript has not. Wrong. ES6 adds this. You turn your back, and suddenly a language has involved. Javascript has Python features! Vi has syntax highlighting! "Everything that rises must converge." Languages grow similar.

But what about the fact that Javascript is just dumb?

40 - '30' -> 10
40 + '30' -> '4030'

In Python you get a helpful traceback when you try this, instead of running into undebuggable errors later on. Ah, but in TypeScript you get the same. They fight back successfully against the broken type system. And you can cross compile it to ES5.

So Javascript is a contender, is gaining features, and becoming safer.

There has always been another language that was more popular than Python. There is nothing new. What are Python's advantages then?

It is becoming the world's default language. Science is moving to Python. Data is moving to Python. At weather conferences there is now a Python track. Simple syntax is perfect for the occasional programmer, who has other stuff to do, like science. You do your science, and one every three weeks you write a small program. Django Girls organizes lots of weekends around the world where women use Python to write their first website. The new programmer needs Python, because it is simple.

The web and Python met when both were immature. In the years, both learned the patterns to make the web possible and sometimes even simple to program for. And even after all this years, we don't even know what it is yet.


First language for a child? Scratch is good for young children. Python is a good language to get used to. In the US it has become the default in universities for courses.

Where at the language syntax level has Python done anything really new? When you open a file, once PyPy got popular, it got clear that files would remain open. We had to have a way to run cleanup code, even when exceptions rose. So we did try/finally, like in the Java world. It solves the problem... and it is ugly because it is Java. So we decided we needed to do one better. I believe we innovated with the with statement. It was not intuitive for me at first, but it is very helpful. It is marvellous for any kind of recursive context management, changing directories being sure that you will return to the previous directory.

In Javascript you have classes now, finally, but the Python community is far more used to it, right, which helps? Yes. Node and Javascript are still in a period of change, features have not been around for a decade. Writing in NodeJS you have spent one of your innovation tokens, writing Python you still have that one.

Find me on Twitter: @brandon_rhodes.

October 20, 2016

Maurits van Rees: Lightning talks Thursday

by Maurits van Rees at 2016-10-20T22:08:14Z

Maurits van Rees: Jens Klein - Modern LDAP User and Group Authentication

by Maurits van Rees at 2016-10-20T20:36:07Z

This is an update on pas.plugins.ldap.

LDAP is Lightweight Directory Address Protocol. This protocol for storing and querying information on users goes back to the eighties. There are various solutions, like OpenLDAP, ActiveDirectory.

There has been LDAP support in Plone for a long time. Products.LDAPUserFolder has been available as basis, replacing the standard user folder. Products.LDAPMultiPlugins wraps it to the PAS API, difficult to install. Products.PloneLDAP integrates is more into Plone, but also difficult to install. is a wrapper around this, making it easy to install in Plone. But everything is stacked on top of each other, very difficult to maintain or understand.

We created a complete rewrite: pas.plugins.ldap. We use an abstraction of the LDAP features, in``node.ext.ldap``, Python 2.7 only, using python-ldap, pure Python.

You can get groups and users from LDAP. It maps LDAP attributes to PAS property sheets. Support for caching, for many users, 98 percent test coverage.

Limitations: you cannot add users or groups, only edit them. The underlying library does implement this, so this is a TODO item, although our sites don't actually need it: users are added or deleted in a different system. No multiple (fallback) servers possible. It patches the user portraits, so this may conflict with other patches that you may have.

We have a test layer that uses an OpenLDAP instance.

node.ext.ugm is used, which is a set of interfaces for User and Group Management.

pas.plugins.ldap uses those base packages, configures it using GenericSetup, provides ZMI and control panel forms to configure LDAP.

Supporting 'many' users is tricky, this was solved using money from a fund raiser campaign. Thank you!