Planet Plone - Where Developers And Integrators Write

Plone 6 Remote Sprint - April 2020

Posted by PLONE.ORG on May 01, 2020 07:21 PM

A Successful Remote Sprint

Last week we held the first Plone remote sprint in the time of the coronavirus pandemic. It was a positive experience and a lot got done! Sprint plans and results were captured on the plone-6-sprint Etherpad and some chatting occurred on the plone/sprint Gitter channel. But the main method of communication was by video. Participants stayed connected by video all day, either on Jitsi or Discord. Each team had their own "room", and it was nice to be able to see everyone while working, even when microphones were muted. This mimicked the sort of frictionless in-person conversations that are the hallmark of sprints.

Sprint Topics

The sprinters tackled six different topics, here are the final reports on each. For more detailed information, see the plone-6-sprint Etherpad.

Modernize Plone's Default Theme

Peter Holzer and Stefan Antonelli led the charge with help from Robert Kuzma, plus Peter Mathis taking the lead on z3c.form. The team made a long list of contributions toward the following PLIPs:

Their work changes the default theme plonetheme.barceloneta to be based on Bootstrap 4 and SCSS. Plone specific styles have only been added where Bootstrap is lacking definitions for Plone elements. There is an overrides directory where all template overrides will be collected until they are merged back into the original packages.

Markup in general was improved. main_template markup was updated and the structure cleaned up, empty and unneeded elements have been removed. Header, navigation, portlets, status messages, batching - all markup and nav styles were updated. The generic view for content types was updated (this includes title, description, byline, related items and keywords). control_panel_overview, control_panel default, and some control panels with custom views were all updated.

For forms, Bootstrap markup was added to plone.app.z3cform to give widgets and fields (buttons, input fields, etc.) a basic styling. Plone's classic backend forms are also fully functional as long as Bootstrap's default CSS is independently available. In fact a theme (e.g. Barceloneta) is not necessary as long as Bootstrap CSS is available.

Icons also got some love. icon_expr for actions, content types, etc. was reused to define SVG icons. bootstrap-icons were added as an available icon resource to plone.staticresources. The team selected and added icons to the control panel settings in the controlpanel.xml in plonetheme.barceloneta.

Mosaic was changed to use Flex for columns based on the updated Barceloneta theme. All LESS files were replaced with SCSS and 'npm run watch' is now used to compile the SCSS files. Redundant CSS classes were cleaned up and the default column limit was set to 6 columns. A reset button was added to Mosaic cells to reset to the automatic column width, and an automatic reset was added for column widths if a new column gets inserted. A label was added to indicate the current column width, with "0" for auto width.

A demo of classic Plone 5.2 with the latest Barceloneta default theme is at demo.plone.org. The Coredev buildout including the current state of work on the PLIP for Bootstrap 4 markup is at classic.plone.de.

The team plans to meet regularly to continue working every Wednesday at 5 PM CEST starting on May 13. They will meet in the plone-6-sprint-barceloneta Jitsi room. Everyone is welcome to join!

Simplify Resource Registry

Maik Derstappen led this project, in coordination with the Barceloneta work. They added a custom.css view, which renders the custom CSS added in the Theming Control Panel, after all other CSS resources are loaded. This allows you to override CSS coming from a Diazo theme.

The team also discussed what really needs to change in the Resource Registry. They decided to remove the build-button from the theming editor because it will not work with SCSS (which is now used by Barceloneta).

Developers should register resources already compiled as Resource Bundles with TTW compiling disabled. They can use the merge flag to merge it with default or logged-in bundles, but this only makes sense if they are always needed. Otherwise the resource can be disabled and loaded on demand via Python in the view/viewlet or by adding the header lines directly into the templates.

Variables can be defined as CSS variables. Variables from the registry are no longer used in SCSS/CSS in plonetheme.barceloneta.

Push Plone to Zope 5

Jens Klein reports that lots of "boring but important work" was done by Maurits van Rees including cleanup of package metadata and releases of packages. Jens Vagelpohl released Zope 5.0a2 and Jens Klein created a zope-5 branch for Plone 6.0 on buildout.coredev. Version dependencies were updated on this branch, and version pins that are orphaned or handled upstream by Zope were removed. Version dependencies still outstanding:

  • z3c.form - this will be handled later
  • Update Pillow from 6 latest to 7.x - check is needed on separate branch, this applies to the Plone 5.2 branch as well
  • Parts of the Robotframework test environment - due to non backward compatible changes in Selenium, and possibly other issues
  • Unicode Encode/Decode  - this one has been open for a long time and Jens would like to see some fellow Plonista take the challenge to update or get rid of it!
  • Cleanup of any orphaned dependencies - for some packages it is difficult to know were they are used, especially for dependencies of our ecosystem packages or tools

Many failing tests were fixed. Four tests (partly related to zope.interface 5.1) are still open and in-progress. Jens would appreciate help to get them fixed and merged.

Volto

Timo Stollenwerk led the Volto efforts with Victor Fernandez de Alba, Rob Gietema, Steffen Ring, Katja Süss, Alok Kumar and Rodrigo Ferreira de Souza. They did a complete overhaul of the folder contents view according to Albert Casado's Pastanaga UI design, and made numerous other UI improvements based on a full UX review that Albert did. Alok (frontend) and Rodrigo (backend) implemented the Add-ons Control Panel and Alok implemented the Users and Groups Control Panel. Alok also implemented print CSS for the Volto front end. Victor fixed the listing block to remove the current item from query results for ZCatalog 5.1. Finally, Steffan and Katja worked on content for the upcoming VoltoCMS.com website.

Image Scaling

Asko Soukka worked on image scaling in coordination with the Volto team, with additional help from Jens Klein and Alexander Loechel. They fixed several bugs:

They implemented an option (disabled by default) to generate all configured scales immediately once an image has been updated. Enabling this would slow down the save of an image field, but would prevent most writes on read for image scaling. The code is in plone.formwidget.namedfile.

They implemented a proof of concept for PLIP #3090 for asynchronous multiprocessing of image scaling, which compliments the above option to generate all configured scales on save. It could provide image scaling with good performance for Plone without any additional services.

They implemented a proof of concept for the use of picture & srcset in Volto for perfectly responsive images. The code is in Asko's Github account.

Mastering Plone 6

Philip Bauer led a team consisting of Steve Piercy, Katja Süss and Janina Hard to create and update the developer training class for Plone 6. Much was accomplished. The first alpha version of chapters 1 through 36 are now usable and will be deployed soon for testing.

Many of the chapters now exist in two flavors: one for Volto and one for Classic, with links between them. Logos at the top clearly identify which flavor you are reading. The default navigation follows the Volto story.

The team plans to do bi-weekly half day sprints and to finish the training at the Beethoven Sprint which is scheduled for May 26th. The outstanding chapters are: 

  • Customizing Volto
  • Membrane
  • Speaker content type and view component
  • Volto Frontpage

Plone Marketing

Sally Kleinfeldt and Érico Andrei investigated tools and planned a series of "how to" screencasts for the Plone Youtube channel about managing content in Plone. Érico also worked on adding the videos from the 2003 to 2009 Plone conferences to the channel.

Sally discussed how to position headless Plone and Volto with Timo Stollenwerk, Victor Fernandez de Alba and Katja Süss. As noted above, the Volto team is working on a new voltocms.com website which will primarily be aimed at React developer on-boarding. We discussed how plone.org/plone.com can coordinate with and compliment their efforts.

After discussion with additional interested parties (Rikupekka Oksanen, Stefania Trabucchi, Fulvio Casali and William Fennie), we decided to create a "What is Plone?" section on plone.org which is aimed at decision makers. It will give a bit of Plone history and describe the current variety of frontend and backend options and when to use them. We began brainstorming ideas and will meet again in 3 weeks. Input from other community members would be welcome.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.
  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.
  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.
  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.


All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Plone and the Pandemic

Posted by PLONE.ORG on April 03, 2020 04:51 PM

The COVID-19 pandemic has upended life across the planet. Scientists, doctors and other experts are engaged in a giant global effort to combat the disease. Open source software is integral to this effort because many tools and libraries used by these experts are open source. That includes Plone, a great platform for building secure, scalable, highly customizable, content-rich websites for scientists and researchers.

One such Plone site is already being used in the COVID-19 fight. The Onkopedia portal publishes medical guidelines for oncology and hematology for Germany, Austria and Switzerland - treatment guidelines, medical studies, protocols, certifications, drug information, etc. for over 60 diseases. Patients with these diseases are at high risk for the new coronavirus, which created an urgent need for providing COVID-19 specific information. In just 3 days the Onkopedia development team was able to extend the site's data model and roll out necessary UI updates to create a new Onkopedia section with COVID-19 content for over 40 different diseases. Congratulations to the team for a great example of agile development, and to Plone for providing a flexible and robust platform.

Do you have a success story about Plone helping to fight the pandemic? Let us know at news@plone.org.

PLOG 2020 Replaced by Remote Sprint

Posted by PLONE.ORG on April 03, 2020 04:01 PM

PLOG Felled by the Pandemic

After months of planning, in early March the PLOG organizers had to face the reality that traveling to Italy in April was not going to be feasible. We tried to reschedule the event in September, but the hotel was already booked up. So we've had to make the sad decision to cancel PLOG for 2020.

PLOG will return in 2021!!!

Remote Sprinting to the Rescue

For the foreseeable future, Plone community sprints will need to happen remotely - individual sprinters in their own spaces, communicating via audio, video, and text. Happily, the big April hole that PLOG's cancellation left in the Plone calendar has now been filled by a remote sprint on Plone 6. Kudos to Peter Holzer, Maik Derstappen and Jens Klein for organizing it! The sprint will tackle a number of topics:

  • Modernize classic Plone's default theme - The goal is to make working with the classic UI fun again. We will make the templates compatible with bootstrap markup and reduce the amount of custom CSS so that it is easy to use any bootstrap template with Plone. 
  • Push Plone to the latest Zope - The Plone CMS runs on top of the Zope web framework which has recently seen some massive performance improvements. To take advantage of that we will work on making Plone 5.2 run with latest Zope 4 and Plone 6 with latest Zope 5.
  • Mastering Plone 6 - We will update the Mastering Plone training to use Plone 6 and Volto for all frontend-tasks.
  • Create marketing materials - Headless Plone and Volto are important to our future, but they aren't clearly presented on plone.org or plone.com. We will work on how-to screencasts for both Classic and Volto UIs for the Plone Youtube channel, and plone.org content about headless Plone and Volto.

Join Us!

Every experience level is welcome, and additional sprint topics are welcome too. Sign up and add your ideas!

The Plone Symposium – Germany, Dresden 2020

Posted by PLONE.ORG on March 25, 2020 03:30 PM

Each year, the Plone German-speaking Community organize a 3 days Symposium (Plone-Tagung 2020) and a two days sprint for German-speaking universities, NGO, companies as well as for users, service providers, and developers.

The symposium this year took place at the Technical University of Dresden (TUD) from 09-13. of March 2020. The Symposium has been supported by the Plone Foundation and sponsored by Werkbank (Gold Sponsor), Zopyx and FlyingCircus (Silber Sponsor) and Abstract-Technology (Bronze Sponsor).

Plone Tagung 2020

Symposium topics

The Plone Symposium in Dresden focused on the opening keynote from Anne-Marie Nebe on accessibility and why web accessibility is so important for user-centered software development. The Humanities and Social Sciences Department of the TU Dresden contributed with their expertise on the linguistic and visual implementation of a barrier-free web presence. And also the second-day keynote "Chaos und Diversity“ from Christian Theune illustrated the characteristics and dynamics of complex systems and talked about new approaches for a conscious use of diversity.

A complete overview of all topics of the Symposium is available, in German, at the Plone Tagung website.

How to make Plone cookies

Building an open-source community isn’t easy but now you can print your own Plone cookie cutter and make these goodies on your own! https://www.thingiverse.com/thing:4226416 thanks to @quirk_dispenser.

Plone Cookies

A big thank goes to the Technical University of Dresden (TUD) for hosting and the Plone-Team TUD
(@plone_tudresden) for the organization of the event and to the speakers and sponsors and to all participants, who made the event special.

And if you weren’t able to make it this year, we hope you’ll join the Plone Tagung next year! See you then.

Volto 4.0 Released

Posted by PLONE.ORG on March 01, 2020 03:30 PM

Volto 4.0 is now ready, following a 9 month alpha period that included many sprints.

A lot of effort has been put into keeping Volto's learning curve low and ensuring that developing with it is a delightful experience.

Here are some of its new features:

* Improved Pastanaga editor
* New Pastanaga editor sidebar
* New mobile first toolbar
* Developing blocks is now easier than ever
* New object browser
* Listing, table of contents, and lead image blocks
* New blocks chooser and future-proof blocks definitions
* Body classes like Plone's hinting at content types, section and current view
* New message system
* React hooks support
* Several internal libraries updated, including Redux, Router ones that support hooks
* Many bug fixes

For further details, please see the Volto 4.0 announcement forum post

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.


If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

Plone Foundation Ambassadors for 2020

Posted by PLONE.ORG on February 15, 2020 04:17 PM

The Plone Foundation Board has appointed these outstanding individuals from around the world to serve as ambassadors for Plone.

They are known for their community involvement in their regions and for their expertise in a particular market of Plone.

In their capacity as Plone ambassadors, they will continue to promote Plone regionally and in their specific markets for Plone.

Plone's ambassadors for 2020 are:

  • David Bain, Jamaica
  • Leonardo Caballero, Venezuela
  • Manabu Terada, Japan
  • Max Jakob, Germany, Education
  • William Fennie, United States, Education
  • Thomas Buchberger, Switzerland, PloneGov
  • Joël Lambillotte, Belgium, PloneGov
  • Ramiro Batista, Brazil, PloneGov


See https://plone.org/foundation/board/ambassadors for more information


"Flags of member nations flying at United Nations Headquarters". 30/Dec/2005. UN Photo/Joao Araujo Pinto.

New Waitress version, and updated 20200121 hotfix

Posted by PLONE.ORG on February 11, 2020 11:29 PM

Waitress

If you use Waitress, please upgrade from 1.4.2 to 1.4.3. 

The Pylons Project released a new version of Waitress to fix a bug in the regular expression that was used to parse HTTP headers. The bug could cause the waitress process to use excessive CPU.

As Plone 5.2.1 uses Waitress 1.4.2, we recommend changing the version pin to 1.4.3 in your buildout.

[versions]

waitress = 1.4.3

Updated 20200121 Hotfix

As announced previously, the 20200121 hotfix includes several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting.

Version 1.1, released on February 11, 2020, includes an update for the SQL Injection fix, which will not be needed for all installations.

If you are not using SQL in your website you do NOT need to upgrade, though you can if you want to. Default Plone does not need it. Upgrading to this version is especially recommended when you use PostgreSQL. Note that RelStorage is not affected. For details and discussion, see DocumentTemplate issue #48

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the update patch on 2020-02-11T22:47:25+0000.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

PLOG 2020 Training - 3 Weeks Left to Register!

Posted by PLONE.ORG on February 08, 2020 10:08 PM

Free Training

PLOG is a unique sprint where you can spend the mornings training, the afternoons sprinting, and all day enjoying the southern Italian weather and food. Thank you to the Plone Foundation, which has provided funding so that we can offer the following FREE training classes. It's a great way to brush up some old skills and pick up new ones at the halfway point between Plone conferences.

Thank you to the Plone Foundation, which is providing funding to support the training classes. 

Hands on Volto -- Taught by the RedTurtle Team -- 2 Mornings

This class will be based on the Volto training given at the last Plone Conference. It assumes some knowledge of React (but you don't have to be an expert.)

How to Get More Value From Your Plone Site Using GatsbyJS -- Taught by Asko Soukka -- 2 Mornings

This class will teach you how to deploy a static website from Plone content, and how to integrate it with the GatsbyJS ecosystem (plugins, themes, cloud services, etc.)

Plone Tips and Tricks -- Taught by Philip Bauer -- 1 Morning

This class will cover a number of advanced Plone developer topics, including:

  • Python Debugging Best Practices
  • Debugging ZODB Issues

Strategic Sprint

Because of its importance to the community, PLOG has been designated a strategic sprint by the Foundation Board and the Marketing Team. A wonderful cross section of people will attend, so there will be sprinting activities for all interests. Some will be working on Volto and RestAPI improvements, others will be creating marketing collateral in the form of "how to" screencasts of Plone features. Bring your own topic - the final plan will be made at the sprint.

Less Than 3 Weeks Left to Register!

The registration deadline is February 28th, after that we cannot guarantee the discounted rates or room availability.

Register now

Want to Learn More?

Read all about the Plone Open Garden, a tradition in our community since 2007. There's even a PLOG video. Dates, prices, and all of the details are given on the PLOG 2020 event page.

Security patch released 20200121

Posted by PLONE.ORG on January 21, 2020 03:10 PM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers: CVE-2020-7936, CVE-2020-7937, CVE-2020-7938, CVE-2020-7939, CVE-2020-7940, CVE-2020-7941.

Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.

The patch was released at 2020-01-21 15:00 UTC.

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the patch at 2020-01-21 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

20200121

Posted by PLONE.ORG on January 21, 2020 03:00 PM
Several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting. Version 1.1 released February 11, 2020, with an update for the SQL Injection fix, which will not be needed for all.

Security vulnerability pre-announcement: 20200121

Posted by PLONE.ORG on January 08, 2020 04:43 AM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers not yet issued.

Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.

The patch will be released at 2020-01-21 15:00 UTC.

Preparation

This is a pre-announcement of availability of this security fix. 

The security fix egg will be named Products.PloneHotfix20200121 and its version will be 1.0. Further installation instructions will be made available when the fix is released.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

Should you not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team will release the patch at 2020-01-21 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as Python packages that may be installed by editing a buildout configuration file and running buildout. For Plone 5.1 and lower they are also available as tarball-style archives that may be unpacked into the products folder of a buildout installation. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM
Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

PLOG 2020 Registration Now Open

Posted by PLONE.ORG on November 27, 2019 04:47 PM

Sign up now!

Registration deadline is February 28, 2020

Training classes on several topics will be held in the mornings, sprinting will happen in the afternoons and discussions will go on all day. The Hotel Mediterraneo is in a beautiful location overlooking the Bay of Naples, with wonderful food and a lovely garden where the sprinting and training will be held.

We have reserved a limited number of rooms and they are available on a first come first served basis. Breakfast and dinner are included. The rooms for three and four persons are the most economical and are expected to sell out early.

We are looking for women to share a triple!

Questions? Email us at plog@plone.org

Event details ~~ Read About PLOG ~~ Watch the Video ~~ Register

Python, the most popular programming language of the year

Posted by CodeSyntax on November 19, 2019 06:56 AM
IEEE Spectrum has published its sixth annual list with the most popular programming languages of the year across multiple platforms, and, once again Python repeats in 2019 as the undisputed leader, as it will happen in 2017 and 2018.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM
I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):

Four Members Join the Plone Foundation

Posted by PLONE.ORG on November 13, 2019 09:47 PM

The Plone Foundation welcomes four new members, after unanimous confirmation by the Foundation's Board of Directors on October 10, 2019.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Fulvio Casali

Fulvio Casali

Fulvio started working with Plone in 2008 and since 2012 he has been operating a consultancy business, Soliton Consulting, specializing almost exclusively on Plone. Fulvio organized Emerald Sprints (2013, 2014), Plone Open Garden (2016, 2017, 2019, 2020), participated in every Plone Conference since 2010 (except one!), has given two conference talks, and attended the Plone Symposium East 2010 and the Plone Konferenz in 2012. Fulvio has provided Plone training for Mastering Plone, Diazo theming, Rapido, Angular, and TTW Dexterity.

Stefania Trabucchi

Stefania Trabucchi

Stefania is the co-founder of Abstract Technology in Germany. She was the organizer of Plone Meet Up Berlin 2004-2014, of World Plone Day Berlin 2009-2015, of Plone Social Sprint Berlin 2014, and of Plone's Europython 2014 presence and associated marketing. She has participated in the Plone Konferenz 2012 as a speaker, the Plone Beethoven Sprint 2019, the Plone Conference 2014 sprint, and the Plone Open Garden sprints in 2014 and 2015. Stefania represented Plone as a member of the CMS Garden 2014-2017 and is active in the Plone Intranet Consortium with Quaive.

Thomas Buchberger

Thomas Buchberger

Thomas is the CTO of 4teamwork AG, has been working with Plone since 2006 and became a code contributor in 2011. He has attended many Plone conferences, beginning with Naples in 2007. He attended many sprints, including the Barcelona Strategic Sprint (2016), the Bonn Beethoven Sprint (2017 and 2018), the Sorrento Sprint on Frontend Modernization and Python 3 Porting (2019), and the Beethoven Sprint (2019). Thomas' main code contributions have been in the plone.rest and plone.restapi modules.

Andrea Cecchi

Andrea Cecchi

Andrea was the lead organizer of the Plone Conference 2019 in Ferrara. He participated at earlier Plone conferences and sprints (Sorrento 2019, Tokyo 2018, Barcelona 2017, Bristol 2014). As part of the RedTurtle team, he has organized and participated in World Plone Day events since its inception. Andrea is the maintainer of over 100 packages on PyPI.org.

 

 

The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.

Plone Foundation Board Officers Selected for 2019-2020

Posted by PLONE.ORG on November 08, 2019 09:22 PM

Following the election held in October 2019, the 2019-2020 Plone Foundation Board held its first meeting on November 7, 2019, and its first order of business was to select its officers:

  • President: Chrissy Wainwright
  • Vice President: Paul Roeland
  • Secretary: Andy Leeb
  • Treasurer (non-voting): Jen Myers

The Board also selected the following committee chairs:

  • Marketing: Érico Andrei (assisted by Kim Nguyen)
  • Membership Co-Chairs: Érico Andrei, Kim Nguyen

The following (unofficial) team liaisons were selected:

  • Framework: Chrissy Wainwright
  • Frontend: Victor Fernández de Alba
  • Education: to be determined
  • Security: Jens Klein
  • Guillotina: Andy Leeb

The new Board is determined to reflect the changing landscape of the Plone Foundation, where it is not just about the Plone CMS but a host of technologies and communities that reflect the same spirit, including Volto, Guillotina, Zope and others.


About the Plone Foundation

About the Plone Foundation Board

New Foundation Board for 2019-2020

Posted by PLONE.ORG on November 07, 2019 05:12 PM

The Plone Foundation Board is now in its 2019-2020 term. 

The results of the Foundation Membership vote were announced at the Annual General Meeting held in Ferrara on October 25, 2019.

The 2019-2020 Board members are:

William Fennie was not elected although he received over 30% of votes.

Our outgoing Board members chose not to run this year. We thank them for their many years of service!

  • Alexander Loechel
  • Carol Ganz
  • T. Kim Nguyen

Lightning talks Friday

Posted by Maurits van Rees on October 31, 2019 07:36 AM

Wolfgang Thomas: OIRA

Demo of tool for online risk assessment. Export to Word (with python-docx). Training module. Export to Powerpoint presentation with python-pptx.

Alex, Guido: Quaive

State of Plone Intranet in 2019. Dozens of packages. Social stream, search, auditing, real-time document collaboration. Classifieds app makes it nice to share info, legal app showing contracts. PAS graph optimizations, see talk by Alessandro. SVG rendering. Progressive web app, we did not want to create a special app, but it does really give a mobile experience.

Busy supporting Plone 5.2 and Python 3, we will manage that. We will use crossbar.io for push notifications. Encryption via CWS. Johannes is joining Syslab to work on Quaive fulltime. iMio is helping too.

Michele Finelli: vini delle sabie

The last of my three easy pieces on Ferrara.

vini delle sabie is a wine. Sand is influencing the grapes. The grapes are actually immune to some diseases. Most typical is de fortana grape. Give it a try. Perfect companion to the dishes I talked about.

Fred van Dijk: collective.ldapsetup

Example package with LDAP setup. Uses pas.plugins.ldap, which is a replacement for plone.app.ldap. The core package now supports wildcard search, with help from Asko Soukka. Robert Niederreiter did a lot of work to get it working on Python 3. LDAPS support.

Code: https://github.com/collective/collective.ldapsetup

Gauthier Bastien collective.documentgenerator

Desktop document generation (.odt, .pdf, .doc, ...) based on appy framework and OpenOffice/LibreOffice. Templating in LibreOffice.

Code: https://github.com/collective/collective.documentgenerator

Maik Derstappen: EasyNewsletter

Create news letters in Plone. Send mails to subscribers. Let email server do this, or use an external delivery service. On a collection you can define how items are rendered in the email. You can customize the output and aggregation template. Or fully write your own. Works now on Plone 5.1/5.2 Python 2.7/3.7.

Planned: integrate Mosaico editor.

Code: https://github.com/collective/Products.EasyNewsletter

Paul Grunewald: Plone Tagung

9 to 11 March 2020 in Dresden. Talks and sprints. Come!

Philip Bauer: Training Plone 6

We plan to create quickstart Plone trainings for Python devs, one for frontend devs, one for users. Mastering Plone 5.2 talks about a lot, you should follow it.

But what about mastering Plone 6. It would need to be about backend and frontend. I would love feedback about what would be required. We do not want to overwhelm people.

Maurits van Rees: 3 authentication add-ons

I present three PAS plugins:

  • collective.denyroles
  • pas.plugins.headers
  • collective.contentgroups

See the full presentation.

Treasure hunt solutions

There was a treasure hunt this week, where every day you had to find three items in the city and make pictures. We show the solutions.

Timo Stollenwerk: Cypress

We use this Acceptance testing framework to test Volto. It has gained traction in the JavaScript world.

I use robotframework and love it. But Cypress is written in JavaScript, so an intern could write extra plugin library for it.

See https://www.cypress.io

Timo Stollenwerk: Sprints

The next two days we will have Plone sprints. Everyone is welcome. It is a great chance to share code, get to know the Plone code, ask questions to coders, and generally work together. We will find someone to pair up with you.

See the list of sprint topics. You can work on other stuff, you can add ideas, also when you will not work on it. You can add your name to topics when you are interested, also multiple topics.

Sally Kleinfeldt: Essential Plone 5 add-ons

People could suggest add-ons, and then people voted for them. Clear winners:

  • collective.easyform
  • plone.restapi (in core now actually)
  • eea.facetednavigation

Others:

  • collective.documentgenerator
  • collective.z3cform.datagridfield
  • collective.taxonomy
  • and more

Plone Conference 2020

The Plone conference next year will be in... Namur, Belgium. November 16-22.

Thank you

Thank you Red Turtle for organising such a lovely conference!

Panel: Frameworks comparison

Posted by Maurits van Rees on October 25, 2019 04:04 PM

Plan:

  • Brief introduction to each framework
  • Followed by discussion focused on use cases
  • Maurits will live blog (thanks!)
  • Framework vs. Product - let’s not worry about that!

For each system:

  • What use cases is it well suited to
  • What use cases is it poorly suited to

Discussion:

  • Focus on client use cases and what frameworks would be a good or poor fit
  • You can also ask questions!

Live blogging.

Use cases:

  • Non-profit with custom forms and import/export
  • Concert venue with strict event listing and advanced settings.
  • NGO needs a database application for their highly confidential data.
  • Generate a map out of data, and store extra information, like images for the data.
  • Three-person company with basically static pages, non-technical users.
  • Real-time application with websockets.
  • A large company wants to monitor how well their customer service staff are doing. Strict form with statistics on it.

Pyramid:

  • Best for APIs, not really CMS or websites, but applications
  • Start small, grow
  • Complex non-profit use case: you would have to build everything yourself.
  • Event listing: we have a site that manages stuff for music bands as example, you just need some endpoints. Maybe websauna on top.
  • NGO privacy: security really good, and simple enough to understand the full system. You can fully build your own fine grained security system. UI is a lot of work.
  • 100 person company wants intranet with various content, installable by local IT team.
  • Intranet: try to create a good solution for multiple clients, not tailored to one in a consulting project. Pyramid is good for building such products.
  • Map: pyramid does not care which database you use. In a project we are mapping electical grids. In Guillotina they think they are nice with AsyncIO, but it makes their whole codebase harder to understand.
  • Real-time: we had websockets for a year, but it was too slow, so we rewrote it in Go-Lang.
  • Form statistics: you can create a form, store it in a database, export it, fine.

Django:

  • Aimed at 80% of use cases, simple auth system
  • Good for CRUD apps, smooth beginning
  • Non-profit: export/import is good, we can do forms. Sharing access gets tricky
  • Event listing: Django can do it, but just use Wordpress. But the technical user can do this in the admin interface.
  • NGO privacy: permissions are too simple OOTB, not per object. You may want end-to-end encryption, which no framework offers. Limited built-in audit log, nice start.
  • Intranet: not good fit for Plone for the people who do not often work with it. Django has easier CMSes for this. The expenses claim form could be easier to integrate into Django. Deployment just needs Postgres. DjangoCMS is Plone Light. Wagtail is WordPress Plus.
  • Map: any of the frameworks except Plone can do the geo stuff in postgres. Images just on the file system. Static and uploaded files are handled fine by Django, can also be in the cloud.
  • Real-time: not many Django users will care about websockets, will never be the focus.
  • Form statistics: you can but I would not do it. This feels like a custom web application. Do it in any framework.

Guillotina:

  • Small framework, scale from small to big
  • Non-profit with forms: decent fit, use JSON schema
  • Event listing: go to wix.com or something, it seems too simple
  • NGO privacy: Good fit, permissions are no problem for this. You need to build the full UI. Okay, you may be able to start with Volto as UI, but that needs integration in the backend.
  • Intranet: backend can handle it, need to build UI, deployment can be with docker and kubernetes.
  • Map: for the large files, Guillotina wins because of AsyncIO.
  • Real-time: we use websockets just fine.
  • Form statistics: you can build an app, depending on your UI skills.

Plone:

  • Good when you have different use groups with complex requirements for security.
  • Edit interface is doable, but less so on mobile.
  • Import/export complicated, but you can hire Jens.
  • Difficult to find skilled people.
  • Non-profit with forms: really good fit, end-users can create the forms. The import/export would need help from a provider.
  • Event listing: do not bother with Plone. It sounds more relational, Plone would be overkill.
  • NGO privacy: you need deep understanding of Plone security, hierarchy for storing data. Audit logging with add-on.
  • Intranet: Plone is fine. You may need to help with installing at first, but it will keep running once installed.
  • Small almost static site: none of us.
  • Real-time: once Asko is done with his ZServer improvements, we can do it.
  • Form statistics: do not use Plone. Use sed, awk, perl. Technically you could use some add-ons.

Maurits van Rees: 3 authentication add-ons

Posted by Maurits van Rees on October 25, 2019 03:28 PM

Three PAS plugins:

  • collective.denyroles
  • pas.plugins.headers
  • collective.contentgroups

collective.denyroles

Deny access to roles like Manager and Editor

  • Use case: Manager only logs in to edit-domain, not live site.
  • By default deny access to Manager, Editor, etc.
  • env DENY_ROLES=0 to disable
  • or Apache/nginx header X_DONT_CHECK_ROLES
  • Actually not a plugin, but a patch.

Code: https://github.com/collective/collective.denyroles/

pas.plugins.headers

PAS plugin for authentication based on request headers.

  • Use case: Apache/nginx adds SAML headers to requests.

  • Configuration in ZMI or profiles/default/pas.plugins.headers.json:

    {
        "userid_header": "uid",
        "required_headers": ["uid"],
        "roles_header": "roles",
        "allowed_roles": ["student", "teacher"],
        "deny_unauthorized": true,
        "redirect_url": "https://maurits.vanrees.org",
        "memberdata_to_header": [
            "fullname|HEADER_firstname HEADER_lastname"
        ]
    }
    

Code: https://github.com/collective/pas.plugins.headers/

collective.contentgroups

Plone PAS plugin for content as groups.

  • Use case: create content item that works as a group.
  • dexterity behavior
  • No Products.membrane, no Products.remember, no dexterity.membrane.
  • No separate membrane_catalog.
  • Only groups, not users.
  • No multiple inheritance, just AccessControl.users.BasicUser.

Code: https://github.com/collective/collective.contentgroups

Alessandro Pisa: PAS adventures

Posted by Maurits van Rees on October 25, 2019 01:36 PM

What is PAS? It is the Pluggable Authentication Service from Zope and Plone. It manages everything related to users and groups, like authentication, permissions, searching. It can get or set information from your site or the request or an external service like LDAP.

It uses plugins, so you can register your own. You can see a lot of those in the ZMI at the Zope root, at http://localhost:8080/acl_users/plugins/manage_active.

There are about twenty different plugin types, and they interact with each other.

In Plone we have Products.PlonePAS. It adds Plone-specific plugins and types in PAS, for example the local roles plugins. Local roles are regular parts of Zope, but not pluggable there. So this is at http://localhost:8080/Plone/acl_users/plugins/manage_active

So this is the basis to manage security, users and groups, using plugins that can be activated, deactivated, and ordered, and these plugins interact with each other.

Now let's go to the kitchen and make a plugin. We want our Plone Intranet to use an LDAP plugin. At the time, we started using Products.PloneLDAP, which builds on a few other Zope packages. (Currently pas.plugins.ldap is the more modern choice.)

We also wanted Products.membrane: this allows to create content that works as user or group. It uses a separate membrane_catalog, a sort-of copy of the portal_catalog, for better or worse.

We wanted collective.workspace to manage roles through groups instead of sharing.

And Plone Intranet itself had some extra sauce (plugins) on top. And important: caching, for performance.

It worked fine! But after a while it slowed down. We had many calls to LDAP, in the wrong way, even with the cache in place. We were using the membrane catalog too much, causing the catalog queue to flush, causing a slowdown in the standard portal_catalog search. We were also writing on SOLR, again external, which did not help for performance.

So what was the reason of the bottleneck? The big deal is when PAS does not find what it is looking for. It then tries all relevant plugins, which are intertwined. For example it finds a workspace group, and needlessly goes to LDAP to search for this group there, where it will not be found, etcetera.

So maybe we could store the LDAP information in Plone. We decided to sync LDAP in the Plone objects.

And then we also patched PAS. Finding a group results in trying to get the properties, parent groups, roles. Our patched PAS skipped plugins that looked unrelated.

There was another problem: a custom local role manager. There were users that belonged to lots of workspaces. So also to lots of groups. So role assignment was slow: you need to get all roles of all groups. We replaced Plone's local role manager with a custom one.

Another plugin is the recursive groups plugin. Standard in Plone. This can be really expensive. For each of those thousands of groups you would check if they have a parent group. We replaced this with a utility that could handle this fast, using graphs with networkx. With this, we could replace all our group plugins, especially the recursive groups plugin.

Possible problem: in the graph we had 20.000 nodes, 25.000 edges, 25 relations. But creating the graph took 50 milliseconds, so that was okay. The nodes are strings like principal_type:UID, for example collective.workspaces:Admins. We stored the information in OOBTrees: uid2path, path2name, name2uid, and an OOTreeSet called edges. And then nx.DiGraph(uid2path) to create the graph. This helped speed things up a lot.

To keep the data structure up to date, we use events.

To recap:

  • remove external connections when possible
  • know the limitations of your plugins
  • you can patch PAS to avoid some plugins
  • you can use a custom group plugin

Further ideas:

  • lazily create collective.workspace groups
  • lazily fetch group and user properties and roles

Takeaways:

  • PAS is great, but mixing and abusing plugins can be deadly
  • Sometimes replace plugins with a new one that does exactly what you need.

Rodrigo Ferreira de Souza: Data migration to Plone 5.2 and Volto

Posted by Maurits van Rees on October 25, 2019 12:26 PM

At KitConcept we have some websites that need to be deployed in Plone 5.2. We could have used collective.transmogrifier to migrate from 4.3 to 5+. And from 5.1 to 5.2 migrate the ZODB from Python 2.7 to Python 3. Transmogrifier is also an option for that part.

Why would we use transmogrifier? There are many generic pipelines available for common cases. Flexibility to deal with different use cases. And it is actually a brilliant wayto use the iterator design pattern. Every piece of the pipeline changes a small thing, and hands the information over to the next piece.

Clients:

  • Large university site
  • High-profile government client
  • Large research institution client website

Challenge: go to Python 3, Plone 5.2 and Volto. With Volto, they spare a migration from Plone 5 to 6, at least partially. Volto is a way to sell the client a Python 3 upgrade: it is the right time to use Volto.

We use Jenkins to test the migration.

General backend things:

  • Old ATTopics to core Collections. Use collective.jsonify to export it, in a way that already looks more like a Collection.
  • Rich text: migrate to Volto blocks.
  • Post migration: collective.cover needed special handling.

What we polish to enter in Volto land:

  • We use collective folderish types.
  • deal with default pages
  • Convert RichText to Volto DraftJS
  • easily point to old website when content is not imported
  • fix some urls (we want to use resolveuid)
  • simple folders we should turn info page with listing block
  • simple collections we should turn info page with collection block

Question: can you open source the module that handles the RichText to Volto blocks migration?

Answer: yes, it is actually just 38 lines with React in Node. I tried to do it in Python, but could not get it to work. And we can sprint on it.

Open Plone Board Meeting

Posted by Maurits van Rees on October 25, 2019 11:46 AM

See the annual report.

Jen Myers has graciously offered to stay on next year as treasurer.

The Zope Foundation has been added to the Plone Foundation. Still ongoing, complex process.

With the Pylons community, those talks have stalled at the moment. Might happen in the future.

Financially we lost more money than we usually do. There was a trademark conflict that cost a lot more than we thought.

Some money should come from the Zope Foundation by the way.

Results of the vote for the board.

48 valid votes, one invalid. 2 went to the spam folder, but we found them. 30 percent of votes were in paper.

We had a vote by the foundation membership. Voted into the Plone board have been: Victor, Erico, Chrissy, Andy, Paul, Jens, Fulvio.

Meeting closed by Paul.

Thanks to Alexander, Kim and Carol who are leaving the board.

Riccardo Lemmi: Deployment Automation

Posted by Maurits van Rees on October 25, 2019 10:34 AM

We wanted to find an easy way to reproduce the installation process. We use:

  • Vagrant
  • Fabric
  • AWS
  • Boto 3 / awscli

Vagrant manages virtual machines and containers, for example with VirtualBox. I use the init command to do some more configuration on a default box.

Then I use Fabric to make an ssh connection to the machine and do remote actions. It uses invoke and paramiko for this. You can let it run the same actions on different machines. You can also use it to transfer files to and from the server, or use sudo to restart Apache.

More libraries with Fabric:

  • fabtools to make sure that for example Python is installed, or a user is created.
  • Cuisine: update files. You can also use this tool to ensure packages and users, so parts are very similar to fabtools.

Next as AWS, Amazon Web Services. With this we deploy production and test machines in a simple and replicable way. You can choose to add more CPUs, bigger disks, more memory, etcetara. I use EC2 (elastic compute cloud), EBS (elastic block storage) and EIP (elastic IP) for the most. Snapshots as simple backup tool. Security group rules as a firewall.

I create a machine with Boto 3 or awscli, both available with pip install. Why would I script this? To have "infrastructure as code". When you manually try to replicate a server, you can easily forget things.

Philip Bauer: Migrations! Migrations! Migrations!

Posted by Maurits van Rees on October 25, 2019 10:02 AM

I have some upgrade code that you can use, or for some parts copy and adapt for your use case.

Code: https://github.com/collective/collective.migrationhelpers

Upgrade steps:

  • Do imports in the function.
  • Do them conditional, so it does not fail if some package does not exist.
  • Do not let a step fail when run a second time.

You may want to disable LDAP temporarily in an upgrade step.

Get the birds eye view, with some code that reports:

  • how many items are there
  • which portal types
  • how big
  • local roles
  • etcetera
  • How many items are there that need to be replaced or removed, like PloneFormGen or Collage.

Divide and conquer:

  • Deal with one problem at a time.
  • Ignore problems that don't block you. You may try to solve something in the old Plone 4.3 site which is already fixed just by completing the migration to Plone 5.

You can register upgrade steps conditionally in zcml if needed, for example with zcml:condition="installed plone-52".

Make big problems small:

  • Write something that removes 98 percent of your content, for testing. Keep the structure in place though: Folders may have portlets or local roles that give problems that you want to know of.
  • Do not migrate blobs. The PDF that lives on the filesystem will not change. Move the blobstorage out of the way, use experimental.gracefulblobmissing during migration, and move blobstorage in again.
  • Copy the Data.fs.index too, if it is a really big site.

Forget the past:

  • remove all revisions
  • maybe manually use collective.revisionmanager for this.
  • pack the database with zero days (bin/zeopack keeps the last day by default).
  • Remove no longer needed portlets.
  • You can use uninstall and upgrade profiles. I like having the upgrade code in Python, but sometimes a profile is much easier.
  • Remove utilities and adapters from add-ons that will be removed.
  • Sometimes you cannot easily remove a package. You can make an alias for it. plone.app.upgrade has code for this, for example to not crash on an old site that still expects the kupu editor package somewhere.

Migrating LinguaPlone:

  • Content editors may have done crazy things, combining folders and content from different languages.
  • We have code to migrate this to plone.app.multilingual in migration.plonehelpers.

Update to Plone 5.2:

  • Use Python 2.7 at first.
  • Include Archetypes if needed for migration.
  • Run the migration to migrate to dexterity.

Archetypes to dexterity:

  • Use the methods from pac_migration from plone.app.contenttypes.
  • Start with the containers/folders.
  • Do one type at a time.
  • Especially for blob-like items there are options to speed this up, like disabling updating SearcheableText.
  • You can write custom migrators, which might just need a few lines.

Alternative: inplace migrator from ftw.upgrade. Interesting for large folders. Might be a nice PR to get this part into core.

To Python 3:

  • We need to remove some archetypes tools.
  • Then remove the Archetypes eggs from buildout and use Python 3.
  • Run the zodbupdate script.
  • When using RelStorage, you may want to switch to filestorage temporarily.
  • Read the documentation please, especially: - upgrade to Python 3 - upgrade zodb to Python 3

Why do it this way? I don't want every Plone company to have their own migration code stashed away somewhere. Please contribute to the core.

Upgrade to Plone 6? Install plone.restapi, use Volto, done.

Asko Soukka: ZServer reloaded with HTTP/2 and WebSocket support

Posted by Maurits van Rees on October 25, 2019 08:46 AM

A year ago, ZServer was one of the last Zope parts that did not run on Python 3. The idea was to use WSGI instead. I wondered why Python 3 would not be possible. So I tried porting it. It worked. And I added websocket support. You can use this to automatically push pages from Plone to Gatsby. Or show a notification when a page needs review, via content rules.

I replaced the old Medusa server with Twisted. ZServer was optionally using Twisted years ago, but this was ripped out, and I don't know why. But I got it working again. So for HTTP and webdav we have a Twisted http server and thread pool with an async FileSender, an AsyncIO/uvloop when avilable, standard ZConfig configuration and logging, And HTTP/2 works as well.

Bonus: websockets with PubSub messaging. It uses the Autobahn Twisted WebSocket protocol. ZeroMQ PubSub cockets using IPC-socket. The connection autosubscribes to local events. The server publishes a guarded event, for example indicating that an event is restricted to Editors.

One problem at this moment with the wsgi setup using waitress, is that the requests are limited to the number of wsgi threads, so server large files can become problematic. There are probably ways around this, but ZServer does not have this problem.

Status:

  • My ZServer fork is at https://github.com/datakurre/ZServer. See mostly branch datakurre/master.
  • You will need a special branch of plone.recipe.zope2instance as well.
  • Plus collective.wsevents, plonectl, collective.taskqueue.

Upstream:

  • In Zope 4, WebDav was made dependent on ZServer, by mistake?
  • Zope 5 assumes ZServer and WebDav no longer exist.

Remaining steps:

  • Get it upstream or not?
  • Eliminate dead code.
  • QA: tests, documentation
  • Release it.
  • Restore wsgi=off on Python 3.

I would be interested to hear when others are interested in using this. It is better if more people use this.

I am an introvert from Finland, so I did not ask the Zope people, or visited the Zope sprints.

Jens: There was no real reason to drop ZServer, except that no one has tried it because the code was so very old. Now you got it working, let's get this into Zope.

See the slides.

Paolo Perrotta: A Deep Learning Adventure

Posted by Maurits van Rees on October 25, 2019 08:14 AM

It is hard to see what part of machine learning is over hyped, and what part is actually useful.

Basis of ML (Machine Learning) is to get input and transform it to output. The ML gets a picture of a duck, and it gives as answer "duck". That is image recognition. You train the ML with images that you already label. And then you give it an image, and hope it gives a good answer. Same: from English to Japanese text. From an fMRI scan to an image visualized.

Simpler example: solar panel. Input: time of day, output: how much generated power. You would train this with data. The ML would turn this into a function that gives an approximately good answer.

Simplest model: linear regression. Turn the data into a function like this:

a * X + b

No line through the training points will be perfect. You find the line that minimizes the average error. So the ML uses linear regression to find a and b.

In our example, we try to guess the amount of mojitos we sell, based on the number of passengers that a daily boat brings to our beach bar.

But this may also depend on the temperature, or the number of sharks. With two variables you would not have a line, but a plane. With n, you would get an n-dimensional shape. You have n inputs, and give a weight to each input, and add them.

This is about numbers. For image recognition, you don't get a number as output. But we can apply a function to the result, and get a number between 0 and 1 that gives a likelyhood. For an image we may have 0.02 certainty that it is a cat, and 0.9 that it is a duck, so our answer will be the highest: a duck. Translated to the mojitos: what is the likelyhood that my bar breaks even?

This system with weights and a reducer function is called a perceptron. With a short Python program I got 90 percent accuracy on the standard NDIST test. We can do better.

We look at neural networks. This is basically: mash two perceptrons together. Or more. We are finding a function that approximates the data, and reduce the error iteratively.

What would I say is deep learning? Why has it grown?

  1. Neural networks with many layers...
  2. ... trained with a lot of data...
  3. ... with specialized architectures.

We helped Facebook with neural networks by tagging our photos. We gave them training data!

A lot of engineering is going into deep learning.

Generative Adversarial Networks: GANs. Example: a horse discriminator. Train the system with images of horses and others, and it should answer with: yes or no horse.

Other part: horse generator. Randomly generate images for feeding to the horse discriminator. You train this by saying: you did or did not manage to trick the discriminator. And you try to get better. I trained this a night, and after about a million iterations, I got pictures that are not quite horses, there is something wrong, but they are amazingly close.

Lightning talks Thursday

Posted by Maurits van Rees on October 24, 2019 09:29 PM

Michele Finelli: cappellacci

My second of three easy pieces on Ferrara.

Now about cappellacci, or caplaz in the local dialect. No, it is not tortelloni or tortellini, please.

Pasta filled with pumpkin is a tradition of manu parts or Northern Italy, but easily butter and grated cheese. With vegetables? Not if you want to avoid getting arrested.

And remember: spaghetti a la bolognese does not exist.

Jens Klein: yafowil, declarative forms

Yet Another Form Widget Library. It is now a drop in replacement for z3c.form, by activating the yafowil behavior per portal type.

For example usage, we have an example package. And documentation.

Federico Campoli: Postgres carbonara

Spaggheti carbonara, the PostgreSQL way. Lots of SQL code and demo.

Code is in a gist.

Eric Brehault: PLIPs

PLIPs are PLone Improvement Proposals. There are no PLIPs at the moment. A lot is happening in Volto, but that is outside core.

If you submit, should you do the work yourself? No! You can, but it is not needed. Just give ideas to people who can develop it. Please do not hesitate.

There is a PLIP about the Dublin Core metadata behavior. This is a meta behavior for four others. This may change. If you have a concern about this, please add your thoughts to the PLIP.

Asko Soukka: robot tests

Robot framework is a way to automate tests. You can combine this with Jupyter notebooks. It can help you write better tests, including auto completion. See https://robots-from-jupyter.github.io/public/

Sven Strack and Erico Andrei: Jekyll and Hyde

How to rank Plone events. So many years, so many pictures. How do you compare pants and pools?

There can only be one answer. Food!

  • Bronze medal: Awesome Tokyo, Plone conference 2018.
  • Runner up: Somni Català, Barcelona Plone conference 2017.
  • Honorable mention: The Plone Cake, Boston Plone Conference 2016.
  • The winner is: Red Turtle Tiramisu, Ferrara Plone conference 2019.

Paul Grünewald: Digital Signage and Plone

[Note from Maurits: I expected this to go about digital signatures, but it is about showing signs on monitors, for example to inform your visitors.]

University Dresden. Content types for monitors, slide sets and slides. Contents: text, images, timetables, fullscreen video, ticker. Editing WYSIWYG inline using CKEditor, preview, scheduling

Code: tud.addons.monitor

Sven Strack: Docs analytics

Margot Bloomstein: "If we don't know if our documentation is successful, how will we know what we need to do to improve it?"

For the Plone docs we use an overwatch dashboard. Alerts when files on docs.plone.org have not been updated in a year. Open issues and PRs. Accessitility, performance, speed. Graphana, Prometeus, Matomo.

Christine Baumgartner and Ilvy: Alpine City Strategic Sprint 2020

Side note: German symposium "Plone Tagung" March next year in Dresden. See https://plonetagung.de/2020/

11 tot 14 februari in Innsbruck, Austria, come sprint with us (and enjoy beautiful and snowy Austria):

https://alpinecity.tirol

Rob Gietema: Volto form editor

I said there wasn't a form editor. But actually there is a schema editor. Saved to JSON schema. No backend implementation, but maybe we can sprint on this.

Panel: Ask Me Anything on Volto

Posted by Maurits van Rees on October 24, 2019 03:05 PM

Is Volto compatible with Guillotina? Not 100 percent. Question is how we keep the APIs in sync. If people figure out a shared generic API that works with both Plone and Guillotina as backend, this can work.

How do you migrate? You can migrate from Plone 4.3 to Volto. We did that with transmogrifier. Biggest problem is moving composite pages, like cover, to the Volto blocks. In a post migration step, you can fix things up.

Why Semantic UI? Rob researched all the popular ones, concentrating on how easy it is to theme or override stuff. Semantic UI makes this easy. You derive from a theme, and only override some parts.

For add-ons, how do you keep the backend part in sync with the frontent part? You don't. You use Python packages in the backend and node packages in the frontend. There will probably be a lot of packages that are only in the backend or only in the frontend. You may need to be careful in the frontend so that it can work with several versions of the backend of this add-on.

TypeScript has won the JavaScript wars. Will you support that? We don't want to support two ways of doing the same thing. So if we switch to TypeScript, we want to stop using ES6, also in the documentation. Same for class based versus function based.

Did you check accessibility? Yes. We have a static code checker ALM that helped us fix issues. Also the Cyprus tool. But we also do manual checks. Plone Foundation is trying to get funding for an audit.

Alok Kumar: Gatsby Preview with Plone

Posted by Maurits van Rees on October 24, 2019 01:54 PM

Gatsby Preview gives you hot reloading of content. So you change something in Plone CMS, and it immediately is visible on your Gatsby site.

Plone CMS -> Web hook -> Gatsby source plugin. When content is created, edited, deleted, we will fire a websocket event.

When you change the title of a folder, you should update the breadcrumbs of the children too, and also navigation in several places. It took me a long time, but I have implemented this.

I have created a Gatsby theme for Plone: https://github.com/collective/gatsby-theme-plone

Maik Derstappen: State of Plone back-end development today

Posted by Maurits van Rees on October 24, 2019 01:33 PM

Frontend is nice, but it needs a backend, so let's talk about that. I will talk about plonecli, plone.api, Plone snippets for VS Code and plone.restapi.

You should use plonecli. It saves a lot of time for boring stuff. It helps you to create a product and enhance it step by step. It can cleanup and create a fresh virtualenv with a specific Python version and requirements and run buildout.

It has sub templates, for example to add a content type to your package. Now a sub template for a restapi service. Also one for upgrade steps, adding an upgrade profile per upgrade step.

We create a structure and files, and if you don't use some parts you can ignore or remove them. We will check if you have a clean git status before, and ask to commit, and afterwards we automatically commit.

You have good test coverage right from the start. All features added by plonecli have at least basic test coverage. You only write the tests for your own code. We make a tox environment to test different Plone and Python versions.

You can configure plonecli/mr.bob to your taste, changing the default answers or ignoring questions in a .mrbob file.

It is extensible, you can write your own custom bobtemplate packages and register them for the plonecli.

Visual Studio Code: snippets for Plone

There are snippets for schemas, fields, registry xml.

Use plone.api. It makes add-on code much easier to understand, without arcane incantations.

Ideas for the future.

  • plonecli:
    • add option to set Interface for view, viewlet.
    • REST API sub templates for de/serializer.
    • Graphical UI (ncurses-like) to make selecting options easier.
  • VS Code: plone.api autocompletion. VS Code is not so smart with namespaces. And it does not know buildout. You can use a buildout recipe that helps here though. Or use the generated zopepy script as your Python.

I would love more contributions. Don't be shy. There is no grand jury who makes decisions. Publish your own bobtemplates. Improve the VS Code snippets, or for other editors. Meet me at the sprints.

Question: is there a bobtemplate to create a new bobtemplate?

Answer: No. Should be possible.

Andreas Jung: Migrating a large university site to Plone 5.2

Posted by Maurits van Rees on October 24, 2019 12:50 PM

https://ugent.be is a Plone 4.3 site of a large Belgian university. Started in 2002 as Zope/CMF site. 90.000 pages, sub sites, 40.000 students, hundreds of editors, 90 add-ons. They wanted to move to Plone 5.2 and Python 3.

  • Traditional in-place migration: too manu add-ons, no one-on-one mapping possible.
  • Transmogrifier: not yet Python 3 at the time, too much magic hidden in too many places with blueprints.
  • So: custom migration solution.

Content types: standard types, plus four custom content types, including PloneFormGen. So that is quite reasonable. There is extensive usage of archetypes schema extenders.

Start: analyze and investigate your dependencies. - Based on Archetypes? Obsolete, replace. - No longer needed? Remove it. In Confluence we compiled a big table with for each package the basic information of how we would handle it: upgrade it, replace it, unknown yet, status of Python support.

Start with a minimal Plone 5.2 setup. Add one verified Python 3 compatible add-on at a time. Test extensively. Focus on content types first. Things like portlets can be handled later.

You need an export. We used a customized version of collective.jsonify. Core numbers: 90.000 json files, 55 GB data, 90 minutes, binary files base64 encoded.

We exported portlet assignments, default pages, layout information, workflow state, local roles, and pre-computed values for further efficient processing.

So we had 90.000 json files on the file system. We imported this in ArangoDB. Why use such a migration database? This allowed us to import only some portal types, or do parallel imports, and test complex migration steps like for PloneFormGen.

We briefly tried MongoDB, but that could not handle data over 16 MB. The json could be dumped unchanged in ArangoDB. This took 45 minutes.

Now we need to import this into Plone. Clean Python 3.7, Plone 5.2, plus the minimal set of packages needed for the content types. Import via plone.restapi. On top we have a dedicated migration package with special views. This handled things like translating between UIDs and paths.

The "magic" migration script is based on configuration in YAML.

  • Phase 1: pre-check the migration, remove target site if it already exists from previous test, create new Plone Site, install add-ons.
  • Phase 2: create all Folders, query ArangoDB for this.
  • Phase 3: create all non folders.
  • Phase 4: global actions. Check and migrate paths to UIDs in rich text fields. Assign portlets. Other specific fixup operations, like reindexing.

We migrated PloneFormGen (Archetypes) to easyform (dexterity). Export: one JSON for the FormFolder, one JSON file per field and action adapter. In easyform this needed to be turned into an EasyForm instance and a schema.

Topics (AT) to Collections (dexterity). Code was largely taken over from the plone.app.contenttypes migration.

From AT schema extenders to dexterity behaviors. First we made a list of which there were, are they in use, what do they do? Check which dexterity replacements there are. Create new behaviors.

Migrate packages to Python 3. This is mostly covered by talks of Philipp Bauer and David Glick. Common problems: utf-8 versus unicode, import fixes, implements to implementer. I rarely used the 2to3 and modernizr tools.

Some reimplementations: - portal skins to browser views - some packages with AT replaced with new packages with dexterity

Other problems:

  • improper file and image metadata
  • migration of vocabulary values, like old to new departments
  • repetitive cycles: always a bug occurs after a day of migration right before the end.

Quality control: - you need to check that migrated content and configuration is complete - "works for me" is nice, but others need to check too

Most of the packages have been removed, the setup is much smaller.

Status: - content migration is complete - must be tested in detail - integrate with the new theme, test this - need a replacement of a specific membrane usage - need work on castle/cas plugin

Takeaways: - Export Plone to JSOJN: 2 hours. Fast. - Import JSON to ArangoDB: 45 minutes. Fast. - Import ArangoDB to plone.restapi: 36-48 hours. Painfully slow. - 1.5 - 2.0 seconds per content object on average - cannot parellellize this import, because you would get conflict errors - So Plone and ZODB and painfully slow for creating mass content.

Question: in a similar setup we did a live migration from a live site to a separate new site. Did you consider that?

Answer: I tried this for other sites, but here I wanted to be able to partial imports, independent of the live site.

Question: default migration patches away all kinds of expensive indexing. You might want to consider looking at that. Migrating ten items per second is possible, although that is inline migration. And can you share the code, especially for the easyform migration?

Answer: could be done, but is not the primary focus of the budget currently.

A slightly older version of the code is on community.plone.org.

Plone Beethoven Sprint 2019

Posted by kitconcept GmbH on September 25, 2019 09:00 AM

beethoven sprint group image

21 developers from nine different countries gathered in Bonn, Germany between June 20th and 24th to work on implementing the upcoming Plone 6. The sprint at the office of the kitconcept GmbH has been declared a “strategic sprint” by the Plone framework team. Sprint topics included working on Volto, the Plone REST API, and Guillotina.

Volto

Volto is a new ReactJS-based frontend for Plone. It was started by Rob Gietema in 2017 and it is actively developed since then. Volto will become the default frontend for Plone 6. It implements a complete new user interface called Pastanaga UI, which was developed by Albert Casado.

Victor Fernandez de Alba and Rob Gietema led the efforts to further enhance Volto together with Paul Roeland, Nicola Zambello, Piero Nicolli, Janina Hard, Jakob Kahl, Thomas Kindermann, Maurizio Delmonte, Stefania Trabucchi, Rodrigo Ferreira de Souza, Fred van Dijk, and Steffen Lindner.

Pastanaga UI Toolbar

The new editing toolbar has been part of Volto right from the start. Though, so far we did not fully implement every detail that Albert has imagined for the Pastanaga UI toolbar. Victor has been working on this for quite a while already. During the sprint, he integrated his work and polished it. Victor and Rob also worked on a new sidebar that shows up on the right side of the Volto user interface that holds additional information like page metadata and controls the settings of the individual tiles.

volto toolbar The new Volto edit toolbar

New Tiles: Collection, Slider and Table

At the Plone conference in Tokyo, we came up with a list of tiles for the new Volto composite page editor that we would like to implement. On top of that list were the listing/collection tile, the slider tile and the table tile. Piero, Rodrigo, and Rob started to work on the listing/collection tile that is supposed to become a replacement for the Collection Type in Plone.

Jakob and Janina worked on copying over the code of a slider tile that was developed by kitconcept. Rob implemented the table tile. Before you could ask Rob about the progress, he already finished his work, like always…

volto table The new Volto table tile

Accessibility

Accessibility has always been a first class citizen in the Plone community. Plone is used by many government and public websites where accessibility is a basic requirement. Paul, Thomas, Timo, and Victor worked on improving the accessibility of Volto. Our goal is to fully support the WCAG 2.1 standard with the “AA” level of conformance.

Timo set up ESlint-based static code analysis checks and Cypress-based axe accessibility tests. Paul, Thomas and Victor then started to fix the reported accessibility issues. At the end of the sprint, we were able to fix all the reported violations. Our CI build now fails if new accessibility violations are committed.

Documentation

Sven Strack, the Plone documentation team leader, joined our sprint remotely from Amsterdam to discuss how to maintain and structure the Volto documentation in the future. We all agreed that we will use Markdown as format for our docs, since this is the de-facto standard in the JavaScript community as well as in most other Open Source communities. As much as we like Restructured Text, it seems that Markdown won that battle.

We currently use MKdocs for the Volto documentation and Styleguidist to document our React components. There are lots of different tools to choose from, Docz, MKDocs, MDX, Gatsby to just name a few. We agreed that we need to do more research to make an educated decision regarding our doc toolchain.

There will be a Google Season of Docs project this year where a technical writer will help us to enhance the Volto docs further. More Volto Features Nicola added a feature to display the currently used Volto version in the Plone control panel, he added animations for page transitions and added an Italian translation for Volto. Together with Victor, he worked on a toast component for user notifications.

More Volto Features

Piero worked on improving and fixing the event type view and added styling for the listing tile.

Victor and Steffen worked on the image sidebar.

Rodrigo added a feature that allows the user to use SHIFT+RETURN to create a new line in a Volto text. By default Volto creates a new text tile on RETURN. Rodrigo also looked into the Add-ons control panel work that Eric Steele started some time ago.

Nilesh joined us remotely from India and continued his work on the users and groups control panel for Volto.

Steffen fixed a bug in the subjects search. Rodrigo and Thomas looked into our Cypress-based acceptance tests.

Fred looked into what consequences it would have for us to get rid of the description tile and how we could customize the domain in Plone to allow to send portal emails via the Volto frontend.

Stefania and Maurizio worked on a product definition of Volto and an elevator pitch for clients and users. Together with Paul, Fred and Timo, they had a longer discussion about how to position Volto and Plone 6 in the market.

Plone REST API

The Plone REST API builds the foundation for the new Volto frontend and Plone 6. It has been under active development in the last four years and seen more than 80 releases in that time period. The REST API is stable, battle-tested and production-ready and part of the Plone core since the release of Plone 5.2.

Thomas Buchberger and Lukas Graf led the REST API efforts during the sprint and worked with Carsten Senger, Roel Bruggink, Janina Hard and Timo Stollenwerk to further enhance the REST API, fix bugs, improve the documentation, and the error handling.

API Team in the graden Plone REST API discussion in the garden

Querystring Endpoint for the Collection Type

One of the most important new features, that has been developed during the sprint, is the new querystring endpoint. This endpoint will allow us to implement the Collection type for the Volto frontend. Lukas implemented this essential feature with help from Rob, who took care of the frontend implementation in Volto.

New Features, Bugfixes, and first time contributions

Thomas added support for the retrieval of additional metadata fields in summaries in the same way as in search results. Lukas made the @types endpoint expandable and the @users endpoint easier to customize. Janina fixed setting the effective date on workflow transitions. This was her first contribution to the Plone core. On the same day she signed the contributor agreement, got her fix merged and released, which is quite an accomplishment for a single day. Way to go Janina!

In addition to that, the team fixed lots of smaller bugs (#780, #729, #764, #755, #777, #732, #738) during the sprint. REST API Error handling Carsten, Nathan, Thomas, Lukas and Timo had a longer discussion about error handling in the Plone REST API and about how we can improve things. We considered and discussed implementing RFC7807 and decided against adopting it.

Developer in office Sprint discussions…

The RFC did not see much adoption and it only slightly differs from what we are already doing. A breaking change does not seem to be worth the effort. We decided to create a separate error reporting component unifying the error reporting and ensuring consistency. We also started to fix a few inconsistencies right at the sprint.

API Documentation

Guillotina is using Swagger to document its API. Nathan explained the approach he took for that and Carsten started to integrate Swagger to generate dynamic api docs in plone.restapi.

REST API Releases

Most of the work we did on Plone REST API during the sprint has been released in Plone REST API 4.1.4, 4.2.0, 4.3.0, and 4.4.0.

Dexterity Site Root

Roel Bruggink joined us on Saturday and Sunday to continue his work on turning the Plone site root into a Dexterity content object. He fixed some complex recursion and acquisition problem when migrating the site root to Dexterity. His Plone Improvement Proposal (PLIP #2454) has been accepted by the Plone framework team and it will build the foundation for a new Plone 6 branch that we plan to cut this year.

Guillotina

Guillotina is a new AsyncIO REST data API. It has been written from the scratch by Ramon Navarro Bosch and Nathan van Gheem. It can easily scale billions of objects via its REST API. With the CMS addon, it is compatible with the Plone REST API. Therefore you can run Volto on a Guillotina backend and Guillotina might be able to replace our existing Plone backend in the future.

Guillotina Developer The Guillotina master thinking hard. :)

Nathan and Ramon worked on Guillotina 5. The Guillotina 5 release brought Python 3.7 support and the use of the context vars APIs. They refactored quite a bit, moved parts of addons into core(swagger, caching, pg catalog), implemented OOTB PostgreSQL catalog, added a Helm/Kubernetes/Docker configurations for the Guillotina CMS addon that can be configured to act as a Volto backend, worked on improving caching and filestorage.

Websockets for Plone

At the Plone Open Garden 2019, Asko Soukka started to work on bringing websocket support to Plone. Websockets allow bi-directional communication between the server and the browser. With websockets, the server can send messages and notifications to the browser. This can be used to implement notification systems or collaborative editing to just name a few possible applications.

Asko continued his work during the Beethoven sprint and created an example where comments that are added to Plone automatically pop up as toast notifications for other users. He published a Docker image with a Twisted-based ZServer and implemented the foundation to use that feature for gatsby-source-plone. This will allow us to create an extension that will allow editors to live-preview changes they do in Plone in a GatsbyJS site.

Developer in Office

Karaoke, Barbecue and Garden Parties

A Plone sprint is much more than just programming. Collaborating, meeting old friends, talk, discuss and having a good time together are the ingredients that make a Plone sprint. We started our sprint on Thursday with a free React and Volto training for newbies. At the evening when everybody arrived, we went to a local Karaoke bar for a revival of the epic Karaoke sessions we had in Tokyo last year.

Karaoke Karaoke at the “Nyx”

On Friday evening we ordered Pizza and hacked the night away at the office and our garden. On Saturday we went to the “Arithmeum”, a mathematics museum owned by the Research Institute for Discrete Mathematics at the University of Bonn.

Arithmeum Our tour at the “Arithmeum”

Afterwards we had a garden party with a barbecue and we spend the afternoon sitting in the garden and the office and continued to drink and hack on Plone.

Developer in garden Barbecue in the garden of the kitconcept office

Summary

The Beethoven Sprint 2019 was a major success. We were able to start working on the main missing pieces for Volto. The new edit toolbar is in place and the foundation for the new tiles/metadata toolbar on the right side of the UI has been finished. We also implemented a new table tile.

We implemented a proof-of-concept for the new listing / collection tile and implemented the required querystring endpoint in Plone REST API. Both Volto and the Plone REST API got lots of enhancements and bugfixes.

Both the Plone REST API and Volto are stable and used in production. We will use the next month to further enhance Volto and implement the missing bits and pieces. Our plan is to present Volto 4 at the Plone conference in Ferrara at the end of this year and cut a Plone 6 branch afterwards that contains the Dexterity Site root, folderish content types and other enhancements that are needed make Volto the default frontend for Plone 6.

Guillotina and the Websockets support for ZServer together with Gatsby as another frontend for Plone are amazing projects by the smartest folks in our community. Our diversity is one of our main strength and we are looking forward what the future will bring!

We had a great time at the sprint. Thank you to everyone who attended! We thank the Plone and Python community for their ongoing support. The Plone Foundation, the Python Software Foundation, and the German Python Software Verband made this event possible with their sponsorship as well as our company sponsors Abstract, Starzel, Werkbank and Zest.

We are looking forward to see you all in Bonn next year!

Developer in garden The “Hofgarten” near the kitconcept office

Digital Signage management with Plone and GatsbyJS

Posted by Asko Soukka on September 01, 2019 09:00 AM

Sometimes complex problems have simple solutions, and this was one of those times. At work, we are piloting a new digital signage solution for our university. Unfortunately, the new system lacks in permission management, and for a while, it looked like we were unable to safely delegate digital signage content management to departments.

Lucky us that we knew a system that is very good in permission management: Plone.

Plone – An enterprise open source content management system

Plone is an open source content management system based on hierarchical object database. That’s why Plone excels in managing many kind of content and related permissions in hierarchies. That’s also why Plone has been a good fit for such hierarchical organizations where content management follows the hierarchy of the organization itself.

Plone excels in managing content and related permissions in hierarchies

So, how we are going to solve the shortcomings of our digital signage system with Plone and GatsbyJS? In brief, we

  • let our users to manage their digital signage content on Plone in the simples possible way: every display group has their own folder on Plone and the responsible users have required rights to manage content in their folders
  • we use GatsbyJS project to fetch that content from Plone and build simple digital signage content players hosted as static web pages
  • we use our on-premises GitLab CI and GitLab pages to build and host those web pages; a Plone feature called content rules is used to trigger a new updating build after each content update.

With this setup, instead of letting the content managers to directly manage the digital signage system (where we were unable to restrict their permissions to their own display groups), we configure the system to display those GatsbyJS built sites and let the users manage their digital signage content on Plone – on the content management system they are already familiar with.

Simple digital signage content loop can be populated with just dropping images into a Plone folder

And that’s just the beginning. Thanks to Plone’s customizable structured content types, we already added new content field for setting the display time of each content page. And in the future we can use this to add more configuration options to ensure that the content is displayed as intended by the content managers.

Plone supports customizing structured content fields directly form the browser

Also built-in content types like first class Image type can be enhanced with custom fields

gatsby-theme-ds-player

Show the code or it didn’t happen!

To be honest, our current solution is so simple that there is not that much to show.

Anyway, I put together a simple open source version where similar GatbyJS built web site is created from given Plone folder so that each subfolder has its own digital signage player to loop through the images dropped into that folder.

I packaged all this into a GatsbyJS “theme” plugin gatsby-theme-ds-player preconfigured to be best compatible with a Plone source using gatsby-source-plone. There is also a live demo of the result, automatically built from the theme package repository at Travis-CI:

Released is released

Posted by Andreas Jung/ZOPYX on August 02, 2019 04:52 PM
Should you be allowed to remove your own packages from PyPI or any other public package repository? Short answer: NO!!!

Medical guideline portal Onkopedia.com relaunched - on top of Plone 5.2 and Python 3.7

Posted by Andreas Jung/ZOPYX on June 24, 2019 10:07 AM
In the third major iteration of the Onkopedia project, ZOPYX and partners relaunched the complete onkopedia.com portal with a new fully responsive layout and an updated Plone backend based on the latest Plone 5.2 version and finally Python 3.7 under the hood.

Using Glitch for Pyramid Learning

Posted by Jazkarta Blog on May 09, 2019 08:02 PM

Screenshot: Pyramid Quick Tutorials showing pyramid-view-decorators, pyramid-view-predicates, pyramid-view-renderers and pyramid-asset-specifications

At this year’s Jazkarta sprint we decided to invest some time in creating Pyramid learning resources. We are fans of the Pyramid web framework, and we felt that it would be nice to work on something that would make it easier to get started with, especially for novice developers.

Enter Glitch. This is a platform that is designed for letting users explore web apps created by other Glitch users. When you see an application that does something that you want to do, you can “remix” it, which means that an exact copy, with all its code and resources, is created in your own workspace (which is some sort of Docker container). You can then change it in any way you like. This is very powerful, because you get a working application that is hosted for you, and every code change that you make is immediately reflected in the web app page.

This turns out to be really useful for teaching web application development concepts, because students can see the code for the example, read an explanation, and then right away remix the code to try out their changes, which they can see working instantly. No need for setting up a workspace, checking out code, or running a web server. Just try out an idea and see the result right there, or share it with other people. Like “view HTML source” but with code, which is one of Glitch’s goals.

We decided to create a Glitch collection that would showcase a few of Pyramid’s most distinctive features, along with a simple Hello World style tutorial. The first thing was to verify that Pyramid could be installed. Glitch’s main development target is Node.js, but its containers install Python and they allow the user to run pip, so it’s possible to install most PyPI packages including Pyramid. (For those who might be wondering: sorry, Plone is not installable, because it needs system dependencies that are not installed by Glitch.)

Glitch supports a setup file named glitch.json, which is what allowed us to install Pyramid:

{
 "install": "pip3 install --user pyramid",
 "start": "python3 pyramid_app.py"
}

In the json above, you can see that we install Pyramid using pip, and we can add any other dependencies for our project there. We can then use the “start” key to tell Glitch how to start our application.

Once we did that, the only other thing needed was to create our Pyramid app, which in this case uses code directly lifted from the trypyramid.com site:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response

def hello_world(request):
    return Response('Hello Glitch')

if __name__ == '__main__':
    with Configurator() as config:
       config.add_route('hello', '/')
       config.add_view(hello_world, route_name='hello')
       app = config.make_wsgi_app()
    server = make_server('0.0.0.0', 3000, app)
    server.serve_forever()

That’s it. Glitch automatically runs the application (the only caveat is that it needs to use port 3000). You can configure it so that any code change automatically restarts the app. After this, if a potential new user sees your project, she can just use the remix button and have a copy made in her workspace. Any change that she makes, like changing the hello text, will be immediately visible on the page.

Overall, we found Glitch to be a very intuitive and easy to use service. You can see the collection of simple tutorials that we created here:

Pyramid Quick Tutorials

 

Jazkarta Goes To LA

Posted by Jazkarta Blog on May 09, 2019 07:47 PM

The Jazkarta team at LAX-C, a city block sized Asian food distributor and restaurant supply

This year Jazkarta’s annual sprint was held in the city of Los Angeles, graciously hosted by Alec Mitchell who lives a block away from LA’s City Hall.

View of the Los Angeles City Hall from Los Angeles Street

We had fun exploring the city – Union Station, Chinatown, LAX-C (pictured above, a gigantic Asian food distributor and restaurant supply), Grand Central Market, Griffith Observatory, and the Hollywood Farmers’ Market where we bought lots of delicious fruit.

Citrus and strawberries from the Hollywood Farmers' Market

We attended an improv comedy performance at the Upright Citizens Brigade Theater, ate lots of tacos, had original Phillipe’s French Dip sandwiches, visited brew pubs and drank lots of local beer.

The Jazkarta team in front of the Highland Park Brewery.

We even had a private tasting of cheese, beer, and wine pairings.

Plate with five different cheeses and a variety of fruits and nuts

Oh, and we also sprinted on a variety of topics!

Alec and Jesse Snyder ported several of our Plone add-ons to Python 3 and Plone 5.2:

Carlos de la Guardia created a set of Pyramid Quick Tutorials on Glitch – more about that coming in another post.

Matthew Wilkes and Jesse researched the best way to set up Pyramid functional tests using pytest and webtest in an efficient manner. They were unhappy with their previous model, which relied on in-memory databases. They settled on a pattern that switched out private variables of the sessionmaker to join a longer-running transaction for each test. This is more self-contained than adapting the session setup machinery in the app itself, and works transparently on different database backends. Matthew made a gist of the relevant code.

Alec made a prototype of an app to provide faceted search of bibliography references kept in Zotero. The app has a Elasticsearch backend and a React front end. It uses the Zotero API to pull the references into Elasticsearch for querying.

Carlos investigated what it would take to move the CastleCMS quality checking features into a Plone add-on. It’s a work in progress.

Left to right: Carlos, Sally, Matthew, Alec, Jesse and Nate in front of a painting of the Griffith Observatory

It’s wonderful to be able to work together in person, instead of remotely from our home offices spread across 3 countries and 2 continents. We try to do it once a year and always have a fantastic time. This year was no exception.

Sprinting in Sorrento

Posted by Jazkarta Blog on May 08, 2019 10:44 PM

Sunset over the island of Capri, as viewed from the sprint venue

This year I helped organize the annual sprint in Sorrento called the Plone Open Garden. Plone is a highly secure and feature rich open source content management system written in Python. Plone has no single company behind it, its IP is held by a non-profit foundation. Progress on the software happens through the efforts of a self-directed community, whose members sponsor sprints to work on particular problems.

PLOG took place April 12-19 and it was a great success, attended by 18 people from 7 countries. There were two training classes:

  • Porting Plone sites and add-ons to Python 3
  • Creating a web application using the restapi and Volto, the new React front end for Plone

We also accomplished a lot of work:

  • Python 3 ports of over a dozen Plone add-ons
  • Accessibility improvements for classic Plone and Volto front ends
  • New Volto features and documentation
  • Marketing discussions
  • Documentation discussions
  • …and various other Plone improvements

I posted a news item on plone.org that describes the details of what we accomplished. But Plone sprints have a social side as well as a technical side. Friendships are renewed, meals and news are shared, and a good time is had by all. This year’s PLOG was true to form. Here are some of the things we enjoyed.

1) Breakfasts at the Hotel Mediterraneo, the sprint venue since 2007. Pictured below is the pastry table; there are also tables for cheeses, meats, fruits, breads, cereals, eggs, and more. You can squeeze your own orange juice and optionally add some Prosecco.

Breakfast buffet at the Hotel Mediterraneo - the pastry table

Dinners, also included, are a wonderful way to end a day of sprinting. First and second course, plus dessert.

A typical dessert at the Hotel Mediterraneo

2) Happy hour at the Mediterraneo. We book the rooftop for happy hour at least once during the sprint, where we enjoy the sunset over Capri (pictured at the top of this post) and the extensive cocktail menu. That’s a Black Mohito at the top and a Negroni at the bottom.

 Drinks from the Hotel Mediterraneo cocktail menu

3) Excursions. One day we walked to Sorrento’s old harbor and had lunch with a view of Mount Vesuvius.

A view looking down at Sorrento's old harbor

4) Being with old friends and meeting new ones. Always the best part.

Sprinters on the rooftop of the Hotel Mediterraneo

 

PloneGov growing in the Basque Country

Posted by CodeSyntax on March 05, 2019 12:59 PM
PloneGov is an international initiative with the goal of getting a powerful on-line eGovernment tool. Most eGovernement needs and requirements are similar and PloneGov wants to satisfy them in a effective and efficient way thanks to its open source project. CodeSyntax is part of PloneGov thanks to its UdalPlone initiative.

Four options to try Plone 5.2 on Python 3

Posted by Starzel.de on January 24, 2019 08:30 AM

Demo pages

There are nightly build of the current Plone 5.2 coredev (the development version) with Python 2 and 3:

Minimal Buildout

Here is a minimal buildout to run Plone 5.2rc1 on Python 3:

[buildout]

parts = instance

extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg



[instance]

recipe = plone.recipe.zope2instance

eggs =

Plone

Pillow

You set it up like this:

$ python3.7 -m venv .

$ ./bin/pip install -r https://dist.plone.org/release/5.2rc1-pending/requirements.txt

$ ./bin/buildout

And start it as usual with ./bin/instance fg

Standalone Development buildout

[buildout]

extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg



parts =

    instance

    zopepy

    packages

    test

    robot



eggs =

    Plone

    Pillow

    collective.easyform



test-eggs =

    collective.easyform [test]



auto-checkout =

    collective.easyform



extensions =

    mr.developer



show-picked-versions = true



[instance]

recipe = plone.recipe.zope2instance

user = admin:admin

eggs = ${buildout:eggs}

debug-mode = on

verbose-security = on



[zopepy]

recipe = zc.recipe.egg

eggs =

    ${buildout:eggs}

interpreter = zopepy

scripts =

    zopepy

    plone-compile-resources



[packages]

recipe = collective.recipe.omelette

ignore-develop = False

eggs = ${buildout:eggs}

ignores = roman



[test]

recipe = collective.xmltestreport

eggs = ${buildout:test-eggs}

defaults = ['--auto-color', '--auto-progress']



[robot]

recipe = zc.recipe.egg

eggs =

    ${buildout:test-eggs}

    Pillow

    plone.app.robotframework[reload,debug]



[sources]

collective.easyform = git git@github.com:collective/collective.easyform.git branch=python3



[versions]

Starzel buildout

The buildout that we at Starzel.de use supports Plone 5.2rc1 with Python 2 and 3.

https://github.com/starzel/buildout

It has some nice features:

  • It extends to config- and version-files on github shared by all projects that use the same version of Plone.
  • It allows to update a project simply by changing the version it extends.
  • It allows to update all projects of one version by changing remote files (very useful for HotFixes).
  • It is minimal work to setup a new project.
  • It has presets for development, testing, staging and production.
  • It has all the nice development-helpers we use.

Quickstart:

$ git clone -b 5.2rc1.x https://github.com/starzel/buildout <SOME_PROJECT>

$ cd <MY_PROJECT>

Remove all files that are not needed for a project but are only used for the buildout itself.

$ rm -rf linkto README.rst README.txt .travis.yml secret.cfg_tmpl VERSION.txt local_coredev.cfg CHANGES.rst

If you're not developing the buildout itself you want a create a new git repo.

$ rm -rf .git && git init

Add a file that contains a passwort. Do not use admin as a password in production!

$ echo -e "[buildout]\nlogin = admin\npassword = admin" > secret.cfg

Symlink to the file that best fits you local environment. At first that is usually development. Later you can use production or test. This buildout only uses local.cfg and ignores all local_*.cfg.

$ ln -s local_develop.cfg local.cfg

Create a virtualenv in Python 2.7 or Python 3.7 (Plone 5.2 only).

$ virtualenv .  # for Python 2.7

$ python3.7 -m venv .  # for Python 3 (Plone 5.2 only)

Install and configure Plone

$ ./bin/pip install -r requirements.txt

$ ./bin/buildout

Wrapup

I hope these options help you get started with Python 3. For serious projects you will likely create you own buildout.

Update (4.2.2019)

You should use the Plone coredev for developing Plone itself:

$ git clone git@github.com:plone/buildout.coredev.git coredev
$ cd coredev
$ git checkout 5.2
$ python3.7 -m venv .
$ ./bin/pip install -r requirements.txt
$ ./bin/buildout
$ ./bin/instance fg

Update 2 (24.2.2019)

Since Plone 5.2b2 wsgi is the default, even when running Python 2. See https://github.com/plone/Products.CMFPlone/issues/2763

This means you do not need to enable wsgi = on since it will be enabled automatically. You can still choose to switch wsgi = off in Python 2 if you have a good reason to do so (e.g. you need FTP or WebDAV). I updated the examples accordingly.

Update 3 (8.3.2019)

Release candidate 1 of Plone 5.2 is pending. See https://community.plone.org/t/plone-5-2rc1-soft-released/8163. I updated the examples accordingly.

Plone finally supports Python 3!

Posted by Starzel.de on November 29, 2018 09:21 AM

At the Plone Conference 2018 in Tokyo I gave a plenary talk in which I told the story how Plone got to support Python 3. Ten minutes before my talk Plone 5.2a1 was released. That release was the culmination of a 3-year long journey (or even 5 years if you start in Brazil). If you want to learn how we got there you should watch the talk.

Where are we now?

Plone 5.2 supports Python 3.7, 3.6 and 2.7. It is currently in alpha but the plan is to release a final version in February 2019 right after the Alpine City Sprint. You should seriously consider joining that sprint if you plan to migrate your own projects. Working together with the people who ported Plone to Python 3 will give you the know-how to succeed in your own migrations!

But in fact you should not wait until February to start working with Plone on Python 3 but start right now so you'll have enough time to migrate to Python 3 before January 1st 2020 (End of Life of Python 2).

To make testing and working with Python 3 easier some tools and addons were already working on Python 3.

The following essential development-tools are already ported:

Also the three addons that achieved the highest ranking during the latest poll at Ploneconf now work on Python 3:

You can also run the new frontend Volto with Plone on Python 3.

Open tasks

Before the final release of Plone 5.2 scheduled for February 2019 there are still a lot of things to do.

Here are the six most pressing issues:

  1. The upgrade- and porting-guide need to be completed.
  2. ZODB migration needs to be finished and documented.
  3. Much more addons need to be ported and released.
  4. We require performance tests.
  5. People who really need a replacement for FTP and/or WebDAV need to figure out how to do that.

There is still a lot to do. You can help Plone by coming to a sprint, testing it and fixing or reporting a bug.

Migrating to Python 3

From 1.1.2020 on you'll have to inform your clients that you are running their super-secure system on an unsupported version of Python. That might be ok if you're running a small website for a friend but for serious projects this simply will not do. Instead you should start to adopt Python 3 in your Plone projects now. You need to start early and plan ahead!

Here are the six steps you have to follow to upgrade to Python 3. Do not attempt to do all steps at once, instead you should work iteratively and even deploy to production in between steps.

1. Upgrade to Plone 5.2

You should still use Python 2.7 for this step. The changes between Plone 5.1 and Plone 5.2 are not huge but you will gain some great features including a new and much faster dropdown-navigation and a bootstrap-based ZMI.

The Upgrade-Guide for 5.2 still needs much more love. So if you encounter any issues please update that document and/or create a ticket in github.

2. Drop Archetypes and only use Dexterity

Archetypes is still supported on Plone 5.2 but only when running with Python 2.7. It is now officially deprecated and is not ported to Python 3. There has been a built-in migration for default types from Archetypes to Dexterity since 2013, and since 2015 there have been helpers and even a form that allows you to migrate your custom content.

3. Migrate your code to Python 3

Follow the documentation in to migrate your custom code and any addons that you need in your packages.

  • Make sure your code works in Python 2 and Python 3. If you are certain that you will not publish that code as a addon or reuse it in another project you can also drop support for Python 2. That will make porting and testing easier.
  • For most addons: Use python-modernize as described in the documentation.
  • For small addons you could even simply try to startup on Python 3 and and fix whatever fails (e.g. relative imports or invalid syntax)
  • Manually fix whatever python-modernize misses (e.g. relative imports)
  • Do the same with tests. You'll need a test-setup with tox and a test-matrix in travis that tests different Python versions. The setup in collective.ifttt is almost done and might be a good example.
  • If you have complex doctests consider migrating them to python tests since these are easier to get to pass in both Python 2 and 3. If you do want to keep them as doctests, change them so that the output for Python 3 is the default and use a Py23DocChecker to make the tests pass in Python 2 as well. See the example in plone.indexer.
  • Once all seems to work you need to test every single feature manually because even a high test coverage is never a guarantee that your code actually does what it should.

4. Migrate Addons and Dependencies

Now migrate any addons you use to Python 3 as explained above. The main difference is that now everyone who uses this addon benefits. You shoud not drop support for Python 2 for those!

I listed the addons that already run on Python 3 above. I'm sure that list will be outdated very soon. Before you start migrating an addon you need to check if there already is a branch that supports Python 3 and check if there is a ticket like the one for collective.easyform. If not create such a ticket and start the process.

5. Migrate your Database

ZODB itself is compatible with Python 3 but a database that was created in Python 2.7 cannot be used in Python 3 without being modified first. I will not go into details about that. You should read the documentation and see David Glick's talk.

The migration will probably require some downtime and in rare cases you might have to write your own mappings to tell zodbupdate how to handle your data.

6. Deploy on Python 3

Plone 5.2 will use WSGI with waitress as the default WSGI server. Alternatively you can use uwsgi or gunicorn. ZServer can only be used with Python 2.7.

Some deployment tools may not ported yet, others like ZRS or RelStorage already support Python 3. You should test your production setup early. There may be changes that you don't want to struggle with a week before a scheduled launch.

Summing up

Start that whole process as early as possible because you may need to deal with unexpected problems and addons that you did not know still used Archetypes. 2019 might be much busier for you than you expect.

Do not wait for a final release of Plone 5.2! Start the process of planning and testing a migration. Consinder hiring experts to help you if necessary.

Again: You can and you should work iteratively. That means you do not have to do everything in one step and one night:

  • Upgrade to 5.2 with Dexterity on Python 2 and deploy that early in 2019.
  • Then take some time to migrate and test your code and addons.
  • You can then deploy the Python 3-compatible code while still running Python 2.7.
  • Then you can migrate your existing database and deploy on Python 3.

FAQ

Here are some questions you might have:

What about Python 4?

That will not be a big issue since 4.0 will be a upgrade like from 3.7 to 3.8. The Python community simply don't want to release a Python 3.10.

When will Plone drop support for Python 2?

Not before we have a good reason. Maybe Plone 7? It would allow us to remove and clean up some code but we would not gain a lot.

What about Zope 5?

There are no real plans for that yet.

Will Archetypes be migrated to Python 3?

Please don't try to postpone the inevitable. We are too small a community to maintain two Python versions, two frontends and two content type frameworks.

How do we replace FTP/WebDAV?

I don't know, for WebDAV you might be able to use a middleware like WsgiDAV.

Why did you make all that effort and not 'simply' migrate to Guillotina?

Guillotina does not aim to be a replacement for Plone. Also it has no equivalent addon ecosystem, feature equality or migration path.

Can I now use async/await?

You can but it won't really help you since Plone and Zope are still fundamentally synchronous.

Why doesn't the Plone Foundation simply buy support for Python 2.7 from RedHat/IBM for all of us?

Are you kidding me?

Does Python 3 make Plone run faster?

Maybe. We still lack performance tests but my guess is that Plone 5.2 will run faster on Python 3 than on 2.

Can I run the same Database in Python 2 and 3?

No.You can neither use a DB create with py2 in py3 nor use one created with py3 in py2.

Is addon xyz already ported?

Most likely not.

Plone supports the latest two major version. Does that extend to Python?

Plone's security update policy supports the latest two major versions (currently, that means the 5.x series and the 4.3.x series) with security updates. That guarantee does not extend to the programming language, the operating systen or the hardware you are running Plone on.

Will this work on Windows?

Very likely. Someone will have to test it.

I need help with migrating our projects to Python 3. Who can help me?

Talk to us at Starzel.de or ask other professionals with experience.

Warm words

Migrating Plone to Python 3 was quite a ride and a great community effort. I had to cope with some brain-melting changes in Zope, fight heartbreaking test-isolation issues and toil through an endless list of packages and tests. But I had the immense privilege and pleasure to do all of that together with some of the smartest and nicest people I know. Thanks to all of you!

Watch the full talk if you want to learn more about how Plone finally landed on Python 3.

Here are all videos from Ploneconf 2018: Day 1, Day 2, Day 3.

Python 3, Plone 5.2, Zope 4 and more

Posted by Starzel.de on October 19, 2018 09:14 AM

With over 30 attendees the SaltLabs Sprint was among the bigger sprints in the history of Plone and Zope.

Fortunately, our host Gocept just moved into a new office and had plenty of space to spare. Stylish as we are, the Plone crowd quickly congregated in the palace-like salon with red walls and renaissance paintings. Even better: Throughout the week the cafe Kaffej downstairs was open for us to eat breakfast, lunch and for work. We tried to keep the barista busy producing excellent Cappuccino. Thanks!

Zope 4

The sprint split in two groups, one working on Zope and one working on Plone. A short report on about what happened at the Zope can be found at https://blog.gocept.com/2018/10/11/beta-permission-for-earl-zope-extended . Zope 4.0b6 was released just after the sprint. The new Beta 6 includes the new bootstrap-styled ZMI, support for Python 3.7 and a metric ton of bug fixes compared to 4.0b5 which was released in May this year. By the way: Zope can now be installed without buildout using pipenv

Python 3

The Plone crowd was huge and worked on several tasks. The biggest task - and the main reason to join the sprint - was continuing to port Plone to Python 3. Jens Klein, Peter Mathis, Alessandro Pisa, David Glick (remote) and Philip Bauer as lead worked mainly on fixing all remaining failing tests and getting the jenkins-builds to pass. We dove deep into crazy test-isolation-issues but also added a couple of missing features to Dexterity that were so far only tested with Archetypes.

During the sprint we got very close to a green build but a couple of really nasty test-isolation issues prevented a green build. A week after the sprint we finally got that done and all tests of Plone now pass in Python 2.7, Python 3.6 and Python 3.7 with the same code-base!

With the test passing we're now merging that work and preparing the release of Plone 5.2a1. The tasks for this are discussed in this ticket

Deprecate Archetypes

Archetypes will not be ported to Python 3 but still has to work when running Plone 5.2 in Python 2.7. Since all tests that used PloneTestCase were still using Archetypes Philip changed that to use Dexterity now and created a new testlayer (plone.app.testing.bbb_at.PloneTestCase) that is now used by all the packages using the Archetypes-stack. See https://github.com/plone/plone.app.testing/pull/51. You could use this layer in your archetypes-based addon-packages that you want to port to Plone 5.2.

Addons

Yes, Plone is now able to run on Python 3. But without the huge ecosystem of addons that would be of limited real-world use. Jan Mevissen, Franco Pellegrini (remote) and Philip Bauer ported a couple of addons to Python 3 that Plone developers use all the time:

Being able to run tests against Plone in Python 3, and with these development-tools now available also, it is now viable to start porting other addons to Python 3. To find out how hard that is I started porting collective.easyform. The work is only 97% finished since some tests need more work but it is now useable in Python 3!

ZODB Migration

Harald Friesenegger and Robert Buchholz worked on defining a default migration-story for existing databases using zodbupdate. To discuss the details of this approach we had a hangout with Jim Fulton, David Glick and Sylvain Viollon. They solved a couple of tricky issues and wrote enough documentation. This approach seems to work well enough and the documentation points out some caveats but the ZODB migration will require some more work.

Relevant links:

ZODB Issues

Sune Broendum Woeller worked on a very nasty and complex issue with KeyErrors raising while releasing resources of a connection when closing the ZODB. That happened pretty frequently in our test-suites but was so far unexplained. He analyzed the code and finally was able to add a failing test to prove his theory. Then Jim Fulton realized the problem and wrote a fix for it. This will allow us to update to the newest ZODB-version once it is released. See https://github.com/zopefoundation/ZODB/issues/208 for details.

WSGI-Setup

Thomas Schorr added the zconsole module to Zope for running scripts and an interactive mode. Using WSGI works in Python 2 and Python 3 and will replace ZServer in Python 3.

Frontend and Theming

Thomas Massman, Fred van Dijk, Johannes Raggam and Maik Derstappen looked into various front-end issues, mainly with the Barceloneta theme. They closed some obsolete tickets and fixed a couple of bugs.

They also fixed some structural issues within the Barceloneta theme. The generated HTML markup now has the correct order of the content columns (main, then left portlets, then right portlets) which allows better styling for mobile devices. Also in the footer area we are now able to add more portlets to generate a nice looking doormat. See the ticket https://github.com/plone/plonetheme.barceloneta/pull/163 for screenshots and details.

They also discussed a possible enhancement of the Diazo based theming experience by including some functionality of spirit.plone.theming into plone.app.theming and cleaning up the Theming Control Panel. A PLIP will follow for that.

New navigation with dropdown support

Peter Holzer (remote) continued to work on the new nativation with dropdown-support which was started on collective.navigation by Johannes Raggam. Due to the new exclude_from_nav index and optimized data structures the new navigation is also much faster than other tree based navigation solutions (10x-30x faster based on some quick tests).

Static Resource Refactoring

Johannes Raggam finished work on his PLIP to restructure static resources. With this we no longer carry 60MB of static resources in the CMFPlone repository, it allows to use different versions of mockup in Plone, enable us to release our libraries on npm and will make it easier to switch to a different framework. Skin scripts removal Katja Süss, Thomas Lotze, Maurits van Rees and Manuel Reinhardt worked hard at removing the remaining python_scripts. The work is still ongoing and it would be great to get rid of the last of these before a final Plone 5.2 release. See https://github.com/plone/Products.CMFPlone/issues/1801 for details. Katja also worked on finally removing the old resource registry (for js and css).

Test parallelization

Joni Orponen made a lot of progress on speeding up our test-runs by running different test-layers in parallel. The plan is to get them from 30-60 minutes (depending on server and test-setup) to less than 10 minutes. For a regularly updated status of this work see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225

Documentation and User-Testing

Paul Roeland fought with robot-tests creating the screenshots for our documentation. And won. A upgrade guide to Plone 5.2 and Python 3 was started by several people. Jörg Zell did some user-testing of Plone on Python 3 and documented some errors that need to be triaged.

Translations

Jörg Zell worked on the german translations for Plone and nearly got to 100%. After the sprint Katja Süß did an overall review of the german translation and found some wording issues with need for a discussion

Plone React

Rob Gietema and Roel Bruggink mostly worked on their trainings for React and Plone-react, now renamed to “Volto”. Both will be giving these trainings at Plone Conference in Tokyo. On the second day of the sprint Rob demoed the current state of the new react-based frontend for Plone.

Assorted highlights

  • In a commit from 2016 an invisible whitespace was added to the doctests of plone.api. That now broke our test in very obscure ways. Alessandro used some dark magic to search and destroy.
  • The __repr__ for persistent objects changed breaking a lot of doctests. We still have to figure out how to deal with that. See https://github.com/zopefoundation/zope.site/issues/8 for details.
  • There was an elaborate setup to control the port during robot-tests. By not setting a port at all the OS actually takes care of this to makes sure the ports do not conflict. See https://github.com/plone/plone.app.robotframework/pull/86. Less is sometimes more.
  • Better late than never: We now have a method safe_nativestring in Products.CMFPlone.utils besides our all-time favorites safe_unicode and safe_encode. It transforms to string (which is bytes in Python 2 and text in Python 3). By the way: There is also zope.schema.NativeString and zope.schema.NativeStringLine.
  • We celebrated the 17th birthday of Plone with a barbecue and a generous helping of drinks.