Planet Plone - Where Developers And Integrators Write

Plone migrations: importing collective.jsonify export data into ArangoDB (Part 3)

Posted by Andreas Jung/ZOPYX on November 27, 2020 08:47 AM
Plone migrations: importing collective.jsonify export data into ArangoDB (Part 3)

In the last blog post, I described how to export a Plone site using collective.jsonify to the filesystem as directory of JSON data. This blog post will explain briefly to import the JSON data into ArangoDB as interim migration database


Why ArangoDB? We are using ArangoDB as a document database for storing the JSON data. This allows us to introspect and query the data to be migrated and imported. This is often necessary in complex migration with unknown data and unknown object relations. Using grep or similar tools on a huge JSON dump is unlikely the right approach. Using ArangoDB also allows us to run partial migrations for testing purposes (e.g. for testing the migration on a particular folder or a particular content type).

ArangoDB is multi-model database (key-value store, document store, graph database) and is available as community edition, an enterprise edition and as cloud edition (ArangoDB Oasis). The community edition is sufficient for migration projects. There are various installations options for all operating systems, all Linux distributions and through Docker.

After the installation of ArangoDB, you need to create a dedicated user account within ArangoDB and a dedicated database. This can be easily achieved through the ArangoDB web UI.  We assume for the following steps that you have created a database named plone and an ArangoDB user account plone with password secret.


Preparing your target Plone site

We assume that the target Plone site is running on Plone 5.2 and Python 3. The core functionality is integrated in the collective.plone5migration add-on for Plone. Prepare your buildout like this:

[buildout]
extends = buildout.cfg

auto-checkout +=
    collective.plone5migration     

sources = sources

[sources]
collective.plone5migration = git git@github.com:collective/collective.plone5migration.git


[instance]
eggs +=
    collective.plone5migration

Importing your JSON data into ArangoDB

After running buildout with the configuration given above, you will find this generated import script:

 bin/import-jsondump-into-arangodb --help
usage: import-jsondump-into-arangodb [-h] [-d DATABASE] [-c COLLECTION] [-url CONNECTION_URL] [-u USERNAME] [-p PASSWORD] [-i IMPORT_DIRECTORY] [-x]

optional arguments:
  -h, --help            show this help message and exit
  -d DATABASE, --database DATABASE
                        ArangoDB database
  -c COLLECTION, --collection COLLECTION
                        ArangoDB collection
  -url CONNECTION_URL, --connection-url CONNECTION_URL
                        ArangoDB connection URL
  -u USERNAME, --username USERNAME
                        ArangoDB username
  -p PASSWORD, --password PASSWORD
                        ArangoDB password
  -i IMPORT_DIRECTORY, --import-directory IMPORT_DIRECTORY
                        Import directory with JSON files
  -x, --drop-collection
                        Drop collection

In the former blog post we created a JSON export in I/tmp/content_test_2020-11-25-15-40-59.

You can import this export directory into ArangoDB using:

bin/import-jsondump-into-arangodb -i /tmp/content_test_2020-11-25-15-40-59 -x -u plone -p secret -d plone

Output:

connection=http://localhost:8529
username=root
database=ugent
collection=import
import directory=/tmp/content_test_2020-11-25-15-40-59
truncating existing collection
truncating existing collection...DONE
......

A nice progressbar will should you the progress of the import operation. The import speed depends on your local system. A typical import of 100.000 JSON files (100.000 Plone objects) with a total size of 50 GB takes about 45 to 60 minutes (largely dependent on the IO speed of your disk).


collective.plone5migration has been developed by Andreas Jung as part of a customer project with the University Ghent (migration of the ugent.be site to Plone 5).

Plone migrations: exporting a Plone site using collective.jsonify (Part 2)

Posted by Andreas Jung/ZOPYX on November 25, 2020 03:17 PM
Plone migrations: exporting a Plone site using collective.jsonify (Part 2)

In this blog post, I describe to how to use collective.jsonify for exporting a Plone to JSON.

Buildout preparation

  • install collective.json (we are using our ugent-fixes branch. This branch will be merged back into the master branch soon)
[buildout]
sources = sources
auto-checkout =  
   collective.jsonify

[instance]

eggs +=
   collective.jsonify

[sources]
collective.jsonify = git git@github.com:collective/collective.jsonify.git branch=ugent-fixes  

Installation of collective.jsonify inside your Zope/Plone installation

bin/instance run src/collective.jsonify/collective/jsonify/scripts/install_migration.py

Run the migration export

bin/instance run src/collective.jsonify/collective/jsonify/scripts/migration_export.py <plone site id>

You will see some output similar to

2020-11-25 15:40:59 WARNING collective.jsonify export >> Directory /tmp/content_test_2020-11-25-15-40-59

Depending on the site of your Plone site and the number of Plone object, the export may take a while. An export on a larger customer site with 100.000 objects and 60 GB data, an export took about 90 minutes...these numbers may vary depending on disk and CPU speed. The export will generate an export directory in /tmp (by default). The directory name is based on the Plone site ID and a timestamp. You can specify a different export directory by setting the $EXPORT_DIR directory.

The next blog post will cover the aspect of setting up ArangoDB as migration data and importing the exported JSON data into ArangoDB.

Plone Migration approaches (Part 1)

Posted by Andreas Jung/ZOPYX on November 24, 2020 01:07 PM
Plone Migration approaches (Part 1)

This is the first of a series of blog posts that deals with Plone migrations. I am trying to tell the story about how we are doing large-scale Plone migrations which usually involves migration of code, switching add-ons and  reorganizing content. This is usually accomplished in our migration through an export of the original Plone site and "magic" importer script on the target system. This migration approach evolved over the years. Our current approach is based on the experience and the work over the last year when we worked on a large-scale migration for the University of Ghent (90.000 content objects, 50 GB of data) and some sites for an industrial customer in Stuttgart.

Migration approaches

In-place Plone migration

Plone 5.2 supports a so-called “in-place” migration which is usually the right choice for simple sites with simple content-types and standard add-ons like PloneFormGen. The database file of Plone 4.3 is migrated in-place and transformed into a Plone 5.2 compliant database file. This is the approach in theory. It is often necessary to perform additional work and additional migration steps to migrate an old Plone database to Python 3 format - in particular when the Plone site has a certain age and history (which often implies a lot of cruft and old configurations that can break or interfere with the standard Plone migration). We use this migration approach only for small and simple sites.

Export/Import using Transmogrifier

The export/import approach allows us to perform a clean export of the content from the old site and to perform a clean import into a fresh Plone 5.2 site. So the new site will not contain any cruft content or obsolete configurations etc. The export of the is usually accomplished using the collective.jsonify add-on for Plone which basically exports all content objects and their metadata as JSON - one JSON file per content object. We also have supplementary code to export additional Plone settings or e.g. the Plone users and groups (if need, unless users and groups are maintained in LDAP or Active Directory or something similar). The exported JSON files are the foundation for a fresh import using the Plone Transmogrifier tool. Transmogrifier is basically a processing pipeline for piping the JSON data into Plone. The import can be customized using so-called “blueprints” in order to inject project-dependent dependencies, filtering steps, etc.

Export/Import using a custom migration pipeline

In our larger Plone migration projects, we often use a variation of the Transmogrifier approach. The export of the data and content from the original site is also done using the collective.jsonify add-on. The exported JSON files are then being imported into a migration database (we use ArangoDB). The migration database allows us to perform a partial import, query, and inspect the migration data in order to work on migration aspects and problems more easily than searching to Gigabyte of JSON files on the filesystem with operating system related tools like “grep”). The import is done using a custom migration package (which is to 80% generic for most Plone migrations, the remaining 20% are project-specific adjustments or additions because every migration is different). Our migration package allows us to run partial migration in order to test migrations more flexible. E.g. the migration package allows us to import a subset of content (either one or more folder or only specific content-types). This is very handy and time-saving. Migrating a large Plone site with a lot of content can take many hours. So testing changes on a particular feature with a subset of the data in advance is always better than having a failure on the full migration set after some hours.

Some new Plone add-on releases

Posted by Andreas Jung/ZOPYX on November 23, 2020 04:05 PM
Some new Plone add-on releases

Today, we released some small Plone add-ons as we developed and used over the last years for customer projects. In many cases we duplicated the same over and over again...so it was time for some refactoring and moving the code into dedicated packages.

zopyx.plone.persistentlogger

zopyx.plone.persistentlogger supports persistent logging where the log data is stored on an arbitrary persistent Plone object (as annotation). Typical usecases are application specific logging e.g. for logging a history per content object directly in Plone rather then having a huge common log on the filesystem. The log entries are stored using object annotations.

zopyx.plone.persistentlogger
Persistent logging for Plone objects
Some new Plone add-on releases

collective.metadataaudit

collective.metaaaudit logs metadata changes to Plone content objects transparently (for all content types). This add-on in in particular useful when you have to answer the question “who was doing what and when”.

collective.metadataaudit
Audit metadata changes
Some new Plone add-on releases

collective.debug

This add-on provides a @@debug browser view that returns some debugging information on the current context object, the current user and its roles, the instance dict of the current context object and request information.

collective.debug
A Plone debug view for introspecting the current context, user, roles and instance dict
Some new Plone add-on releases

collective.contentsync2

Content sync between Plone sites over plone.restapi

Features

  • sync content folders or individual content objects to a remote Plone site
  • full sync for initial sync
  • incremental sync for content updates
  • configurable sync behavior per content type
  • trigger immediate sync upon create or update operations through content rules
collective.contentsync2
Content sync between Plone sites
Some new Plone add-on releases

Foundation Board of Directors Nominations Open

Posted by PLONE.ORG on November 20, 2020 10:56 AM

The Plone Foundation is a not-for-profit, public-benefit corporation with the mission to promote and protect Plone. It protects the trademark, copyrights and other intellectual property, hires the release manager, works with our various communities and committees and develops policies where needed. It is a rewarding and much appreciated way to give back to the Plone community.

The official announcement describes the nomination process and gives more information about the work involved in being a board member.

Please Consider Serving!

Plone 5.2.3, Plone 5.1.7 and Plone 4.3.20 released!

Posted by PLONE.ORG on November 20, 2020 08:15 AM

General notes:

Plone 5.2.3

Plone 5.2.3 is a bug fix release of Plone 5.2.

Download Plone 5.2.3

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.3/versions.cfg.

  • Linux/BSD/Unix users: Use the Unified Installer. It is a configuration and setup kit with build scripts.
  • Windows 10 users: Use the Unified Installer. See Windows-specific installation instructions. Consider using the Unified Installer within the Windows Subsystem for Linux (WSL).
  • OS X users: use the Vagrant kit or install XCode command-line tools and use the Unified Installer.
  • Automated provisioning: See Plone's Ansible Playbook for a full-stack installation kit.
  • Cross-platform Docker: install Docker and use the Plone Docker image.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

Some highlights of this release are:

  • zope.interface: Fixed potential memory leak, see https://github.com/zopefoundation/zope.interface/issues/216. Fixed inconsistent resolution orders, see https://github.com/zopefoundation/zope.interface/issues/199.
  • Zope: fixes for a few template syntax errors. HTTP header encoding support.
  • A few possible information disclosure problems in handling of XML and of ical urls were reported by MisakiKata. They have been fixed by the Plone Security Team. Since they require an attacker to already have Manager or Site Administrator rights, we decided it was not necessary to create a hotfix for this. See https://github.com/plone/Products.CMFPlone/issues/3209
  • plone.recipe.zope2instance: added options clear-untrusted-proxy-headers and max-request-body-size.
  • Products.MailHost: support messages with explicit Content-Transfer-Encoding 8bit, see https://github.com/zopefoundation/Products.MailHost/issues/30. Note: in add-ons this may require changes to the tests.
  • mockup: fix plone toolbar action links being updated only on the first navigation action in the folder_contents structure pattern.
  • plone.staticresources: updated Bootstrap Icons to 1.0.0 final.
  • plone.app.contenttypes: allow passing a custom catalog-query to migrateCustomAT to constrain which objects to migrate.
  • plone.dexterity: make sure that Dynamic schema is updated on all ZEO clients on change.
  • z3c.form/plone.app.z3cform: fixed compatibility with changed repeat syntax in Zope 4.4, see https://github.com/zopefoundation/z3c.form/issues/94.
  • Products.ATContentTypes: drop use of test() in templates, unsupported since Zope 4.4.
  • Lots of deprecation warnings fixed, especially during startup.

For detailed changelog, go to https://plone.org/download/releases/5.2.3

Plone 5.1.7

Plone 5.1.7 is a bugfix release of 5.1. Note: this is the last release in the 5.1 series.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.1.7/versions.cfg.

Some highlights are:

  • Integrate Plone20200121 hotfix.
  • Security: depend on Products.isurlinportal. Version 1.1.0 has hardening against white space.
  • plone.recipe.zeoserver: Windows fixes
  • mockup and plonetheme.barceloneta: various frontend fixes, including translations
  • Lots of translation updates
  • Lots of bug fixes in many packages.
  • plone.namedfile: Range support
  • plone.scale:
    The mode argument replaces the old, now deprecated, direction argument.
    The new names are contain or scale-crop-to-fit instead of down,
    cover or scale-crop-to-fill instead of up and scale instead of thumbnail.
  • plone.supermodel: added support for choices of integers for improved registry.xml export.
  • Products.PluggableAuthService: Added new events to be able to notify when a principal is added to or removed from a group.

For detailed changelog, go to https://plone.org/download/releases/5.1.7

Plone 4.3.20

Plone 4.3.20 is a bugfix release of Plone 4.3. Note: this is the last release in the 4.3 series.

Note that support for Python 2.6 was dropped a while ago. It might still work, but you should use Python 2.7.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/4.3.20/versions.cfg.

Some highlights are:

  • Integrated PloneHotfix20200121 for increased security.
  • Moved the security check if a url is in the portal to a small separate package: Products.isurlinportal. You can immediately use this on Plone 4.3 and higher. Keep an eye on updates for this package: newer versions will increase the security. Often the impact of fixes is too small to warrant a real security hotfix package, but we want to do more regular fixes here.
  • Use Products.isurlinportal 1.1.0 with security hardening against whitespace: https://github.com/plone/Products.isurlinportal/issues/1
  • Removed broken X-XSS-Protection header from classic theme and unstyled theme.
  • Products.PluggableAuthService: Added new events to be able to notify when a principal is added to or removed from a group. Notify these events when principals are added or removed to a group in ZODBGroupManager. See https://github.com/zopefoundation/Products.PluggableAuthService/issues/17
  • z3c.autoinclude: When environment variable Z3C_AUTOINCLUDE_DEBUG is set, log which packages are being automatically included.

For detailed changelog, go to https://plone.org/download/releases/4.3.20

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

Data-driven documents with Volto and Plone

Posted by PLONE.ORG on November 09, 2020 12:05 PM

Executive Summary

Innovations in the Plone environment continue to expand the platform's capacities to provide scalable solutions. The BISE project realized by Eau de Web demonstrates the potential of joining Plone's world-class CMS back-end with the flexibility of the Volto front-end user interface. In this case, the integration enabled a highly flexible display of data-driven dashboard information with factsheets that are editable through the web by content managers. Get the full story below.


An Innovative Approach to a Known Problem

The BISE website (biodiversity.europa.eu) is one of the thematic environmental information websites run by the European Commission (DG-ENV) in cooperation with the European Environmental Agency. It used to run on Plone 4, but as part of the obligatory upgrade to Plone 5, we (Eau de Web) have migrated to a system based on Plone 5 and Volto 8.

Product of almost one year of work, as it stands right now, BISE is yet another example of the benefits of bringing closer the frontend development world to the solid foundation provided by Plone. The highlights of the BISE portal are, probably, the Country Biodiversity Factsheets, available in the Countries section.

Screenshot_20201029_155123.png

A mix of data, charts, and specially created visualizations would surely imply, in classic Plone development fashion, a dedicated content type and Python code, but in our case, the content type has been created through the web and holds little logic or knowledge about the end result. Instead, the factsheets are fully editable and composable through the web, based on somewhat generic Volto blocks: “plain” rich text blocks, chart blocks, a couple of “data table” types, and a “columns block” to enable composing these in a more complex but mobile-responsive layout. For even more flexibility, we’ve created a generic block styling framework that can be used for any Volto block and enables basic styling, such as alignment, text colors or to choose from a palette of predefined styles.

Everything, starting with the data definition (which can be provided either as plain CSV documents uploaded as Files in Plone or as SQL queries for a custom content type that communicates with the SQL server via REST) is completely generic and editable through the web, as part of the Volto UI.

The newly arrived “Volto dexterity-based content layout” bridges the gap between the goodness of Volto blocks and the templating power of content types. Add on top of that a thin layer that automatically adjusts templated content data (text, charts, data tables) to the context metadata (in our case, country code) and we get a high productivity environment that empowers the content experts to provide significant enhancements to the website without relying on the developers

plotly.png

Volto-Slate and Other Contributions

One of the key components of this website is also invisible due to its ubiquity: a new rich text editor, the volto-slate. The Biodiversity website relies heavily on the text editor: the website has a lot of textual information and its editors need to rely on a feature-rich editor. Also, the country factsheets use “templated text”, predefined (but editable) text that embeds numeric facts that come from the data connectivity framework. These “data-connected text entities” need to be fully configurable and styled by the rich text editor.

Early in the development cycle and having the experience of a prototype built with Volto’s default Draft.js editor, the development team decided to abandon it and reimplement a drop-in replacement with another library: SlateJS. Rather than providing an editor with a default configuration, SlateJS is different: it comes with the building blocks and a rich React api to enable building any kind of richtext editor. The Volto blocks and its full-page editing philosophy are rather specific, so we needed a library that can be used for this kind of deeper integration.

In the end, the volto-slate editor reached a stage where it can provide a level of integration that exceeds that of Volto’s built-in editor: improved block splitting and joining, better pasting with block splitting, automatic image and table block extraction and better overall handling of lists and sublists, just to name a few. It is now a stable target and provides plugin primitives for other rich-text integration tasks, sprouting other Volto addons dedicated to it: volto-slate-zotero, volto-slate-metadata-mentions, etc.

Screenshot_20201028_150648.png

More additions to the overall Volto ecosystem

In fact, volto-slate constitutes a substantial part of the overall development effort, but it wasn’t the only contribution that the BISE project made to the overall Volto ecosystem. Among them we can list:

  • Major improvements to the Volto addons architecture
  • Various bug fixes and enhancements to Volto core
  • Contributions to various Volto-connected projects, such as plone.restapi, the yeoman-based Volto app template generator and the styleguidist documentation
  • Many addons available as open source from the EEA github repo:

dotted-table-chart.png

Overall, the BISE website delivers a unique product that could have been created only with Volto and Plone. Through the freedom that they provide, Volto blocks are a foundation for innovation that enables Plone to step in line with the latest state of the art for web development.

Overall, the BISE website delivers a unique product that could have been created only with Volto and Plone.

Eau de Web is a software company specialising in Web development. The drivers of our activities are the open-source culture and the use of open standards due to the benefits of code reuse, openness in participating in projects worldwide, peer review and contributions from a wide community of developers. Our group has used Plone for web development from the early days, Eau de Web is a Plone foundation member, and our team members continue to demonstrate a long-term commitment to share and contribute the knowledge, software and resources to benefit others through Plone’s open source community.

Mikel Larreategi becomes a member of the Plone Foundation

Posted by PLONE.ORG on October 29, 2020 03:02 PM

The Plone Foundation welcomes a new member, after unanimous confirmation by the Foundation's Board of Directors on October 22, 2020.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.


Mikel Larreategi

erral.jpgMikel started working with Plone with the 2.1 version after joining CodeSyntax. Mikel attended several Plone Conferences (Naples, Washington DC, Budapest, Bristol (twice), San Francisco, Arnhem, Bucharest, Barcelona and Ferrara, and, between other community activities, maintains the translations of Plone into Spanish and Basque.

The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Plone Conference 2020 - A Sneak Peek

Posted by PLONE.ORG on October 28, 2020 08:04 PM

Over 65 Talks Submitted

The topics range from Plone, Zope, Volto and Guillotina to Python and Pyramid, and from fancy JavaScript to cool case studies. There are 30 and 45 minute talk slots, as well as lightning talks. 

The talks are targeted at a variety of different audiences such as designers, developers, integrators, users, and beginners alike. There will be something for everyone and the schedule will be planned so, that it is easy to find suitable talks for every need!

Volto Talks Galore

Volto, Plone's React front end, continues to gain interest, and there will be plenty of talks to catch you up on the latest developments:

  • Volto FTW
  • Building Volto Addons
  • Volto: A Journey Towards Personalisation
  • Plone and Volto in a Jamstack Project

What's in Store for Plone 6

Many people are helping move Plone 6 closer to reality - here are some of the topics they will be presenting:

  • Plone 6 Theming: Barceloneta
  • Plone 6 Theming with Diazo
  • Modernizing Plone's Classic UI

Agile, Patterns, Projects

We will dive into what goes into a successful project - from project management to technical architecture to building community:

  • Second Guessing the Single-Page-App Pattern
  • Asking Questions for the Benefit of your Future Self - Growing with the Plone Community
  • Agile Race To Zero
  • The Effectiveness of Open Source to Achieve a Common Heritage for Cities

And More!

Here are a few more intriguing-sounding talks:

  • Guillotina: Real Use Cases & Roadmap
  • State of Pillow
  • Collaborating With Orchid Data
  • Rapidly Building An Extensible Corruption Tool
  • Oh, the Places We've Been! Plone 2001 - 2020

There will be something for everyone!

Get you Plone Conference tickets now at https://2020.ploneconf.org/

New print-css.rocks release

Posted by Andreas Jung/ZOPYX on October 16, 2020 07:45 AM
New print-css.rocks release

Earlier this month, I released an update of my PrintCSS Portal print-css.rocks.

Thiis update includes support for new tools like

  • WeasyPrint
  • Typeset.sh
  • PagedJS
New print-css.rocks release

I also want to mention the portal printcss.live by Andreas Zettl. printcss.live is a very nice and supplementary addition to the PrintCSS community because it allows you to test tools directly within the browser. Andreas Zettl also maintains a list a PrintCSS articles on medium.com.

Blog moved to Ghost

Posted by Andreas Jung/ZOPYX on October 14, 2020 04:59 PM
Blog moved to GhostGhost Logo
Blog moved to Ghost

After almost 12 years I moved my personal blog blog.zopyx.com from Plone 4 to the Ghost blogging platform. You may ask why? First, my old Plone 4 installation reached end of lifetime a long time ago and it was time for moving the content to a different system. I investigated a lot of time into options like static site generators like Eleventy, Huge and others but I was not very much happy with them. I just wanted a simple system where I can add and publish content easily.

The migration from Plone 4 to Ghost was a bit painful. First, I tried to use plone.restapi or collective.jsonify but the Plone 4 site gave me a bunch of headaches and errors. So I decided to create a static copy of the original bog using httrack. With some Python code I was able to extract the content part from the HTML markup of each document and some necessary metadata like publication date, title, description etc. All extracted data was written to a JSON file.

Ghost itself provides some JSON format as import/export format. Due to limited resources I hired a Ghost developer (for a very small budget) that converted my JSON format into the Ghost JSON format.

Then setting up Ghost (running in a Docker container) behind Apache was more or less straight-forward. As with every new system: you need to learn and try things out.

For the moment, I am happy with the result and outcome. There are clearly some issues like a better theme or some CSS fixes to be implemented.

I hope to blog again more in the future.

Plone Conference 2020 - Submit a Talk and Get Your Tickets!

Posted by PLONE.ORG on October 14, 2020 04:17 PM

Welcome to Plone Conference 2020 from where ever you are!

This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event https://2020.ploneconf.org. The conference will be held from 5th to 13th December 2020. No matter where you are, you can participate.

Some preliminary information (might change still):

  • Training will happen over the weekend - December 5 - 6.
  • 4 days of talks + 1 of open spaces - December 7 - 11.
  • Sprint - December 12 - 13.

Call for papers is open, please submit your talk proposal

The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond. There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences too

https://2020.ploneconf.org

Important dates

Stay tuned for more information! Follow @ploneconf and #ploneconf2020 on Twitter, Instagram, and TikTok!

Zope 5.0 released

Posted by PLONE.ORG on October 10, 2020 06:55 PM

Zope is the original open source web application server written in Python, featuring a component architecture and an object-oriented, hierarchical data model.

This release

  • drops support for Python 2.7 and 3.5
  • drops support for Zserver
  • adds support for the newly released Python 3.9

Although this release is a major one, it should be possible to use it without any changes if the following preconditions are met:

  • Your project runs on the latest Zope 4 release on Python 3.6+ and does not use ZServer.

  • Neither your tests nor running your Zope instance shows DeprecationWarnings.

Additionally, this release contains some minor features and bug fixes that have been backported to Zope 4 where possible.

As Zope 5 is now released, Zope 4 is put into bugfix mode until 2021-12-31.
(See the roadmap for details: https://zope.readthedocs.io/en/latest/roadmap.html 3)

For details about this release see the what’s-new-section and the changelog:

To install the new version see https://zope.readthedocs.io/en/latest/INSTALL.html

Plone 5.2.2 released

Posted by PLONE.ORG on September 27, 2020 01:00 PM

Plone 5.2.2 is a bug fix release of Plone 5.2. Release Manager for this version is Maurits van Rees.

Download Plone 5.2.2

Note: We want to do releases more often, to avoid having very big changes between two bugfix releases. Plone 5.2.3 is expected in October 2020. We announce pending releases for testing on https://community.plone.org, and we announce a final release there once it is out of the pending phase, even when the installers are not ready yet.

Installers are being made, so not all links below will work yet. Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.2/versions.cfg.

  • Linux/BSD/Unix users: Use the Unified Installer. It is a configuration and setup kit with build scripts.
  • Windows 10 users: Use the Unified Installer. See Windows-specific installation instructions. Consider using the Unified Installer within the Windows Subsystem for Linux (WSL).
  • OS X users: use the Vagrant kit or install XCode command-line tools and use the Unified Installer.
  • Automated provisioning: See Plone's Ansible Playbook for a full-stack installation kit.
  • Cross-platform Docker: install Docker and use the Plone Docker image.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

Release notes for Plone 5.2.2

This release packs a lot of changes: the previous release was from January 2020. Some highlights are:

  • Zope was upgraded from 4.1 to 4.5. This means the support for WebDav is back. And the logic around template engines got updated. There should now be fewer differences in behavior between various kinds of templates (Zope/Acquisition-aware or not) or which template engine is used (zope.pagetemplate or chameleon). There were some problems due to this change, but all should have been fixed now.
  • ZODB goes from 5.5.1 to 5.6.0. These  updated packages are needed for zodbupdate and ZEO. So if you are doing low-level transactional stuff, please take a look at its changes.
  • plone.recipe.zope2instance: changes for relstorage, sqlite, webdav, Windows.
  • Fixed problems running Plone on Windows. You may want to keep an eye out for newer releases of plone.recipe.zope2instance. See especially this issue: https://github.com/plone/plone.recipe.zope2instance/issues/144
  • Mockup and plone.staticresources: lots and lots of improvements and bug fixes in styling and javascript.
  • Big push in fixing various internationalization issues.
  • Integrated PloneHotfix20200121 for increased security.
  • Moved the security check if a url is in the portal to a small separate package: Products.isurlinportal. You can immediately use this on Plone 4.3 and higher. Keep an eye on updates for this package: newer versions will increase the security. Often the impact of fixes is too small to warrant a real security hotfix package, but we want to do more regular fixes here.
  • Use Products.isurlinportal 1.1.0 with security hardening against whitespace: https://github.com/plone/Products.isurlinportal/issues/1
  • plone.app.discussion: Extended existing review workflow by states rejected and spam.
  • plone.app.theming: Add custom CSS settings to theming control panel. You now have a textarea where you can simply add CSS without needing to override things or play with less/scss.
  • plone.namedfile and several other packages: support for Range requests.
  • plone.outputfilters has a possibly breaking change: change the image caption template to use <figure> and <figcaption>. If you want to change this, there is a new ImageCaptioningEnabler utility.
  • plone.restapi was upgraded from 6.1.0 to 6.13.7, with lots of new features.
  • plone.scale: the method signature for scaling was made clearer. The mode argument replaces the old direction argument. Direction is still supported, but now deprecated. The new name for direction down is: mode contain or scale-crop-to-fit. The new name for direction up is: mode cover or scale-crop-to-fill. The new name for direction thumbnail is: mode scale.
  • plone.testing 8.0.1 drops the z2 extra. If you need this, use the zope extra instead.
  • Products.CMFPlone: Add markdown extension settings to markup control panel.
  • z3c.autoinclude: When environment variable Z3C_AUTOINCLUDE_DEBUG is set, log which packages are being automatically included.
  • zope.interface was upgraded from 4.6.0 to 5.0.2. This has performance fixes that may have backwards incompatible changes. This makes the inheritance and method resolution order for interfaces more logical and more in line with how it works for 'normal' classes. In corner cases there can be subtle differences when there are several utility or adapter registrations for similar interfaces. For example, when looking for browser view X and there are two such views, Zope looks for the most specific matching interface, and this may have changed.

Changes

For detailed changelog, go to https://plone.org/download/releases/5.2.2

The First Plone Steering Circle Meeting

Posted by PLONE.ORG on September 26, 2020 05:00 PM

The Initiative

Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.

  1. A series of Steering Circle meetings will be held to discuss the issues and plot a way forward.
  2. Beginning at the November Plone Conference, the Board will set up a series of open discussions about the organizational structure of all our projects. All community members will be welcome!

This initiative is described in more detail in the Board‘s statement on the Plone governance process.

The First Meeting

The first Steering Circle meeting was held on Zoom on 22nd September 2020. Representatives from each community team were invited and 22 people from around the world attended. Discussion items included:

  • The purpose, process, and schedule of the Steering Circle
  • Status of the teams - are any inactive, should new ones be formed
  • The Zope team's concerns about expectations regarding meetings, minutes - no, this is not necessary
  • The Framework Team vs. the Roadmap Team vs. a proposed new Plone Core Strategic Team
  • How to foster transparent communication throughout the community - Plone, Zope, Guillotina, Volto

The meeting minutes are posted on community.plone.org.

Volto 8 Released With Exciting New Features

Posted by PLONE.ORG on September 25, 2020 05:38 AM

Volto is Plone's snappy, modern React front end, and the Volto team has added two major new features to the latest release.

Through the Web Custom Types

Beginning with Plone 4, site admins have been able to create customized content types easily in the browser using Plone's standard administrative UI. Now, this feature has been added to Volto. You can:

  • Add a custom content type
  • Edit its fields and fieldsets, help text, etc.
  • Select from numerous predefined field types (selection, text field, date, file upload etc.)
  • Rearrange fields on the edit form
  • Abstract content type elements into behaviors which can be used on other types

In short, it's possible to customize a content type in every possible way, and the resulting type is automatically searchable and addable.

Through the Web Custom Type Layouts

Blocks are Volto's way of adding and organizing various elements on a page. They can be text elements, images, videos, dynamic lists, video embeds, or whatever you decide to develop. In Volto 8, you can combine the features of blocks editing with your newly created content type, vastly enhancing the functionality and customizability of your custom content - all in the browser with just a couple of clicks. 

Try Volto Today

Download Volto from Github or kick the tires on the Volto demo site. 

Matthew Wilkes Publishes Book: Advanced Python Development

Posted by PLONE.ORG on September 15, 2020 04:43 PM

As some Plone community members know, one of our own, Matthew Wilkes, has been hard at work writing a book for the last year. We're very excited to announce that it has been published! It is Advanced Python Development: Using Powerful Language Features in Real-World Applications and it can be purchased at Matthew's website, Apress or Amazon.

The book follows the design and build of a real-world application example from prototype to production quality. In this context, it explains Python language features that aren’t routinely covered and introduces techniques that demonstrate large scale system design and development processes. Plus there are useful asides, best practices and library recommendations galore.

Matthew specializes in Python development, especially in the areas of web security and performance. He has been developing Python web applications for 15 years and has been very active in the open-source community in general and the Plone community in particular.

Matthew began using Plone in 2004 when it was at version 2.0. He has worked on various aspects of the core Plone software and was on the Plone framework and security teams through the Plone 4 era. He led the security team for over four years, modernizing security team practices by instituting unit testing of fixes across all versions, cooperation with Red Hat and other downstream vendors, and a pre-announcement policy. He has also contributed to the security of Zope, Django CMS and Pyramid, including recently rewriting Pyramid’s CSRF protection. This all gives him a wider view of security as a process rather than just deep knowledge of one system.

Matthew served as the administrator for Plone's participation in Google Summer of Code for many years. He has also served on the Plone Foundation's Membership Committee and Board of Directors, and he was instrumental in helping to overcome the many legal and logistical hurdles to bringing Zope under the Plone Foundation's umbrella.

Matthew is a welcome and frequent presence at Plone conferences and sprints and has been known to be involved in adventures in late night drinking. For example after the party at the first Bristol conference, Matthew was instrumental in arranging for the pub where many Plonistas were hanging out to continue serving them after official closing time. The party continued until 5 in the morning and we learned later that the place took in £16,000 that night. Matthew also collaborated with David Glick on the classic zodb.ws April Fool’s Day launch in 2011 when they managed to get a hacked-up version of the ZODB running in the browser - a prank which required both technical chops and a particular sense of humor.

Congratulations to Matthew!

Plone Conference 2020, November 14 - 22 Will Be an Online Event!

Posted by PLONE.ORG on September 11, 2020 08:23 PM

Welcome to Plone Conference 2020 from where ever you are!

This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event https://2020.ploneconf.org.

Some preliminary information (might change still):

  • Training will happen over the weekend - November 14 & 15.
  • 4 days of talks + 1 of open spaces - November 16 - 20.
  • Sprint - November 21 & 22

The conference will be held from 14th to 22nd November 2020. No matter where you are, you can participate.

Stay tuned for more information soon!

Call for papers is open, please submit your talk proposal at
https://2020.ploneconf.org

And one more thing...

Plone Conference 2021 will be held in Namur, Belgium!

Stay tuned for more information!

Plone Conference 2021 teaser

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'xxx@xxx.de': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'xx@xxx.de': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone xxx@xxx.de'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Plone Foundation Launches Governance Initiative

Posted by PLONE.ORG on August 20, 2020 06:07 PM

Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.

  1. Every two months beginning in September, a series of Steering Circle meetings will be held to discuss the issues and plot a way forward. Representatives from all Plone teams are being invited to the Steering Circle meetings. People who are not affiliated with a team may request an invitation from the Board.
  2. Beginning at the November Plone Conference, the Board will set up a series of open discussions about the organizational structure of all our projects (Plone, Zope, Guillotina, Volto, REST API, etc.) All community members will be welcome to attend and virtual participation will be possible.

This initiative is described in more detail in the Board‘s statement on the Plone governance process. Thanks to the Board for proposing this and we all look forward to a series of fruitful discussions!

Great Progress on Volto at the 2020 Beethoven Sprint

Posted by PLONE.ORG on July 24, 2020 01:40 PM

Everybody in the Plone community loves sprints. Not only because of the software we enhance or the code we write. The most crucial ingredient of a great sprint is meeting the people you like and that you haven’t seen in a while. This is harder to do in these coronavirus times, but the Beethoven sprint, held remotely from May 27th through the 30th, succeeded.

There were 34 participants, including substantial representation from kitconcept, RedTurtle, and Eau de Web. They used Discord voice channels throughout the day for a real sprint experience. The channels functioned like tables in a sprint room, where sprinters can go from one table to another to hear what people are discussing. The organizers also used Zoom calls for daily standups and Remo.co video calls in the evenings, which are structured like a bar. Sprinters brought their favorite beverage, good food, relaxed on a couch, and had good conversations in the evenings!

In addition to socializing, lots of work got done. The sprint was all about Volto, which is the new React front end for Plone 6. Highlights included:

  • Bug fixes, date widget improvements, a recurrence widget, a sort function, and work on widget reusability
  • Discussions and progress on a new Volto add-ons architecture based on slots
  • Improvements to Volto's multilingual story, including a test fixture and a translation for Romanian
  • Improvements to the Cypress accessibility test coverage
  • Discussions about Volto's theming story, including how to separate the CMS and public-facing themes
  • A form builder prototype, as well as lots of form builder discussions
  • A Volto listing block that includes sorting, view templates, and a table view
  • Progress on a Dexterity content types control panel for Volto, including a schema editor and RestAPI endpoint
  • Additional Volto training materials, as well as training test drives
  • Volto marketing and onboarding work, including a Volto website and screencasts
  • Discussions and improvements to the Plone backend, including a plone.restapi release
    and work on the thorny problem of a Dexterity site root object
  • Plone 6 roadmap discussions

Thank you to everyone who attended - Plone 6 is getting closer to reality!

Product Lifecycle Management with Plone

Posted by PLONE.ORG on July 14, 2020 07:02 PM

Plone at PNZ Produkte GmbH

PNZ Produkte GmbH is a niche manufactory of oil-based paints and coatings made from renewable resources. Next to its company brand product line, a substantial amount of business is generated from private label (PL) sales. These PL products are offered as a turn-key service: including promotional material, product information sheets and packaging. For some private label customers, PNZ also takes care of fulfillment to end consumers.

A typical finished product is derived from:

  • A Base Recipe which determines the ingredients, production methods, usage safety indications and applicable mandated regulations (e.g. GHS)  
  • A Master Product which defines the application specifications, usage instructions, packaging sizes, pigments and other general use attributes
  • A Finished Product which includes branding characteristics, marketing descriptions, article EANs and other customizations

Since 2015, Plone has been used as the engine for creating and maintaining product specifications. About 20 employees in multiple departments interact regularly with the software. The project received a “Digitalbonus” innovation award from Germany's Landesamt Oberbayern.

PNZ Product Settings Page

A Typical Product Lifecycle

When a new product is formulated, has passed its application and safety testing phase, and is ready to go in production, its specifications will be split in three separate information layers. Custom content types in Plone connect these layers and manage their contents. Plone's ability to quickly prototype content types through-the-web makes it easy to design an initial structure and refine it based on user experiences. Each information layer consists of 30 to 50 individual fields and can be extended as needs change over time. Connecting different types of data from dynamic and external sources is relatively simple as Plone offers a large variety of edit/display widgets out of the box. 

At each hierarchical level, new content undergoes a quality and regulations compliance control check before going through the Plone publishing workflow. This makes the document available for use in lower layers. Any base recipe can potentially power several master products, which in turn results in a large number of finished products. The current system powers some 25,000 individual articles or SKU's. 

At the lowest level, a "Finished Product" content type includes product-specific overrides and any information not inherited from the recipe and master product layers.

PNZ Product Information Flow

Since products may be offered to multiple geographical markets, all information must be translated. Plone's built-in multilingual capabilities make it possible to have each individual document available in several languages. Fifteen languages are currently in use within the PNZ system. Each field in a content type can be designated as translatable or multilingual, so that updates of multilingual fields can automatically propagate throughout the various translations.

Automating Content Creation

The information contained in its product lifecycle management system allows PNZ to automate the generation of a variety of ready to use content. Plone provides the BrowserView mechanism, used in conjunction with Zope Page Templates (ZPT), to determine what should be included in the content and how it should be rendered. PNZ Wrappers with inDesign and XML from Plone

PNZ creates colortone samplers and product information sheets, with a customized layout for each brand, using the tools provided by Plone. The documents then undergo a final conversion to PDF using “Produce & Publish” software, created by longtime Zope/Plone community contributor Andreas Jung at Zopyx.

In the packaging process, Plone powers the creative design of all product labels. A BrowserView is used to draw data from multiple content types and the company’s ERP system. This will generate a set of XML files for each product and its SKU variants, plus related dynamic images such as EAN codes, colortones, certifications and GHS warning signs.

PNZ Public Site With PloneThe XML files and images are then read into Adobe InDesign templates by the company's creative designers, resulting in a professionally designed wrapper for each SKU that always contains accurate information and complies with industry standards and specifications.

Other types of content include the generation of raw HTML and images for use in a PlentyMarkets shop, and the creation of Product Information Management (PIM) CSV export files for large customers such as DIY store chains. Finally, using plone.restapi, the system powers the product catalog of PNZ's soon to be relaunched public facing website which of course also uses Plone!

A Very Successful 2020 Python Web Conference

Posted by PLONE.ORG on June 26, 2020 03:09 PM

June 17-19, Six Feet Up hosted the 2nd Annual Python Web Conference using their newly developed all-in-one virtual event platform, LoudSwarm. Focusing on Python for the Web, the conference featured talks and trainings from several members of the Plone community.

The conference had a total of 266 attendees participating from 37 countries and 14 timezones. A unique aspect of the registration was that for each ticket purchased, a ticket was donated to a developer residing in a developing country. Those that received a free ticket would not have been able to otherwise attend, and they were able to gain valuable knowledge on the state of Plone and Python on the web.

Being a completely virtual event, several accommodations were set up to make the event as interactive as possible. Attendees could join the Gallery after each talk to ask the presenter questions and have discussions about the talk topic. Slack was also heavily used to foster discussions and interaction among the attendees. Evening Socials were created to network, play games, and keep the fun going after the day’s talks were done.

Having a Plone presence was great for the attendees who had never heard of Plone, and provided many with the opportunity to see Plone again in a new way. Many great discussions were had around new Plone features, customization tricks, and fond memories of past Plone Conferences.

Recordings of the talks are currently only available to registered attendees. Post-video access tickets are still available if you weren’t able to attend the live event. All talk and tutorial videos will be made publicly available on YouTube in September.

Plans for Python Web Conf 2021 are already underway. Visit https://2020.pythonwebconf.com to learn more about this year’s event.

Zope and the Plone Foundation

Posted by PLONE.ORG on June 25, 2020 02:59 PM

Plone was very early to open source governance - the Plone Foundation was one of the first 501(c)(3)s to be established to protect open-source software. In the years that have followed it's become more common for software to be protected under an umbrella organization, but that wasn't possible in the early days.

Zope started off as the property of Digital Creations, which became Zope Corporation. But in the early 2000's there was a desire to move the protection of Zope to a community effort following the same model, so a new non-profit was formed.

This worked well for ten years, but by the mid 2010's there were significantly fewer active developers of Zope and most were not interested in the administrative tasks required to maintain a non-profit corporation. Over the years, the Zope Foundation fell behind in its regulatory compliance until it became inactive. It still owned the copyright to Zope, as well as some funds, but it had very limited rights over them.

The Plone Foundation's mission to protect Plone implies a responsibility to protect Zope, as Plone depends on Zope for much of its core infrastructure. Informal discussions began around this time about Plone taking over the role of the Zope Foundation, but no substantial progress was made.

As the years passed, the need to solve the regulatory problem became more pronounced. It was clear that the Zope Foundation would not restart operations in any meaningful sense, but Zope development was undergoing a resurgence thanks to the need to port to Python 3.

At the 2018 Plone conference in Tokyo, after quite a few whiskies, people resumed talking about this problem, and we decided to try to solve it again. Because one of the biggest blockers was a lack of enthusiasm in the Zope community with regards to dealing with the bureaucracy, some people who were members of both Foundations reached out to the IRS, the Delaware Department of State and some experts in US Corporation law to plan a way forward.

A few months later in early 2019, the Zope Foundation held a special general meeting to pass a motion on the future of the Foundation. The membership mandated the board of directors to transfer all assets to the Plone Foundation as a donation, and to formally close the Zope Foundation. Later that same day the board of directors voted to donate all property of the Zope Foundation, including copyright in the Zope software, to Plone.

Since then, Plone community members have been working on the bureaucratic aspects of this move. We've harmonized contributor agreements to improve international enforceability, and work is underway to ensure that Zope contributors are properly represented in Plone's democratic structure before the next internal election in November. Zope contributors can already apply to be members of the Plone Foundation, and many are, but we're looking to streamline the process for former Zope Foundation members.

The Zope and Plone Security Teams have already been integrated, and now we need Zope representatives on the Membership Team. Any Zope people who are interested in helping out should please contact Érico Andrei.

Plone’s Migration to Python 3

Posted by PLONE.ORG on June 24, 2020 01:32 AM

July 19, 2019, marked the day Plone officially entered the world of Python 3 when version 5.2 was released.  Support for Python 3.6, 3.7, and 3.8 is now included with Plone 5.2, opening up the future for Plone’s continuing 19-year reputation for unrivaled security and stability.

Plone is the venerable open-source Python web content management system that was created in 2001. It has continued to be maintained and enhanced by hundreds of contributors around the world and is protected by the nonprofit Plone Foundation.

Plone 5.2 is the culmination of over 3.5 years of development that was driven by the looming deadline of January 1, 2020: the impending end of support for Python 2. Work on Plone 5.2 ran in parallel with continuing feature-centric work on versions 5.0 and 5.1.  The move to Python 3 was to be a major undertaking, given its huge codebase, its large ecosystem of add-ons, and its dependency on the Zope web application framework (https://www.zope.org), which itself would have to be upgraded to Python 3.  Many sprints were held over these past few years in which Plone and Zope developers collaborated in this joint effort.  One of the earliest fruits of this collaboration was to see all 9000 tests passing as a result of work done at the Alpine City Sprint in February 2017, but more work was still needed.  At the core of the security model for which Zope and Plone are renowned is a feature known as Restricted Python.  Although core developers had feared that Restricted Python could not be ported to Python 3, this was accomplished in Innsbruck, Austria: Restricted Python now runs in Python 3.6 and up.

ZODB, the ACID compliant and extremely robust database, was upgraded to the latest version for Plone 5.2, along with a Zope instance running via a WSGI server instead of ZServer.

Zope developers came together and released the Python 3 compatible Zope 4 in May 2019. Plone developers were able to release its Python 3 compatible version 5.2 soon after, complete with its traditional in-place upgrade, with an additional database upgrade step required to convert to Python 3 strings.  Plone 5.2 is the first release to fully support Python 3 while maintaining complete compatibility with Python 2.7, which is expected to be dropped with the upcoming release of Plone 6.

Once available as a separate add-on, a REST API is now also included in the core of Plone 5.2, making it even easier to use Plone with modern JavaScript front ends such as Angular and React and to integrate Plone with other applications.

This release provides the Plone community the certainty of having a stable backend without an expiration date and the excitement of developing a new frontend.  Plone is more than 19 years old, and Zope is more than 22 years old. Yet both projects have reinvented themselves several times and are maintained by a worldwide, dynamic community. These two platforms provide a very secure, flexible, and mature solution in the often uncertain, unstable landscape of web applications.

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

moniviestin workflow menu

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

moniviestin workflow

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

moniviestin example

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

moniviestin content rule

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

moniviestin rpa poll

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

  1. Video author selects “Publish with subtitles”.
  2. Plone content rule calls Jenkins webhook to schedule a new robot.
  3. Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
  4. Jenkins schedules a new job to poll pending ASR jobs.
  5. Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
  6. Plone content rule notifies the video author with email that automatically generated subtitles are now available.
  7. ???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

study guide tabs

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

hasura view

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

hasura relationships

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

volto contents

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

  1. Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

plone dexterity editor ,

  1. Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

volto sisu connector

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

study guide edit

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

study guide image

Something old, something new and something blue.

The perfect match.

Plone 6 Remote Sprint - April 2020

Posted by PLONE.ORG on May 01, 2020 07:21 PM

A Successful Remote Sprint

Last week we held the first Plone remote sprint in the time of the coronavirus pandemic. It was a positive experience and a lot got done! Sprint plans and results were captured on the plone-6-sprint Etherpad and some chatting occurred on the plone/sprint Gitter channel. But the main method of communication was by video. Participants stayed connected by video all day, either on Jitsi or Discord. Each team had their own "room", and it was nice to be able to see everyone while working, even when microphones were muted. This mimicked the sort of frictionless in-person conversations that are the hallmark of sprints.

Sprint Topics

The sprinters tackled six different topics, here are the final reports on each. For more detailed information, see the plone-6-sprint Etherpad.

Modernize Plone's Default Theme

Peter Holzer and Stefan Antonelli led the charge with help from Robert Kuzma, plus Peter Mathis taking the lead on z3c.form. The team made a long list of contributions toward the following PLIPs:

Their work changes the default theme plonetheme.barceloneta to be based on Bootstrap 4 and SCSS. Plone specific styles have only been added where Bootstrap is lacking definitions for Plone elements. There is an overrides directory where all template overrides will be collected until they are merged back into the original packages.

Markup in general was improved. main_template markup was updated and the structure cleaned up, empty and unneeded elements have been removed. Header, navigation, portlets, status messages, batching - all markup and nav styles were updated. The generic view for content types was updated (this includes title, description, byline, related items and keywords). control_panel_overview, control_panel default, and some control panels with custom views were all updated.

For forms, Bootstrap markup was added to plone.app.z3cform to give widgets and fields (buttons, input fields, etc.) a basic styling. Plone's classic backend forms are also fully functional as long as Bootstrap's default CSS is independently available. In fact a theme (e.g. Barceloneta) is not necessary as long as Bootstrap CSS is available.

Icons also got some love. icon_expr for actions, content types, etc. was reused to define SVG icons. bootstrap-icons were added as an available icon resource to plone.staticresources. The team selected and added icons to the control panel settings in the controlpanel.xml in plonetheme.barceloneta.

Mosaic was changed to use Flex for columns based on the updated Barceloneta theme. All LESS files were replaced with SCSS and 'npm run watch' is now used to compile the SCSS files. Redundant CSS classes were cleaned up and the default column limit was set to 6 columns. A reset button was added to Mosaic cells to reset to the automatic column width, and an automatic reset was added for column widths if a new column gets inserted. A label was added to indicate the current column width, with "0" for auto width.

A demo of classic Plone 5.2 with the latest Barceloneta default theme is at demo.plone.org. The Coredev buildout including the current state of work on the PLIP for Bootstrap 4 markup is at classic.plone.de.

The team plans to meet regularly to continue working every Wednesday at 5 PM CEST starting on May 13. They will meet in the plone-6-sprint-barceloneta Jitsi room. Everyone is welcome to join!

Simplify Resource Registry

Maik Derstappen led this project, in coordination with the Barceloneta work. They added a custom.css view, which renders the custom CSS added in the Theming Control Panel, after all other CSS resources are loaded. This allows you to override CSS coming from a Diazo theme.

The team also discussed what really needs to change in the Resource Registry. They decided to remove the build-button from the theming editor because it will not work with SCSS (which is now used by Barceloneta).

Developers should register resources already compiled as Resource Bundles with TTW compiling disabled. They can use the merge flag to merge it with default or logged-in bundles, but this only makes sense if they are always needed. Otherwise the resource can be disabled and loaded on demand via Python in the view/viewlet or by adding the header lines directly into the templates.

Variables can be defined as CSS variables. Variables from the registry are no longer used in SCSS/CSS in plonetheme.barceloneta.

Push Plone to Zope 5

Jens Klein reports that lots of "boring but important work" was done by Maurits van Rees including cleanup of package metadata and releases of packages. Jens Vagelpohl released Zope 5.0a2 and Jens Klein created a zope-5 branch for Plone 6.0 on buildout.coredev. Version dependencies were updated on this branch, and version pins that are orphaned or handled upstream by Zope were removed. Version dependencies still outstanding:

  • z3c.form - this will be handled later
  • Update Pillow from 6 latest to 7.x - check is needed on separate branch, this applies to the Plone 5.2 branch as well
  • Parts of the Robotframework test environment - due to non backward compatible changes in Selenium, and possibly other issues
  • Unicode Encode/Decode  - this one has been open for a long time and Jens would like to see some fellow Plonista take the challenge to update or get rid of it!
  • Cleanup of any orphaned dependencies - for some packages it is difficult to know were they are used, especially for dependencies of our ecosystem packages or tools

Many failing tests were fixed. Four tests (partly related to zope.interface 5.1) are still open and in-progress. Jens would appreciate help to get them fixed and merged.

Volto

Timo Stollenwerk led the Volto efforts with Victor Fernandez de Alba, Rob Gietema, Steffen Ring, Katja Süss, Alok Kumar and Rodrigo Ferreira de Souza. They did a complete overhaul of the folder contents view according to Albert Casado's Pastanaga UI design, and made numerous other UI improvements based on a full UX review that Albert did. Alok (frontend) and Rodrigo (backend) implemented the Add-ons Control Panel and Alok implemented the Users and Groups Control Panel. Alok also implemented print CSS for the Volto front end. Victor fixed the listing block to remove the current item from query results for ZCatalog 5.1. Finally, Steffan and Katja worked on content for the upcoming VoltoCMS.com website.

Image Scaling

Asko Soukka worked on image scaling in coordination with the Volto team, with additional help from Jens Klein and Alexander Loechel. They fixed several bugs:

They implemented an option (disabled by default) to generate all configured scales immediately once an image has been updated. Enabling this would slow down the save of an image field, but would prevent most writes on read for image scaling. The code is in plone.formwidget.namedfile.

They implemented a proof of concept for PLIP #3090 for asynchronous multiprocessing of image scaling, which compliments the above option to generate all configured scales on save. It could provide image scaling with good performance for Plone without any additional services.

They implemented a proof of concept for the use of picture & srcset in Volto for perfectly responsive images. The code is in Asko's Github account.

Mastering Plone 6

Philip Bauer led a team consisting of Steve Piercy, Katja Süss and Janina Hard to create and update the developer training class for Plone 6. Much was accomplished. The first alpha version of chapters 1 through 36 are now usable and will be deployed soon for testing.

Many of the chapters now exist in two flavors: one for Volto and one for Classic, with links between them. Logos at the top clearly identify which flavor you are reading. The default navigation follows the Volto story.

The team plans to do bi-weekly half day sprints and to finish the training at the Beethoven Sprint which is scheduled for May 26th. The outstanding chapters are: 

  • Customizing Volto
  • Membrane
  • Speaker content type and view component
  • Volto Frontpage

Plone Marketing

Sally Kleinfeldt and Érico Andrei investigated tools and planned a series of "how to" screencasts for the Plone Youtube channel about managing content in Plone. Érico also worked on adding the videos from the 2003 to 2009 Plone conferences to the channel.

Sally discussed how to position headless Plone and Volto with Timo Stollenwerk, Victor Fernandez de Alba and Katja Süss. As noted above, the Volto team is working on a new voltocms.com website which will primarily be aimed at React developer on-boarding. We discussed how plone.org/plone.com can coordinate with and compliment their efforts.

After discussion with additional interested parties (Rikupekka Oksanen, Stefania Trabucchi, Fulvio Casali and William Fennie), we decided to create a "What is Plone?" section on plone.org which is aimed at decision makers. It will give a bit of Plone history and describe the current variety of frontend and backend options and when to use them. We began brainstorming ideas and will meet again in 3 weeks. Input from other community members would be welcome.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.
  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.
  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.
  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.


All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Plone and the Pandemic

Posted by PLONE.ORG on April 03, 2020 04:51 PM

The COVID-19 pandemic has upended life across the planet. Scientists, doctors and other experts are engaged in a giant global effort to combat the disease. Open source software is integral to this effort because many tools and libraries used by these experts are open source. That includes Plone, a great platform for building secure, scalable, highly customizable, content-rich websites for scientists and researchers.

One such Plone site is already being used in the COVID-19 fight. The Onkopedia portal publishes medical guidelines for oncology and hematology for Germany, Austria and Switzerland - treatment guidelines, medical studies, protocols, certifications, drug information, etc. for over 60 diseases. Patients with these diseases are at high risk for the new coronavirus, which created an urgent need for providing COVID-19 specific information. In just 3 days the Onkopedia development team was able to extend the site's data model and roll out necessary UI updates to create a new Onkopedia section with COVID-19 content for over 40 different diseases. Congratulations to the team for a great example of agile development, and to Plone for providing a flexible and robust platform.

Do you have a success story about Plone helping to fight the pandemic? Let us know at news@plone.org.

PLOG 2020 Replaced by Remote Sprint

Posted by PLONE.ORG on April 03, 2020 04:01 PM

PLOG Felled by the Pandemic

After months of planning, in early March the PLOG organizers had to face the reality that traveling to Italy in April was not going to be feasible. We tried to reschedule the event in September, but the hotel was already booked up. So we've had to make the sad decision to cancel PLOG for 2020.

PLOG will return in 2021!!!

Remote Sprinting to the Rescue

For the foreseeable future, Plone community sprints will need to happen remotely - individual sprinters in their own spaces, communicating via audio, video, and text. Happily, the big April hole that PLOG's cancellation left in the Plone calendar has now been filled by a remote sprint on Plone 6. Kudos to Peter Holzer, Maik Derstappen and Jens Klein for organizing it! The sprint will tackle a number of topics:

  • Modernize classic Plone's default theme - The goal is to make working with the classic UI fun again. We will make the templates compatible with bootstrap markup and reduce the amount of custom CSS so that it is easy to use any bootstrap template with Plone. 
  • Push Plone to the latest Zope - The Plone CMS runs on top of the Zope web framework which has recently seen some massive performance improvements. To take advantage of that we will work on making Plone 5.2 run with latest Zope 4 and Plone 6 with latest Zope 5.
  • Mastering Plone 6 - We will update the Mastering Plone training to use Plone 6 and Volto for all frontend-tasks.
  • Create marketing materials - Headless Plone and Volto are important to our future, but they aren't clearly presented on plone.org or plone.com. We will work on how-to screencasts for both Classic and Volto UIs for the Plone Youtube channel, and plone.org content about headless Plone and Volto.

Join Us!

Every experience level is welcome, and additional sprint topics are welcome too. Sign up and add your ideas!

Plone and COVID-19

Posted by Andreas Jung/ZOPYX on April 02, 2020 12:00 AM
Plone and COVID-19

Plone and COVID-19

These days are tough and challenging for all of us. COVID-19 affects everyone across the globe. The Coronavirus hits every country differently. Fortunately, Germany seems to be better prepared than other countries. However, there will be a huge fallout from this crisis and many things will not be the same as they were for a while. So, please #stayathome and stay safe.

But let me tell you a small success story around Plone and COVID-19.

We are running the portal Onkopedia (www.onkopedia.com) for over a decade where the medical guidelines for oncology and hematology for Germany, Austria and Switzerland are published. The portal contains treatment guidelines for over 60 diseases which also includes medical studies, protocols, certifications and information about individuals drugs and active components. With the growing Corona crisis in Europe, high-risk patients with pre-existing conditions like cancer came into the focus. There was an urgent need for providing COVID-19 specific information for most diseases to the public.

The Onkopedia portal was always running and powered by Plone (nowadays running on Plone 5.2, Python 3). Within two or three days we had to extend our data model for COVID-19 in order to provide the functionality required to publish the COVID-19 information specific for each disease. With the help of the whole Onkopedia development team, we were able to to roll out the necessary Onkopedia update within a very short time from of three days. The required changes affected the Plone data model for the guidelines and a bunch of UI aspects like a new COVID-19 section and individual markers in guideline documents for referencing the related COVID-19 information in the COVID-19 master guideline.

As a result, we managed to roll out the new COVID-19 section on onkopedia.com (German only) yesterday with individual COVID-19 content for over 40 different diseases.

Key success factors: Plone and a great team (Abstract Technology, Practice Innovation, AppWeeve)

Thank you all for making this happen and for contributing in the fight with COVID-19.

Plone Tagung 2020 – Dresden, Germany

Posted by PLONE.ORG on March 25, 2020 03:30 PM

Each year, the Plone German-speaking Community organizes a 3 day Symposium (Plone-Tagung 2020) and a two day sprint for German-speaking universities, NGOs and companies as well as for users, service providers, and developers.

The symposium this year took place at the Technical University of Dresden (TUD) from the 9th to the 13th of March, 2020. The Symposium was supported by the Plone Foundation and sponsored by Werkbank (Gold Sponsor), Zopyx and FlyingCircus (Silver Sponsor) and Abstract-Technology (Bronze Sponsor).

Plone Tagung 2020

Symposium Topics

The Plone Symposium in Dresden focused on the opening keynote from Anne-Marie Nebe on accessibility and why web accessibility is so important for user-centered software development. The Humanities and Social Sciences Department of the TU Dresden contributed with their expertise on the linguistic and visual implementation of a barrier-free web presence. The second-day keynote "Chaos und Diversity“ from Christian Theune illustrated the characteristics and dynamics of complex systems and talked about new approaches for a conscious use of diversity.

A complete overview of all topics of the Symposium is available, in German, at the Plone Tagung website.

How to Make Plone Cookies

Building an open-source community isn’t easy but now you can print your own Plone cookie cutter and make these goodies on your own! https://www.thingiverse.com/thing:4226416

Thanks to @quirk_dispenser.

Plone Cookies

Thank You

A big thank you goes to the Technical University of Dresden (TUD) for hosting, the Plone-Team TUD (@plone_tudresden) for organizing the event, and to the speakers, sponsors and all participants, who made the event special.

If you weren’t able to make it this year, we hope you’ll join the Plone Tagung next year! See you then.

Volto 4.0 Released

Posted by PLONE.ORG on March 01, 2020 03:30 PM

Volto 4.0 is now ready, following a 9 month alpha period that included many sprints.

A lot of effort has been put into keeping Volto's learning curve low and ensuring that developing with it is a delightful experience.

Here are some of its new features:

* Improved Pastanaga editor
* New Pastanaga editor sidebar
* New mobile first toolbar
* Developing blocks is now easier than ever
* New object browser
* Listing, table of contents, and lead image blocks
* New blocks chooser and future-proof blocks definitions
* Body classes like Plone's hinting at content types, section and current view
* New message system
* React hooks support
* Several internal libraries updated, including Redux, Router ones that support hooks
* Many bug fixes

For further details, please see the Volto 4.0 announcement forum post

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.


If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

Plone Foundation Ambassadors for 2020

Posted by PLONE.ORG on February 15, 2020 04:17 PM

The Plone Foundation Board has appointed these outstanding individuals from around the world to serve as ambassadors for Plone.

They are known for their community involvement in their regions and for their expertise in a particular market of Plone.

In their capacity as Plone ambassadors, they will continue to promote Plone regionally and in their specific markets for Plone.

Plone's ambassadors for 2020 are:

  • David Bain, Jamaica
  • Leonardo Caballero, Venezuela
  • Manabu Terada, Japan
  • Max Jakob, Germany, Education
  • William Fennie, United States, Education
  • Thomas Buchberger, Switzerland, PloneGov
  • Joël Lambillotte, Belgium, PloneGov
  • Ramiro Batista, Brazil, PloneGov


See https://plone.org/foundation/board/ambassadors for more information


"Flags of member nations flying at United Nations Headquarters". 30/Dec/2005. UN Photo/Joao Araujo Pinto.

New Waitress version, and updated 20200121 hotfix

Posted by PLONE.ORG on February 11, 2020 11:29 PM

Waitress

If you use Waitress, please upgrade from 1.4.2 to 1.4.3. 

The Pylons Project released a new version of Waitress to fix a bug in the regular expression that was used to parse HTTP headers. The bug could cause the waitress process to use excessive CPU.

As Plone 5.2.1 uses Waitress 1.4.2, we recommend changing the version pin to 1.4.3 in your buildout.

[versions]

waitress = 1.4.3

Updated 20200121 Hotfix

As announced previously, the 20200121 hotfix includes several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting.

Version 1.1, released on February 11, 2020, includes an update for the SQL Injection fix, which will not be needed for all installations.

If you are not using SQL in your website you do NOT need to upgrade, though you can if you want to. Default Plone does not need it. Upgrading to this version is especially recommended when you use PostgreSQL. Note that RelStorage is not affected. For details and discussion, see DocumentTemplate issue #48

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the update patch on 2020-02-11T22:47:25+0000.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

PLOG 2020 Training - 3 Weeks Left to Register!

Posted by PLONE.ORG on February 08, 2020 10:08 PM

Free Training

PLOG is a unique sprint where you can spend the mornings training, the afternoons sprinting, and all day enjoying the southern Italian weather and food. Thank you to the Plone Foundation, which has provided funding so that we can offer the following FREE training classes. It's a great way to brush up some old skills and pick up new ones at the halfway point between Plone conferences.

Thank you to the Plone Foundation, which is providing funding to support the training classes. 

Hands on Volto -- Taught by the RedTurtle Team -- 2 Mornings

This class will be based on the Volto training given at the last Plone Conference. It assumes some knowledge of React (but you don't have to be an expert.)

How to Get More Value From Your Plone Site Using GatsbyJS -- Taught by Asko Soukka -- 2 Mornings

This class will teach you how to deploy a static website from Plone content, and how to integrate it with the GatsbyJS ecosystem (plugins, themes, cloud services, etc.)

Plone Tips and Tricks -- Taught by Philip Bauer -- 1 Morning

This class will cover a number of advanced Plone developer topics, including:

  • Python Debugging Best Practices
  • Debugging ZODB Issues

Strategic Sprint

Because of its importance to the community, PLOG has been designated a strategic sprint by the Foundation Board and the Marketing Team. A wonderful cross section of people will attend, so there will be sprinting activities for all interests. Some will be working on Volto and RestAPI improvements, others will be creating marketing collateral in the form of "how to" screencasts of Plone features. Bring your own topic - the final plan will be made at the sprint.

Less Than 3 Weeks Left to Register!

The registration deadline is February 28th, after that we cannot guarantee the discounted rates or room availability.

Register now

Want to Learn More?

Read all about the Plone Open Garden, a tradition in our community since 2007. There's even a PLOG video. Dates, prices, and all of the details are given on the PLOG 2020 event page.

Security patch released 20200121

Posted by PLONE.ORG on January 21, 2020 03:10 PM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers: CVE-2020-7936, CVE-2020-7937, CVE-2020-7938, CVE-2020-7939, CVE-2020-7940, CVE-2020-7941.

Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.

The patch was released at 2020-01-21 15:00 UTC.

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the patch at 2020-01-21 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

20200121

Posted by PLONE.ORG on January 21, 2020 03:00 PM
Several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting. Version 1.1 released February 11, 2020, with an update for the SQL Injection fix, which will not be needed for all.

Security vulnerability pre-announcement: 20200121

Posted by PLONE.ORG on January 08, 2020 04:43 AM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers not yet issued.

Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.

The patch will be released at 2020-01-21 15:00 UTC.

Preparation

This is a pre-announcement of availability of this security fix. 

The security fix egg will be named Products.PloneHotfix20200121 and its version will be 1.0. Further installation instructions will be made available when the fix is released.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

Should you not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team will release the patch at 2020-01-21 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as Python packages that may be installed by editing a buildout configuration file and running buildout. For Plone 5.1 and lower they are also available as tarball-style archives that may be unpacked into the products folder of a buildout installation. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM
Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

PLOG 2020 Registration Now Open

Posted by PLONE.ORG on November 27, 2019 04:47 PM

Sign up now!

Registration deadline is February 28, 2020

Training classes on several topics will be held in the mornings, sprinting will happen in the afternoons and discussions will go on all day. The Hotel Mediterraneo is in a beautiful location overlooking the Bay of Naples, with wonderful food and a lovely garden where the sprinting and training will be held.

We have reserved a limited number of rooms and they are available on a first come first served basis. Breakfast and dinner are included. The rooms for three and four persons are the most economical and are expected to sell out early.

We are looking for women to share a triple!

Questions? Email us at plog@plone.org

Event details ~~ Read About PLOG ~~ Watch the Video ~~ Register

Python, the most popular programming language of the year

Posted by CodeSyntax on November 19, 2019 06:56 AM
IEEE Spectrum has published its sixth annual list with the most popular programming languages of the year across multiple platforms, and, once again Python repeats in 2019 as the undisputed leader, as it will happen in 2017 and 2018.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM
I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):

Four Members Join the Plone Foundation

Posted by PLONE.ORG on November 13, 2019 09:47 PM

The Plone Foundation welcomes four new members, after unanimous confirmation by the Foundation's Board of Directors on October 10, 2019.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Fulvio Casali

Fulvio Casali

Fulvio started working with Plone in 2008 and since 2012 he has been operating a consultancy business, Soliton Consulting, specializing almost exclusively on Plone. Fulvio organized Emerald Sprints (2013, 2014), Plone Open Garden (2016, 2017, 2019, 2020), participated in every Plone Conference since 2010 (except one!), has given two conference talks, and attended the Plone Symposium East 2010 and the Plone Konferenz in 2012. Fulvio has provided Plone training for Mastering Plone, Diazo theming, Rapido, Angular, and TTW Dexterity.

Stefania Trabucchi

Stefania Trabucchi

Stefania is the co-founder of Abstract Technology in Germany. She was the organizer of Plone Meet Up Berlin 2004-2014, of World Plone Day Berlin 2009-2015, of Plone Social Sprint Berlin 2014, and of Plone's Europython 2014 presence and associated marketing. She has participated in the Plone Konferenz 2012 as a speaker, the Plone Beethoven Sprint 2019, the Plone Conference 2014 sprint, and the Plone Open Garden sprints in 2014 and 2015. Stefania represented Plone as a member of the CMS Garden 2014-2017 and is active in the Plone Intranet Consortium with Quaive.

Thomas Buchberger

Thomas Buchberger

Thomas is the CTO of 4teamwork AG, has been working with Plone since 2006 and became a code contributor in 2011. He has attended many Plone conferences, beginning with Naples in 2007. He attended many sprints, including the Barcelona Strategic Sprint (2016), the Bonn Beethoven Sprint (2017 and 2018), the Sorrento Sprint on Frontend Modernization and Python 3 Porting (2019), and the Beethoven Sprint (2019). Thomas' main code contributions have been in the plone.rest and plone.restapi modules.

Andrea Cecchi

Andrea Cecchi

Andrea was the lead organizer of the Plone Conference 2019 in Ferrara. He participated at earlier Plone conferences and sprints (Sorrento 2019, Tokyo 2018, Barcelona 2017, Bristol 2014). As part of the RedTurtle team, he has organized and participated in World Plone Day events since its inception. Andrea is the maintainer of over 100 packages on PyPI.org.

 

 

The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.