Planet Plone - Where Developers And Integrators Write

The Plone Newsroom Podcast

Posted by PLONE.ORG on October 18, 2021 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. Technical and non-technical topics will be covered, including Plone, the Plone community, and whatever else they come up with to keep us informed!

Suggest a topic to include in the next episode by sending an email to Philip or Fred (first name 'at' plone dot org).

The premier episode featured a discussion of Plone Open Garden and a roundup of conference and release news, plus information about collective explicit acquisition and the Volto Search Block. Go to the Newsroom page to view other episodes.

Plone Store Is Now Open - Get Your Plone Gear!

Posted by PLONE.ORG on October 13, 2021 12:00 AM

The Plone Foundation proudly presents:

https://store.plone.org

The new shop for all Plone-related gear from T-shirts and hoodies to stickers and such is now open!

For now, you can get:

The shop is built on the TeeMill platform by the Plone Marketing Team, and we will be adding more Plone-themed gear over the coming weeks and months.

We made an extra effort to bring the site online before the Plone Conference 2021 online, so if you order your conference T-shirt now you might get it just in time before the conference opens 23rd October!

The Plone Foundation provides all items in the store without any markup, so the prices should be relatively low.

All clothing items are high-quality certified organic products, made from post-consumer remanufactured organic cotton in a renewable energy-powered factory, audited for a wide range of social and sustainability criteria. Read more at https://plone.teemill.com/the-journey.

The Plone Foundation Welcomes Two New Members

Posted by PLONE.ORG on October 12, 2021 12:00 AM

The Plone Foundation welcomes two new members after unanimous confirmation by the Foundation's Board of Directors on September 30, 2021.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Nicola Zambellos presenting at Plone conference

Nicola Zambello

Nicola has been developing with Plone since 2016 and has become an important contributor to the Volto project. He started RawMaterial, a Plone-based company, last year and will present his vision for a practical Green Web strategy at this year's conference. "I love Plone," he writes. "I feel at home with the community and I strongly want to support and invest in Plone, taking it to new horizons while maintaining what makes it so good."

Nicola will present one of the trainings at this year's conference; he lives in Ferrara, Italy.

Tiberiu IchimTiberiu Ichim

Tiberiu works with EauDeWeb and has been a member of the Plone community since 2004. Since 2019 he has been focused on Volto, and is a member of the Volto Core Developers team. He is the initial author or major contributor for several Volto add-ons, including volto-slate (alternative rich text editor) and volto-block-style (generic styling for Volto blocks).

Tiberiu lives in Oradea, Romania.


The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Nominations Open for Plone Foundation Board of Directors

Posted by PLONE.ORG on September 27, 2021 07:00 AM

If you have an interest in helping the governance of Plone, and particularly the energy and time to pitch in, please consider nominating yourself to serve on the Plone Foundation board of directors for 2021-2022.

Nomination Process

  1. Log in on plone.org and go here: 
    https://plone.org/foundation/meetings/membership/2021-membership-meeting/nominations
  2. Add a page there with your name in the title.
  3. For the body, discuss:
    • Who you are
    • Why you're interested
    • What you think you can add to the Plone Foundation
    • Most importantly, the name(s) of one or more Plone Foundation members who "second" your nomination
  4. Once ready, click "submit" in the workflow drop-down menu to get a reviewer to look at your nomination.
  5. Nominations will be accepted until October 22 2021, 23.59, UTC. The election will be conducted in conjunction with the annual meeting, which will take place during the Plone Conference 2021. All active members of the Plone Foundation will be eligible to vote.

About Board Membership

The Plone Foundation is a not-for-profit, public-benefit corporation with the mission to "promote and protect Plone". That has meant that the board is involved in:

  • protecting the trademark, copyrights and other intellectual property, including considering licensing and usage issues;
  • hiring the release manager;
  • working with sub-communities like Zope, Guillotina, and Volto
  • working with various committees, including marketing and membership;
  • handling "other stuff in the community" as needed
  • but not: directing Plone development. The board facilitates, but does not direct, the development of Plone itself.

While there's lots of work that happens online, much of the critical business of the board is conducted during video meetings every two weeks — typically, board meetings last about an hour to 90 minutes though occasionally they can run over to handle time-critical issues.  Please consider whether this fits your schedule, since missing more than an occasional meeting severely limits the ability of the board to reach quorum and conduct business.

Historically, board meetings have been organized to occur during daytime hours in America and evening hours in Europe, currently at Thursday nights, 19.00 UTC in northern hemisphere summer and 20.00 UTC in northern hemisphere winter. That can always change with new board members.

In addition, there is a board mailing list (private), where we discuss things in addition to the meetings.

This is a working board. Be ready to regularly take on and complete responsibilities for board business.

The board writes no code and makes no development decisions. It is much more concerned with marketing, budgets, fundraising, community process and intellectual property considerations.

You do not need to be a Foundation member to serve on the board (in fact, board leadership is an excellent way to become a Foundation member). All you need is to get an active Foundation member to second your nomination.

The Plone Foundation is interested in broadening the diversity of our leadership, with regards to gender, ethnicity, and geography.

If you have questions about the nomination process, contact the board: board@plone.org

Plone.org Is Getting a Facelift

Posted by PLONE.ORG on September 16, 2021 04:21 PM

Years have passed since the 2016 sprint at Penn State where a team of community members worked on a new theme and madly reorganized content on the Plone 5 version of plone.org. The site dates back to 2002 and the Plone 1 days, and the software and content had been upgraded in place over the years with only minor theme changes - to Plone 2 and 2.5, then Plone 3, then Plone 4, and finally Plone 4.3. It served us well, but because Plone 5 brought many changes, including a new out-of-the-box theme (Barceloneta), we mounted a major effort to refresh the design as well as upgrade the content and software.

What was new then is now looking old, and the marketing team has embarked on a modernization effort. The ultimate goal is to upgrade to Plone 6 and create a React-based theme using the new front end. But meanwhile we've been having a series of mini-sprints to improve what we have now.

Our first major initiative was to improve the News section, which holds an amazing collection of content. Browsing it can take you back in time - to the Plone 1.0 RC1 release announcement for example, or Alan's 2002 thoughts on what Plone should be, or approval of the Plone Foundation as a 501(c)(3) organization. It was already possible to browse news items by year, but we thought categorization by topic would also be useful. So we tagged every news item, and now you can browse news items by category. Fulvio Casali chronicled this effort in his 2020 Plone Conference talk Oh the Places We've Been!

A not very attractive display of news items and listings was another issue. So we sketched out a cleaner look, with a standardized lead image aspect ratio and a more useful byline. Then the more technically adventurous members of the marketing team (Norbert, Fulvio, Érico) strapped on helmets and figured out how to make changes to the site's theme. You are looking at our initial improvements, and there's more to come.

Our other major initiative is to move the contents of the plone.com site over to plone.org. Over the years plone.com became very difficult to maintain, so we have discontinued it. (Contact the marketing team if you need to retrieve any plone.com content.) With that in mind, we created a What is Plone? section on plone.org which is oriented towards the plone.com audience. It is also a place for us to describe all the pieces of the Plone ecosystem and how they fit together.

In addition to these bigger jobs we've been making lots of little improvements during our mini-sprints, including fixing bugs old and new as recorded on the plone.org issue tracker.

Would you like to help with this effort?

We'd love to have you!

  • Join our effort to promote Plone by publishing regular plone.org news items - successes, new developments, controversies, generally telling a broad audience what's happening in the Plone world
  • Do you have design skills? We don't and we need help with design improvements and eventually a new theme for Plone 6
  • If you are a theming wizard please help us modernize the site styles - more 2021 and less 2016
  • Show off Plone's built in search by creating a beautiful search results listing
  • Help us with our ongoing efforts to fix bugs and curate content
  • Help us migrate plone.org to Plone 6

Please contact the marketing team to get involved. Anyone with technical, design or content editor skills is welcome.

 

The Plone Foundation Welcomes Two New Members

Posted by PLONE.ORG on September 09, 2021 10:09 PM

The Plone Foundation welcomes two new members after unanimous confirmation by the Foundation's Board of Directors on September 2nd, 2021.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

cleber.jpg

Cléber J. Santos

With over 15 years of Plone experience, Cléber is an important member of the Brazilian Plone community. He was part of the team that helped organize three editions of the Plone Symposium South America and the Plone Conference in Brasilia.

Cléber lives in São Paulo, Brazil.

steve.jpgSteve Piercy

Contributor of many projects under the Pylons Project, including Pyramid, WebOb, Waitress, Colander, Peppercorn, and WebTest, Steve is also a constant presence in Plone community chats and forums. In recent years, he has collaborated with the Plone Documentation and Training materials teams and attended Plone Conferences as a speaker and a trainer.

In his spare time, Steve volunteers technical support for bicycling and environmental activist organizations. He lives in Eugene, Oregon, USA.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Plone Conference 2021 Online - Tickets for Sale Now!

Posted by PLONE.ORG on August 19, 2021 07:38 PM

The annual conference is a chance for the Plone community to come together to share new developments, success stories, and the future of the community. Taking place over 8 days, the conference will feature training, keynotes, talks, open spaces, sprints, and social activities.

This year’s conference will be entirely online, through the LoudSwarm virtual platform. No matter where you are, you can participate!

Plone Conference website now online

Behold https://2021.ploneconf.org/

The conference website is, of course, built with Plone.

Tickets are now available

Discounted tickets, available until September 30th, also a special price for developing countries.

Get your tickets here.

Submit your talk proposal

The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond.

There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences too.

Submit your talk proposal!

Stay tuned for more info

Follow Plone Conference at Twitter https://twitter.com/ploneconf

Follow Plone at https://twitter.com/plone

Questions for the August Steering Circle?

Posted by PLONE.ORG on August 11, 2021 11:55 AM

As described in the Foundation's July 2020 discussion of Plone governance, a series of Steering Circle meetings is being held to discuss our organizational structure and processes, and any hot topics of the moment. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be August 17th at 2:00 PM UTC. Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Plone 5.2.5 Released!

Posted by PLONE.ORG on August 06, 2021 07:10 PM

General notes

Plone 5.2.5 is a bug fix release of Plone 5.2. The release Manager for this version is Maurits van Rees.

Note: this is a fresh release. Installers are not ready yet but will be made available.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.5/versions.cfg.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

See https://plone.org/download/release-schedule for the planned release schedule.

Plone 5.2.5

Plone 5.2.5 is a release of Plone 5.2.

Download Plone 5.2.5

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.4/versions.cfg.

Useful links:

Some highlights of this release are:

  • Security fixes in AccessControl and Products.isurlinportal.
  • Security fixes from Products.PloneHotfix20210518 taken over in core.
  • Zope: 4.5.5 to 4.6.3
  • Products.CMFPlone: Add PLONE52MARKER Python marker.
  • plone.app.iterate: Add proper support for Dexterity folderish content.
  • plone.folder: restore webdav support.
  • plone.registry: Allow plone.schema.JSONField to be stored in registry (dictionary-like).
  • plone.namedfile: Cache stable image scales strongly.
  • plone.recipe.zope2instance: customize WSGI, profiling, python-env.
  • plone.restapi: JSONField, sub blocks, use_site_search_settings.
  • Lots of bugfixes, especially improving Python 3 compatibility.

For detailed changelog, go to https://plone.org/download/releases/5.2.5

Join Plone Chat, Now at Discord!

Posted by PLONE.ORG on August 05, 2021 03:13 PM

There are many ways to reach out to other Plone developers and users. The two most important platforms are the Community forum at https://community.plone.org/ and our chat platform, which has been moved from Gitter to Discord.

Online Chat at Discord

Discord is now the best way to chat with members of the friendly Plone community. There are various channels to choose from.

Join the Plone Discord.

Please remember that you will be chatting with volunteers. Read more about chat info.

Please do not use chat to ask for support. Support questions should be directed to either the volunteers in the Plone forum or commercial providers.

Guillotina and its contributors

Posted by PLONE.ORG on July 01, 2021 08:18 PM

Guillotina is a modern, asynchronous back end designed for building high-performance, horizontally scaling JavaScript applications.

Who are the Guillotina contributors?

First, let’s introduce our new contributors: Roger Boixader and Joan Antoni. They both work at Iskra which is a well-known company in the Plone community as they have been actively using and supporting Plone for years (for decades actually!).

Roger was kind enough to answer few questions about his involvement in Guillotina:

Q: As a developer, where are you coming from and what do you do? How much Python has been in your career until now?


Roger: My name is Roger Boixader Güell, I come from Berga, Barcelona, Catalunya and I am 28 years old. I studied computer engineering in Girona, Catalunya. I had never used Python before entering Iskra and for the first year I have mainly used JavaScript. My first steps with Python were with Django and then with Guillotina.


Q: Why is using Guillotina relevant in your technical context?

Roger: Because it is the main framework that we use in Iskra on the backend side.


Q: What is the thing you like the most in Guillotina?


Roger: The simplicity to create an application, with a few lines of code of Python or with a YAML file you can start a project easily
.

Q: What do you think should be improved in Guillotina?

Roger: Documentation and transactions.


Q: Have you been involved in an open source community before?

Roger: No, it's my first time.


Q: What would you expect from the Plone Community?

Roger: I think the Plone Community can help recruiting more Guillotina contributors, and the more contributors we have, the better Guillotina will get!

As we could expect, Guillotina attracts people who are not necessarily connected to the CMS world, but it is quite interesting to see Guillotina is considered as a valuable alternative to Django by young Python developers.

How is the core project managed?

The Guillotina project belongs to the Plone Foundation, just like the Plone project does, but it is obviously much younger and does not have (yet) all the Plone development and decision-making workflows.

So far, there is a periodic meeting every three weeks which allows us to discuss evolutions, important pull requests, and new use cases. Currently there are six participants in this meeting.

The repository is in the Plone Github organization.

When they join, Guillotina contributors do sign the Plone Contributor Agreement (as does any contributor to any Plone Foundation project), but they are not core contributors immediately (unlike Plone contributors). At the moment, there are three core contributors: Ramon Navarro Bosch, Nathan Van Gheem, and Jordi Massip. Their review and approval is needed to merge any pull request.

Apart from the core repository, Guillotina has a full ecosystem, managed in the Guillotinaweb GitHub organization; the most important elements are:

  • guillotina_elasticsearch
  • guillotina_gcloudstorage
  • guillotina_ldap
  • guillotina_react
  • guillotina_s3storage
  • guillotina_stripe
  • guillotina_volto

Each of these projects has an official manager (not necessarily a Guillotina core contributor).

Sprints

We organize a sprint in southern Europe every year when there is no global pandemic! We are also often involved in Volto sprints (and these happen usually in northern Europe, but also when there is no global pandemic).

You should join!

Security patch 20210518 version 1.5 released

Posted by PLONE.ORG on July 01, 2021 08:05 PM

This is a routine patch. There is no evidence that the issues fixed here are being used against any sites.

Version 1.5 of the hotfix is available from:

This version is a recommended upgrade for all users.

Zope users are advised to upgrade to Zope 4.6.1 or 5.2.1. If this is not possible, you can try this new version of the hotfix.

See the original 20210518 hotfix announcement

From the changelog:

1.5 (2021-06-28)

  • Fixed new XSS vulnerability in folder contents on Plone 5.0 and higher.
  • Added support for environment variable STRICT_TRAVERSE_CHECK.
    • Default value is 0, which means as strict as the code from version 1.4.
    • Value 1 is very strict, the same as the stricter code introduced in Zope 5.2.1 and now taken over in Zope 4.6.2. There are known issues in Plone with this, for example in the versions history view.
    • Value 2 means: try to be strict, but if this fails we show a warning and return the found object anyway. The idea would be to use this in development or production for a while, to see which code needs a fix.
  • Fix Remote Code Execution via traversal in expressions via string formatter. This is a variant of two earlier vulnerabilities in this hotfix. This was fixed in Zope 4.6.2, which takes over the already stricter code from Zope 5.2.1.

Note: we don't usually release another version almost six weeks after the original one, and three weeks after the previous version, and including a fix for a vulnerability which was only reported last week. However, this contains a fix for a close variant of one of the original vulnerabilities and needs a fix in the same code, so it seemed easiest for the security team and for Plone users who patch their sites to release a newer version.

Security patch 20210518 version 1.4 released

Posted by PLONE.ORG on June 14, 2021 04:11 PM

This is a routine patch. There is no evidence that the issues fixed here are being used against any sites.

Version 1.4 of the hotfix is available from:

This version is a recommended upgrade for all users.
Zope users are advised to upgrade to Zope 4.6.1 or 5.2.1. If this is not possible, you can try this new version of the hotfix.

See the original 20210518 hotfix announcement

From the changelog:

1.4 (2021-06-08)

  • Use safe html transform instead of escape for richtext diff. Otherwise the inline diff is not inline anymore.
    (Note: I forgot to add this to the changelog on PyPI/plone.org).
  • With PLONEHOTFIX20210518_NAMEDFILE_USE_DENYLIST=1 in the OS environment, use a denylist for determining which mimetypes can be displayed inline. By default we use an allowlist with the most used image types, plain text, and PDF. The denylist contains svg, javascript, and html, which have known cross site scripting possibilities.
  • By popular request, allow showing PDF files inline. Note: browser preference plays a part in what actually happens.
  • In untrusted path expressions with modules, check that each module is allowed. In the first version of the hotfix we disallowed modules that were available as a 'private' alias, for example random._itertools. But if random.itertools without underscore would have been available, it was still allowed, even though itertools has not been explicitly allowed. (itertools might be fine to allow, it is just an example.)

Security patch 20210518 version 1.4 released

Posted by PLONE.ORG on June 14, 2021 04:11 PM

This is a routine patch. There is no evidence that the issues fixed here are being used against any sites.

Version 1.4 of the hotfix is available from:

This version is a recommended upgrade for all users.
Zope users are advised to upgrade to Zope 4.6.1 or 5.2.1. If this is not possible, you can try this new version of the hotfix.

See the original 20210518 hotfix announcement

From the changelog:

1.4 (2021-06-08)

  • Use safe html transform instead of escape for richtext diff. Otherwise the inline diff is not inline anymore.
    (Note: I forgot to add this to the changelog on PyPI/plone.org).
  • With PLONEHOTFIX20210518_NAMEDFILE_USE_DENYLIST=1 in the OS environment, use a denylist for determining which mimetypes can be displayed inline. By default we use an allowlist with the most used image types, plain text, and PDF. The denylist contains svg, javascript, and html, which have known cross site scripting possibilities.
  • By popular request, allow showing PDF files inline. Note: browser preference plays a part in what actually happens.
  • In untrusted path expressions with modules, check that each module is allowed. In the first version of the hotfix we disallowed modules that were available as a 'private' alias, for example random._itertools. But if random.itertools without underscore would have been available, it was still allowed, even though itertools has not been explicitly allowed. (itertools might be fine to allow, it is just an example.)

Plone Powers a New Registration Portal

Posted by PLONE.ORG on June 04, 2021 03:55 PM

Challenge

The Open University of the University of Jyväskylä in Finland faced a daunting challenge: With a robust offering of international studies courses and a student audience beyond the usual set of university enrollees, registration needed to be faster, less complicated, and more flexible. For example :

  • Students from countries other than Finland needed a flexible authentication protocol
  • Students part-way through the registration process needed to get quickly back to the place where they had stopped
  • Coursework needed to be quickly and easily browsable, and purchase through university systems needed to be seamless
  • Course administrators needed a system where course listings could be updated with ease
  • Every step of the process needed to be loggable so that administrators could identify choke points and eliminate them

Solution

The development team at the University of Jyväskylä drew upon the strong core foundation of Plone to handle the all-essential data storage and management in a secure and accessible framework. Using Plone's modern RestAPI, the team was able to integrate, first, a wicked fast front end for managing data entry and browsing. Next, they used similar APIs to link to university systems for payment, user management, and the learning management system involved in course delivery. The ReactJS-based Volto front end (which will be standard on Plone 6) made dedicated and custom layouts very easy for the developers to create and even easier for administrators to add, edit, and organize.

One last integration using the RestAPI addressed the logging challenge. The team incorporated an open-source business process management tool, BPMN, into the platform to provide a window on every transaction, including how long each registration required and when and where registration ran into trouble.

Continuous improvement of the Plone framework has resulted in a fast, easy-to-use front end that editors love, deeply integrated with Plone's highly secure CMS backend built on Python and Zope. Based on Plone's unbeatable security record, the team knew they could trust the data to be safe. What's more, the framework's granular permission management was harnessed to allow teachers to access some areas, while students only got to see their own dashboard.

Feedback

User feedback has been overwhelmingly positive from both teachers and students at Open University. Problems with the identification and registration process have diminished greatly and, surprisingly, this had the side effect of improving the atmosphere in the courses.

More Information

Visit the English language version of the portal.

Read a technical blog post about the business process management integration.

Security patch released 20210518

Posted by PLONE.ORG on May 18, 2021 10:00 AM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers: CVE numbers have been assigned; see the individual pages.

Versions Affected: All supported Plone versions (4.3.20 and any earlier 4.3.x version, 5.2.4 and any earlier 5.x version).

Versions Not Affected: None.

Zope: Zope is also affected. See details below.

Nature of vulnerabilities:

The patch will address several security issues:

  • Remote Code Execution via traversal in expressions. Reported by David Miller.
  • Writing arbitrary files via docutils and Python Script. Reported by Calum Hutton.
  • Various information disclosures: mostly installation logs. Reported by Calum Hutton.
  • Stored XSS from file upload (svg, html). Reported separately by Emir Cüneyt Akkutlu and Tino Kautschke.
  • Reflected XSS in various spots. Reported by Calum Hutton.
  • XSS vulnerability in CMFDiffTool. Reported by Igor Margitich.
  • Stored XSS from user fullname. Reported by Tino Kautschke.
  • Blind SSRF via feedparser accessing an internal URL. Reported by Subodh Kumar Shree.
  • Server Side Request Forgery via event ical URL. Reported by MisakiKata and David Miller.
  • Server Side Request Forgery via lxml parser. Reported by MisakiKata and David Miller.

Thank you to all who contacted the Plone security team to report problems!

Version support: The hotfix is officially supported by the Plone security team on the following versions of Plone in accordance with the Plone version support policy: 4.3.20, 5.0.10, 5.1.7, 5.2.4. Previous versions, like 4.2, could be affected but have not been tested. On such old versions, the hotfix might have worse side effects than what it tries to fix.

The fixes included here will be incorporated into subsequent releases of Plone, so Plone 5.2.5 and greater should not require this hotfix. 

Warning: The hotfix has not been tested with Python 2.6. Originally Plone 4.3 was supported on Python 2.6, but since a few releases this is no longer the case since. It gets ever more difficult to test on Python 2.6. By now, you may have trouble installing any package with Python 2.6.

Zope support:

Zope is also affected. New versions for Zope and other packages are available. Upgrading to those is the recommended way.

If you cannot upgrade yet, you can try the Plone hotfix. It has not been tested on Zope only, but we try not to let the Plone-specific code get in the way, so it should be okay.

These vulnerabilities mentioned above are relevant for Zope:

  • Remote Code Execution via traversal in expressions via aliases.
    Fixes released in Zope 4.6 and 5.2.
  • Remote Code Execution via traversal in expressions (no aliases).
    Fixes released in Zope 4.6.1 and 5.2.1.
  • Various information disclosures.
    Fixes released in Products.PluggableAuthService 2.6.0, Products.GenericSetup 2.1.1, and Zope 4.5.5.
  • Reflected XSS in various spots.
    Fixes released in Products.CMFCore 2.5.1 and Products.PluggableAuthService 2.6.2.

The patch was released at 2021-05-18 15:00 UTC.

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the patch at 2021-05-18 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

Security vulnerability pre-announcement: 20210518

Posted by PLONE.ORG on May 05, 2021 11:00 PM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers not yet issued.

Versions Affected: All supported Plone versions: 4.3, 5.0, 5.1 and 5.2. Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Zope: Zope is also affected. A new version of Zope, or a related package, will be available around the time of the hotfix. If you cannot upgrade to this new version, you can try the Plone hotfix on your Zope site.

Nature of vulnerability: High, possible data exposure and remote code execution for already privileged users, various XSS (Cross Site Scripting) vulnerabilities.

The patch will be released at 2021-05-18 15:00 UTC.

Preparation

This is a pre-announcement of availability of this security fix. 

The security fix egg will be named Products.PloneHotfix20210518 and its version will be 1.0. Further installation instructions will be made available when the fix is released.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

Should you not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team will release the patch at 2021-05-18 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as Python packages that may be installed by editing a buildout configuration file and running buildout. For Plone 5.1 and lower they are also available as tarball-style archives that may be unpacked into the products folder of a buildout installation. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

World Plone Day 2021 - Over 50 Videos from 16 Countries

Posted by PLONE.ORG on May 04, 2021 03:34 PM

World Plone Day, held on April 28th 2021, was a worldwide 24-hour online streaming event. The goal was to promote and educate the public about the benefits of using Plone and being part of the Plone community.

The event was a massive success! The amazing Plone community produced 56 videos totaling 22 hours of content, now available on our YouTube channel. More than 50 speakers from 16 countries presented case studies, tech tips and community insights in 11 languages. This includes an introduction to Plone 6 in EN, DE, NL, CA, IT, PT-BR, FI, and soon JP, ES, and EU.

How to Access World Plone Day Videos

All content is available on the Plone YouTube channel. Play the videos on the World Plone Day 2021 playlist to relive the entire event.

Remember to subscribe while you are there!

Highlights

General Interest

Plone 6

Technical Talks

Case Studies

...and if you speak Italian

World Plone Day in Italy features 3 hours of topics!

World Plone Day Italy

Follow Plone and Join the Community!

Stay up to date with Plone and join us:

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released

Posted by PLONE.ORG on April 23, 2021 07:16 PM

Here are two important releases on the road to Plone 6:

  • Plone REST API 7 introduces a new link-integrity feature for blocks-based pages.
  • Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

This is a short version of a full blog post at Kitconcept site.

Plone REST API 7

Keeping links within a website intact is one of the core features of any Content Management System. In Plone, editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unique IDs (UUIDs) instead of relative or absolute paths when adding a link to another page.

Volto introduces blocks-based page layouts that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout information allows more complex page layouts. Though, because of this change, the existing portal-transforms mechanism that rewrites links to UUIDs did not work any longer.

Many participants joined to improve this feature in a community-wide effort:

  • Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020
  • Timo Stollenwerk from Kitconcept started to draft a possible solution and wrote the first prototype
  • Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy
  • Andrea Cecchi from RedTurtle joined the effort and refactored the resolveUID algorithm into a blocks transformer that made the resolveUID transformer more generic and flexible
  • After that, a first plone.restapi 7 alpha was released and entered a period of quality assurance and testing

After the first alpha release, the ResolveUID transformation was added to links and images. In addition, plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

Additionally, a "smart fields" concept allows integrators to mark a blocks field as searchableText.

Also, a new @contextnavigation endpoint was added that allows for local navigations.

Plone REST API 7 was included in Plone 5.2.4 release.

Volto 12

A new Volto configuration registry is the new central point to store and retrieve Volto configurations. The configuration registry ensures a setting only exists once.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Volto is the new React front end for Plone, communicating with the back end through the RestAPI.

Road to Plone 6

Plone REST API 7 and Volto 12 are two very important releases on the road to Plone 6. Next will come another Plone REST API 8 branch and release for Plone 6 supporting Python 3 only.

Volto will continue to move at a very high pace towards Plone 6. A series of Plone 6 “Micro-Sprints” will be organized to push things further.

Check out the Volto Roadmap on Github for more details.

Questions for the April Steering Circle?

Posted by PLONE.ORG on April 16, 2021 02:00 PM

As described in the Foundation's July discussion of Plone governance, a series of Steering Circle meetings is being held to discuss issues with our organizational structure and processes. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be April 20th at 3:00 PM UTC. Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released!

Posted by kitconcept GmbH on April 12, 2021 09:24 AM

stephen leonardi 5CH1TNfcZoo unsplash Photo by Stephen Leonardi on Unsplash

Two weeks ago we cut two important releases on the road to Plone 6. Plone REST API 7 introduces a new link integrity feature for blocks-based pages. Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

Plone REST API 7

Linking pages is one of the core idea of the world wide web. Keeping links within a website intact is therefore one of the core features that any Content Management System needs to provide.

One of the core features of Plone has always been that editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unigue IDs (UUIDs) instead of relative or absolute paths when adding a link to another page. Plone uses “portal transforms” internally to rewrite those links in RichText fields on save operations.

Plone 6 (aka Volto) introduces blocks-based page layouts, that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout allows more complex page layouts. Though, because of this change, the existing portal transforms mechanism that rewrites links to UUIDs did not work any longer.

volto blocks edit mode Blocks-based Volto page in edit mode (with kitconcept-blocks-grid)

Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020, since they needed that for a client project.

I started to draft a possible solution and wrote a first prototype. Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy.

Andrea Cecchi from RedTurtle joined our efforts and refactored the resolveUID algorithm into a blocks transformer, that we started to use for other use cases and that made the resolveUID transformer more generic and flexible.

After that, we cut a first plone.restapi 7 alpha release and entered a longer period of quality assurance and testing.

Two weeks ago, after we tested the new feature in multiple projects at kitconcept, I cut a final release of Plone REST API 7.

After the first alpha release, the ResolveUID transformation was added to links and images. In addition plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

We also added a “smart fields” concept that allows integrators to mark a blocks field as searchableText field.

A new @contextnavigation endpoint was added that allows for local navigations. We enhanced the navigation endpoint to expose an optional navigation title (“nav_title”) field.

Since the final 7 release, we cut six more releases and REST API 7 was included in Plone 5.2.4 release.

Volto 12

gaelle marcel vrkSVpOwchk unsplash Photo by Gaelle Marcel on Unsplash

Tiberiu and Victor worked on a new Volto configuration registry which is the new central point to store and retrieve Volto configurations. The configuration registry is a singleton that ensures a setting only exists once. With more an more Volto add-ons that started to depend on each other, we started to run into circular dependencies issues that we can avoid now with the new configuration registry.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Plone 6

Plone REST API 7 and Volto 12 were two very important releases on our road to Plone 6.

We plan to cut another Plone REST API 8 branch and release for Plone 6, which will support Python 3 only.

Volto will continue to move at very high pace towards Plone 6. We started to organize a series of Plone 6 “Micro-Sprints” to push things further.

Check out the Volto Roadmap on github for more details.

Plone Training Is Now Available Online

Posted by PLONE.ORG on April 09, 2021 10:08 PM

When the pandemic forced the annual Plone Conference online, the conference training classes went online too. We sure did miss seeing everyone in person, but there's a small silver lining. The online training sessions were all recorded and they have now been posted to the Plone YouTube channel. Now anyone can watch and re-watch them!

The following videos are available. All are about 4 hours long.

Mastering Plone 6 - Part 1

Philip Bauer, Instructor

Learn how to develop custom projects with Plone 6 and Volto, the new React-based frontend. Covers the core technologies involved in Plone 6 programming, including how to write your own add-on package and customize your Plone site by writing Python code and React components. Covers the first half of the material in training.plone.org/5/mastering-plone.

Mastering Plone 6 - Part 2

Philip Bauer, Instructor

A continuation of Part 1. Covers the second half of the material in training.plone.org/5/mastering-plone.

React and Volto - Part 1: React

Jakob Kahl and Alok Kumar, Instructors

Get started with React so you can create your own site using Volto. Learn the basis of React, Redux and React-Router. Covers the material in training.plone.org/5/react/index.html.

React and Volto - Part 2: Volto

Jakob Kahl and Alok Kumar, Instructors

Learn how to how to quickly bootstrap and customize a Volto project, and how create your own website based on Volto. Covers the material in training.plone.org/5/volto/index.html and training.plone.org/5/voltohandson/index.html.

Volto Add-ons - Part 1

Tiberiu Ichim and Víctor Fernández de Alba, Instructors

Learn how to develop Volto add-ons and other useful Volto patterns. Learn how to quickly develop a real world Volto add-on and how to structure your code to make it simple, reusable and provide extensible components. Covers the material in training.plone.org/5/voltoaddons/index.html.

Volto Add-ons - Part 2

Tiberiu Ichim and Víctor Fernández de Alba, Instructors

A continuation of Part 1. Covers the material in training.plone.org/5/voltoaddons/index.html.

Getting Started With Your Plone Site

David Bain, Instructor

Content management principles for Plone. A little background, principles & concepts, logging in & out, preferences and password management, folder management and the basic publication workflow.

Pyramid

Steve Piercy, Instructor

A hands-on, quick tutorial covering "a little about a lot". Practical introductions to the most common features. Fun, fast-paced, and targeted to newcomers to both Python web application development and the Pyramid web framework.

Implementing user epics with BPMN

Posted by Asko Soukka on March 21, 2021 12:00 PM

Many user stories start simple. For example: “A user self registers into a course”. The final version, of course, tend to be more like: “A new user, after being verified to be an authentic real person, creates their new user account, immediately forgets and resets their new password, finally logs into the system, finds the course again, completes payment for the registration, and is then eventually enrolled into the course.” This is no longer a user story, but a series of stories supposed to be completed in a specific sequence. Let’s call these user epics.

An epic is a sequence a user stories

Individual user stories, by their definition, should be relatively straightforward to implement. For example, a stereotypical user story may require interaction with specific user interface element to start the story, following one or more views to gather the required user input to continue the story, finally ending with user’s goal for the story. Where did the user come from, or where does the user continue afterwards, is simply not part of the story. On purpose.

But user epics are different.

Epics must know the consequences of users' actions

An implementation of a user epic must know where the user came from, and must be able to decide, which stories the user should be guided to complete next. Epic should also cover the consequences of user failing to complete a story, or completely abandoning their fate. Therefore implementing an epic is not only about managing state, but also about programming the often complex business logic. Both being among the least fun parts of software development.

As an insult to injury, less and less are users doing their work alone, more and more they are being accompanied by various forms of automation. So, not only should an implemented epic be aware of the actions of its user, but it should also be able to react on known results or side effects of automation – anything from synchronous function calls to asynchronously executed software robots.

With just code, implementations for this kind of epics could get dirty and hard to understand, fast.

Epics must know the consequences of users' actions

Offloading state and logic to BPMN engine

It’s not a coincidence that my example user epics above are drawn as BPMN diagrams. That’s what I have been doing for the last couple of months in our latest project. We had to modernize a process, where users were found to require a lot of hand-holding to be able complete all the required steps and in the right order.

Also, with the previous implementation, when users got lost in the middle of their epics, or an unexpected system error blocked them, it was often surprisingly hard to figure out what really had happened.

This time all this should be different.

The active user task may be anywhere in the current process tree

Whenever a user starts a new epic, a new BPMN modeled process is being started. The model may be as long and complex as needed. It may even be a composition of multiple nested models, resulting in an arbitrarily deep process tree when executed. But this is not a problem, because this is exactly what dedicated BPMN process execution engines have been built for.

The fun part: With a BPMN engine executing our user epics directly from their BPMN models, we don’t really need to “implement” them at all. Instead, we:

  1. Implement user interface routes and views for individual user tasks.
  2. Make the views to read their preconditions from the process engine state.
  3. Eventually complete the task at the engine (with user input when required).
  4. Poll the engine for the next available task and redirect the user there.

Now our user task views are standalone, with no magical dependencies between them to code and maintain.

Plone, Volto and Camunda FTW

I would not be writing this, unless all the above could not be implemented by using open source software only:

  • Because every application seems to need a CMS (sooner or later), we build on top of Plone, the open source CMS we trust. Thanks to its powerful and complete REST API, we can do all the integrations we need with just HTTP requests – no need to know about Plone internals.

  • For modern user experience, we use Volto – ReactJS based web experience on top of Plone REST API (with Razzle based server side rendering support). Every part of Volto is easily customizable and extensible. For example, custom server side routes, like the ones we need to authorize and proxy calls to BPMN engine, are common and well documented use case.

  • Finally, for the BPMN engine, we choose Camunda BPM Community Edition. While the open source version of Camunda is probably missing countless convenient features of their enterprise offering, its BPMN engine and REST API come uncompromised.

Thanks to all above, it’s not a (technical) problem at all that users tend to forget their passwords just after sign up. Now the process engine remembers were they came from, and what they were supposed to do, just after resetting their password one more time…

With a reusable sub process, password change can part of any epic

Philip Bauer: Growing Pains: PosKeyErrors and Other Malaises

Posted by Maurits van Rees on January 05, 2021 10:33 AM

This talk is about the issues that you face when your project grows, the code base grows, the database grows, the problems grow. This is about the causes and some of the remedies.

Symptom 1: huge database

Cause 1: a huge number of revisions or versions.

Remedies:

  • Remove all versions and pack the database. When you migrate to a new Plone version, and you ask your client, they will usually be okay with this.
  • Manage or limit revisions. Easiest is to use collective.revisionmanager for this. Especially, revisions may have been left behind for content that no longer exists. You can easily remove it with this tool.
  • Disable versioning of Files. It is disabled by default, but maybe someone has switched it on.
  • Enable manual versioning instead of automatic. Then the editor needs to check a box when they make a major change that they want to be able to rollback.

Cause 2: no packing.

Remedy: just pack it. Use the zeopack script, which part of plone.recipe.zeoserver. Add a cronjob for this, weekly seems best for most sites.

Cause 3: unused content.

Remedy: delete it. You have to find it first. Of course no code can tell you which content is safe to delete. You could use statistics.py from collective.migrationhelpers to get an idea of where which content is.

Cause 4: the SearchableText index is huge

Remedies:

  • Use solr or elasticsearch and possibly remove the SearchableText index.
  • Don't index files. They are converted to text, but this may not be needed for your site.

Cause 5: large blobs For example, plone.de had a Linux iso image, which was huge.

Remedies:

  • Limit the upload size. You could do this in nginx/apache. Archetypes had something, you can likely do this in Dexterity too.
  • Get stats and remove or replace too large items.

Cause 6: aborted uploads (rare)

Remedy: check IAnnotations(portal).get('file_upload_map').

Symptom 2: slow site

Cause 1: unneeded full renders of content

Remedy: use Python in page templates. By default, page templates use path expressions like this: tal:define="foo context/foo". But this tries to render  foo as html if possible. Use foo python:context.foo instead.

Cause 2: wake up many objects

Remedies:

  • Always try to use brains and metadata. The difference is huge, also with Dexterity.
  • Listing 3000 brains: 0.2 seconds
  • Listing 3000 objects: 2 seconds
  • Same is true for Volto when you use the search-endpoint with fullobjects.

Of course most page templates in Plone will not list thousands of objects, but will be paginated. Still: just use brains, they are so much tastier.

Cause 3: no caching

Remedies:

  • Switch on the built-in caching
  • Add varnish
  • Manage the zeocache (that is a bit of science, ask the community)
  • Use memoize in your code.

Cause 4: hardware

Remedies:

  • Don't be cheap.
  • Buy enough ram to keep the database in memory.
  • Remember that your consulting time probably costs more than buying better hardware would.

Cause 5: slow code

Remedies:

  • Learn and use profiling. A very handy toy for that is py-spy. Sample use: sudo py-spy top --pid 12345
  • Do not call methods multiple times from templates. Call them once, store the result, and use this.

Cause 6: slow data sources

Remedies:

  • decouple, for example using redis or celery
  • Use your choice of async implementations
  • Use lazyloading of images if they come from outside of your Plone site.

Symptom 3: conflict errors

Conflict errors happen when two requests work at the same time and both change the same object. This is complicated, but Zope and the ZODB have built-in conflict resolution.

Cause 1: conflict resolving is not enabled. The zeoserver needs access to the same code that your zeoclient has, otherwise conflicts cannot be resolved and the transaction will be aborted.

Remedy: add all application code to the zeoserver:

[zeoserver]
eggs = ${buildout:eggs}

Cause 2: long running requests change data

Remedies:

  • Prevent writes.
  • If it takes long, do intermediate commits when possible.
  • Prevent crossfire: disable cronjobs and editors when a long request needs to run.
  • Use async. Talk to Asko about that probably.

Symptom 4: PosKeyErrors

Cause 1: missing blobs

Remedies:

  • Copy all blobs of course.
  • Use experimental.gracefulblobmissing in development to create dummy blobs where needed.
  • Find and delete afflicted content in a browser view.
  • There can be cases when you have two zeoclients and the syncing does not work well. Talk to Alessandro about that.

Symptom 5: broken data

Now for the really interesting part. These are errors like:

ModuleNotFoundError
AttributeError
ImportError
PostKeyError
BrokenObject

I could read you my whole blog post about zodb debugging.

Cause 1: code to unpickle som data is missing

Remedies:

  • Ignore the errors, if normal operation still works, and the site only has to stay up for a limited time, because zeopack probably also fails.
  • Fix it with a rename_dict. See zest.zodbupdate for some examples that are actually really useful. [Thanks! MvR]
  • Work around it with an alias_module patch, like plone.app.upgrade does in several cases. Then imports can work again.
  • Find out what and where broken objects are and then fix or remove them safely. Use zodbverify.

Steps for the last one:

  • Call bin/zodbverify -f var/filestorage/Data.fs to get all broken objects.
  • Pick one error type at a time, with an oid (object id) that has a problem.
  • Call bin/zodbverify -f var/filestorage/Data.fs -o <oid> -D to inspect one object and find out where it is referenced.
  • For the extra options, you should use the branch from my pull request, which I still have not finished yet, but it runs fine.
  • Remove or fix the object.
  • Important: make notes, write upgrade steps, keep the terminal log, because you will forget it and need it again.

To remove or fix the object, it helps to start the actual Plone site with some special zodbverify sauce:

./bin/instance zodbverify -f var/filestorage/Data.fs -o <oid> -D

Then you can use your debugging skills to try and fix things. Note that after you fixed it, you need to commit the changes explicitly:

import transaction
transaction.commit()

Note that the bad object is still in the database, until you pack it.

Frequent culprits are IntIds and Relations, especially if you migrated from Archetypes to Dexterity. Using collective.relationhelpers you can clean this up:

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()

Symptom 6: bad code

Unreadable, untested, unused, undocumented, unmaintained, complicated, overly complex, too much code. If you can convince a client to not want a feature because they will only use it once, that is a win. Every line of code that is not written, is a good line of code.

Maik Derstappen: Plone 6 Theming with Diazo

Posted by Maurits van Rees on December 10, 2020 10:16 PM

How does Plone theming look in classic UI?

  • html5 theme plus a mapping configuration
  • deploying themes as ZIP-file for shared hosting is possible
  • With Diazo you can map any Plone html to a static theme layout.

Separating frontend and backend theme. Don't reinvent the backend views! You could theme the backend, so for content editors and admins, but it looks fine, not needed. You should focus on the frontend layout for visitors. To use the default backend layout you can include backend.xml in your rules, with some conditions.

Diazo is not for everything! If the backend markup differs from what you need, do not try to solve it with Diazo or XSLT. Instead, fix the backend templates directly, most likely with a z3c.jbot override.

You can use theme fragments or browser views to add new templates. The theme fragments can also be used as tiles in Mosaic.

I like using SASS mixins. Say you have a <div class="main-wrapper container">. Cleaner would be <div class="main-wrapper">. You can do this with mixins with @include in your selector.

For more information on Diazo, see https://diazo.org.

New features in Plone 6

From the backend we get Bootstrap 5 compatible html. Result is that Bootstrap themes are easier to integrate in Plone 6.

You have custom CSS in the theming control panel, for small changes. This actually sneaked into Plone 5.2.2 as well.

We have simplified Diazo rules.

Create a theme with plonecli:

  • pip install plonecli
  • plonecli create addon plonetheme.yourtheme
  • cd plonetheme.yourtheme
  • plonecli add theme
  • plonecli build

Theme from this presentation will be published as collective.bunapisicuto when it is ready for you to inspect.

Stefan Antonelli: Plone 6 Theming from Scratch

Posted by Maurits van Rees on December 10, 2020 05:08 PM

How to create a theme for Plone 6. Quite easy, because the templates use Bootstrap 5 classes. We build a theme from scratch, no Barceloneta, no Diazo.

First step is to create an empty plone_addon package with plonecli or mr.bob. For the questions you can answer: use Plone 5.2.1. We will switch later. My theme is plonetheme.munich.

I recommend to cleanup the standard package a bit. I remove tests, constraints for Plone 4 and 5. Check it out in the commits.

Now switch to extend Plone 6 and run the buildout.

You can add theme structure with a bob template, but I prefer creating my own.

Some interesting files:

  • package.json lists various tasks, especially the watch task.
  • In the theme manifest.cfg we more or less disable Diazo by emptying the rules line.
  • The compiled CSS and JavaScript are registered in registry.xml.

You can compile SASS to CSS using npm or yarn. Do yarn install in the top of your package. Later, with yarn dist you make it ready for production.

After these steps Plone is partially broken, or at least ugly. I do some basic fixes and it looks better.

For templates that you need to change you add z3c.jbot overrides. Personally I always kick out the "search in section" checkbox.

I don't like columns, but for this example I kept them. In most cases I need just one column. Plus maybe a side bar for portlets, but portlets must die.

With plonetheme.tokyo everything is Bootstrap, no columns, so no portlets, really fully responsive. This was the package where we built on Barceloneta Plone 5.2 and introduced lots of template overrides to put in Bootstrap. For Plone 6 we can just remove the overrides.

What about the toolbar? Yes, we dropped it. We bring editing features and navigation together. This is now a few feature: collective.sidebar. It is only one template to override. It works for Plone 5.2 at the moment, and I may work on it for Plone 6 during the sprints.

Question: is TTW still a viable path?

Answer: I like to concentrate on one path. I am not an expert in TTW theming. I switched to file system, except really small customizations. For small CSS customizations there is a field in the theming control panel.

Stefan Antonelli and Peter Holzer: Modernize Plone’s Classic UI

Posted by Maurits van Rees on December 10, 2020 04:02 PM

What was new in Plone 5? We had beautiful new theme: Barceloneta. Diazo theming by default. We switched to CSS compilation with less.

During the Tokyo conference Stefan thought up Tokyo Theme. Clean responsive theme for Plone 5. Tons of overrides to tackle problems in Plone 5. Issue with navigation and editing on mobile we solved with collective.sidebar.

We had community discussions, especially during several Plone events. Everyone tried to use Bootstrap (components). First idea: map variables from Barceloneta to Bootstrap, because they have similar ideas using different terms.

We have PLIP to modernize markup in templates, and another PLIP to modernize the default theme: Barceloneta LTS. Forms using z3c.form are already using the new classes.

Make things easier: UI, development. Creating a modern UI for the web is complex. You need to support different devices, responsiveness. In Bootstrap there are patterns for most useful things.

Developer perspective: expect one way to do things. Developers should not have to worry about design. When busy in the backend, you should focus on Python, not on it looking nice and shiny. Don't think about markup, just use components. The good news: there is documentation. The Bootstrap documentation is our documentation.

What is new in Plone 6?

  • Volto is the default UI.
  • There still is the Classic UI with Barceloneta look and feel, but updated.
  • No TTW (through the web) theming.
  • But there is a textarea to add some simple CSS (already in 5.2.3).
  • Some CSS variables may be changeable TTW.
  • Finally jQuery 3

Bootstrap is still the most popular front-end framework. Well documented, tested and maintained. It is so easy to create stuff, I enjoy it a lot.

What is new in Bootstrap 5?

  • Improved overall look and feel.
  • Updated and extended the color system
  • Custom properties: css variables
  • SVG Icon library
  • Pure javascript
  • Dropped IE10 and IE11 support
  • Bootstrap 5 is currently alpha 3.
  • See https://v5.getbootstrap.com

Features: what do we get from these changes?

  • Core templates use Bootstrap 5 markup. Instead of overrides in plonetheme.tokyo, we have lots of branches for the actual packages.
  • All major templates have been touched already.
  • For the current state, see the unofficial demo at https://classic.plone.de
  • The Bootstrap documentation has lots of snippets than you can copy.
  • You don't need much more CSS on top of it if you paste most examples. We added a little for own components, like navigation.

Barceloneta appearance is fully customizable. It is basically on opinionated set of bootstrap variables. Every aspect can be changed with variables: colors, fonts, sizes, spacings, grid gutters, etc. There are overall properties, like shadowed, rounded, gradients. Just turn on or off.

Theming workflow. plonetheme.barceloneta will also be published as npm package. bobtemplates.plone will have a template for the new theming workflow. You can do quick and dirty customizations through the CSS overrides field in the theme controlpanel.

Diazo will still be there, will work as before. Some optimizations in the rules.xml to make content are customizations easier.

How to deal with icons? What if you want to change the content type icons? Used to be hard. Now we come up with the idea of an icon resolver. We decided to use the bootstrap icons. Icons are registered via GenericSetup, for example with a record name plone.icon.alarm pointing to an SVG. You can then override this in your own GS profile.

Example icon use:

<tal:icon replace="structure python:icons.tag('love', tag_class='custom-class', tag_alt='foobar')" />

You get get back an inline SVG or an image tag.

Note: all z3c form widgets in Plone are now in plone.app.z3cform, and not scattered over lots of packages.

We will restart our weekly Plone 6 Classic UI sprints, starting Januari 13 2021, 10:00 (UTC+1).

Keynote: The User Experience - Editing Composite Pages in Plone 6 and Beyond

Posted by Maurits van Rees on December 10, 2020 02:55 PM

It may be a surprise to non-technical people to learn that pages created in Volto are not currently interoperable with traditional Plone's page editing. If you think about it, the reason becomes obvious. Volto, like Mosaic, creates tiled layouts, and like Mosaic it stores page data in special fields for the individual blocks and their layout. Neither Volto nor Mosaic pages are editable in TinyMCE, which expects just one rich text field.

Is this divergence between sites created in Volto and sites created in traditional Plone a problem? It does make it harder to describe what Plone is, and it might mean that there is no way to mix both approaches - for instance when part of a larger site is available as a Volto-based sub-site. Would it be possible to have one tool and one representation for tiled layouts so that we can avoid this divergence? Is there some other solution? Is it even a problem? Will Plone 6 be backwards compatible with Plone 5 and include a smooth upgrade path?

We will tackle these questions in this strategic panel discussion, moderated by Sally Kleinfeldt. Panelists will include Paul Roeland, Philip Bauer, Timo Stollenwerk, Victor Fernandez de Alba, and Eric Steele.

First, Philip has a message from Max Jacob who is very ill with pancreatic cancer and may not survive this weekend. He wants to thank the Plone community for what they allowed him to do: like organize the German Plone Konferenz. Thanks for all the friendships. Such a pity that this is happening now, he wanted to jump into only doing Plone for the next few years, due to changes at his job, and looked forward to that.

Classic, Mosaic and Volto Pages

We have Classic, Mosaic and Volto Pages. They have different internal representations and are not compatible. Is this a problem? Is there a solution? Is one tool, one representation possible? If we really need three, how to position?

Paul: for me as user this presents a problem. When do you switch over your site? We would like to not write 700 pages from scratch, again, like we did for previous composite pages.

Timo: We migrated quite a few large projects from classic Plone to Volto. One of those had collective.cover (other composite page system). Problem in general with such systems, is that they are pretty specific. They solve specific use cases and come from different eras. After any migration, it will not look the same any more. Whatever you do: the page will initially look ugly. So you put a lot of effort into migration, but then have to put manual effort into every page anyway. It can help: you at least have a start. We created a system where we migrated overview pages, and editors could click to migrate other pages one at a time.

Philip: We have code to migrate from non-folderish to folderish content types. There will be code to migrate to dexterity site root on Plone 6. We can make sure to migrate any standard content types. Mosaic is another story. So for pages you would at least have the text available. Maybe only visible for editors to pick and choose from. You may lose portlets, unless they get implemented in Volto.

Timo: When you go to Plone 6 and do a redesign at the same time, then you can jump on Volto. Otherwise you could stay at Classic Plone for now. There will be an overlap period.

Victor: For Mosaic you could dump all tiles into html and insert it in a block.

How to have a big Classic site with a subsite made in Volto?

Victor: Definitely doable, though we have not done this ourselves.

Sally: But what happens when an editor in Classic goes to the Volto subsite? The Volto page would not be editable then, right?

Philip: You should not offer this. I see no upside, no use case. Split them into separate applications, with shared authentication maybe.

Paul: Use case: large site with several departments. The marketing department may want snazzy new Volto things.

Timo: Just create another site then.

Cost and benefit of upgrading big Classic Site to Plone 6

So you just had a big migration to Plone5, and now what would you get for going to Plone 6.

Philip: We have this discussion every major upgrade. Communicate every upgrade as a relaunch. The relaunch is the reason for the upgrade. "There is a new version so you need to upgrade" does not fly for my clients.

Timo: We became a very developer oriented community, and every develop understands the need and benefits. We should really get back to giving more value at major releases, so clients really want to upgrade themselves. Plone releases should sell themselves.

Eric: It looks like we think Plone 6 + Volto is a costly upgrade with lots of benefits. For Classic 5 to Classic 6 the upgrade is not costly.

Alex Limi's vision for Deco: are we there yet?

Was this fulfilled by Mosaic? Volto? Something still needed?

Paul: Not quite, but slowly getting there. For me it would be Volto, plus some power features that Eau de Web (EEA) adds.

Timo: I think we went beyond what Limi envisioned.

Eric and Victor: What we have seen from Volto, is pretty close to what Alex wanted.

Victor: We gave the users powerful tools, so beware of them.

Philip: Partly yes. Volto is close, and it is for normal users.

Now some questions from users.

Migrations

Philip: Plone 4 to 5.2 was three migrations in one. Plone 6 is less of a problem.

Eric: 5.2 had a lot of backend migrations. A split between backend and frontend with plone.restapi in between makes things easier.

[The question on multiple variations of Volto, especially editors, went a bit too fast for me to write intelligible notes down.]

What's the future of using CT's/behaviors in Plone to design information architecture?

With Volto the trend seems mixing/adding 40 different blocks for every page.

Timo: Blocks are definitely the way to work. But the underlying power of content types and behaviors still exists.

Philip: We need blocks that represent a field or a behavior. That is unavoidable.

Next steps

Timo: We plan to have an open space on page compositions and Volto, and want to sprint on it.

Paul: Good if there is a longer term vision. I would rather have more power that a Site Admin can lock down, than having to choose between three different versions. I don't want choice stress.

Lightning Talks Wednesday

Posted by Maurits van Rees on December 09, 2020 06:21 PM

Lukas Guziel: Continuous Deployment

CD means deploying code automatically. It saves time, reduces human error. It Gitlab you can add gitlib-ci.yml and configure it. Include a base template that you use in multiple projects. End result can be a site that the customer can test.

Erico Andrei: World Plone Day 2021

We are a global community. Almost 300 people from 36 countries are at this online conference. World Plone Day is an annual Plone event. Next year of course online. April 28th 2021.

We want to stream 24 hours live on our YouTube channel. Showcase Plone. Technical talk, use cases, interviews, demo.

It should not all be in English, please use your own language. Talk to your local community.

Please help and join. See https://plone.org/events/wpd

Andreas Jung: collective.contentsync2

Syncing content between Plone sites through plone.restapi. It is a behavior. You have a source Plone site and one or more target Plone sites. You need Plone 5.2 under Python 3.

  • Create a dedicated user account with global role Editor.
  • Configure on the content sync control panel.
  • Automatically creates two content rules to sync content when added or modified.
  • You can enable it on all content types, also Folders.

See https://github.com/collective/collective.contentsync2

Philip Bauer: Why relations are weird

These packages have a part in relations:

  • zc.relation: abstract relation catalog
  • z3c.relationfield: fields and values on objects
  • plone.app.relationfield: converters from field to widget and vv
  • plone.app.z3cform: widgets
  • mockup: actual widget UI

In a schema use a RelationChoice field with vocabulary plone.app.vocabularies.Catalog, and set pattern directives.

It is not straightforward. So I wrote collective.relationhelpers.

See https://github.com/collective/collective.relationhelpers

Maybe use uuid instead of all this code.

[About ten people in the chat want to merge this package into plone.api. Actually, see this issue.]

Christopher Lozinski: Simple JSON Schema GUIs

Create a JSON schema, automatically generate the UI. Search for basic JSON editor library. He shows JSON in ZODB, so you can browse it, if I understood correctly.🙂

Eric Brehault: Second Guessing the Single-Page-App Pattern

Posted by Maurits van Rees on December 09, 2020 05:42 PM

SPA (Single Page App) is about providing an entire app by exposing a single physical web page containing an enormous javascript bundle. It breaks the original web paradigm in many ways. Surprisingly enough, we invest a lot of efforts to mimic the regular web behaviour.

Isn’t it time for modern frontend to reconsider the SPA approach?

[Note: Eric presented by using a projector to show his slides on a black Plone conference T-shirt. :-)]

Why are we doing this? Originally we always requested a whole page and this was considered slow. But we have good bandwidth now. And if you don't have good bandwidth, the super big bundle is not good either.

With SPA we try very hard to bring back the original working of the page, especially the browser history, being able to browse and then share the link to the current page.

To mitigate problems, we created an enormous stack. And we deny the complexity. New tools create new problems, even when their individual creators does not see the complexity.

"SPA isn't stable or efficient." But there is no way back. For example, you cannot create Google Docs with server side rendered pages. Web 2.0 is 15 years old. It is still about content.

SPA is separation of concern, which is a good principle. But we mix the browser layer (how you get and view the page) and the content layer (the page content).

It seems a take it or leave it situation: either use SPA or don't. What do we want? We want proportionate complexity. Do we need 100 percent SPA?

You can use micro components, see for example the demo of Maik Derstappen in the lightning talks on Monday, using Svelte. Micro frontend is bigger than that. It is a part of the application, that you develop separately. For example, you could do the Plone Sharing page like this.

Can we compile each page separately? Then each page is an app.

ES6 native support would be interesting. Combine with HTTP/2 and you need no bundles. Bundling is the most brutal thing ever. Horrible. Get rid of it.

Respect the layers. SPAs are monolithic. Break them down.

We should have a generic browser layer, common to many different use cases, for example for logging in. I don't want to code that, but plug it. Second step: push this layer to the browsers themselves.

Asko Soukka: Deploying Plone and Volto, the Hard Way

Posted by Maurits van Rees on December 09, 2020 04:22 PM

Here are the slides.

How about building Plone without buildout? Running Plone on Python 3 without WSGI? Deploying Plone and Volto with containers without Docker? Building all this in re-usable and safe manner in sandbox with restricted network access with Nix? Welcome to hear about our hipster setup where we lock, build and configure Plone deployments with Nix, insist to keep ZServer running on Python 3 for the love's sake, build software deployments into standalone tarball archives, and run them with Nomad – the simple on-premises-friendly alternative for K8S.

  • The easy, documented way: buildout, WSGI, Docker (if you need containers), Registry.
  • Our way: pip, TxZServer, Nomad, Nix

When you use a container infrastructure, you have multiple containers for running a Plone site, for example zeo clients, zeo servers, load balancer. Nomad helps there, and is much simpler than Kubernetes. We have one job file to rule them all: task groups, instance count, update policy, server resources, volumen mounts, tasks, consul services, vault secrets, environment variables, exec artifacts.

Nomad has "isolated fork / exec driver". No docker image needed. We have a Nix-built artifact, a tarball that we extract in the root of the container.

With Nix, you get 100 percent reproducible artifacts. Production equals development. You have a full dependency graph. The result is a standalone tarball, perhaps 100 MB. Disadvantage is that there are no conventions, no metadata, no shared layers, no documentation. It needs learning and practice. Well, some documentation now: https://nixos.org and https://nix.dev, partially made by people that were using Plone previously.

Some ugly parts from Nix:

  • Every language has their own Nix-conventions
  • dependency generator ecosystem is comples
  • cyclic dependencies are not supported
  • no storage device is big enough for /nix/store

Our (legacy) approach for Plone 5.2.1 without Buildout and with pip:

  • generated requirements.txt with buildout
  • create Python env with pip and nix
  • use pip-branch of z3c.autoinclude
  • disabled <includeDependencies />
  • generate instance skeleton with nix
  • forked plone.recipe.zope2instance

Plone 6 without Buildout should be pip-installable out of the box, but that is hear-say.

We use TxZServer in production, so ZServer using Twisted.

Nicola Zambello: Theming Volto without SemanticUI: Is It Possible?

Posted by Maurits van Rees on December 09, 2020 02:22 PM

We will walk through the process of building a product for Italian Public Administrations using a bootstrap-based theme. I'm presenting io-comune, RedTurtle's first product based on Volto and the strategies we used. We will see the possibilities in Volto for theming without SemanticUI, using bootstrap and sass and what are the next ideas we could work on.

Scenario:

  • We wanted to adopt Volto in our new project.
  • We needed to include Bootstrap.
  • Volto uses SemanticUI instead.
  • Two such frameworks will conflict, for example fighting over the same selector.

We tried. We tried harder. A cheap approach did not seem possible, so we looked for a sane one.

A new theme: pastanaga-cms-ui. Load only the CSS needed for Volto admin UI, see Volto PR 970. And public-ui for public pages. In your src/theme.js do not import the css/less from semantic-ui, but the pastanaga-cms-ui. In theme theme.config also use pastanaga-cms-ui. Also razzle-config.

You should normalize your base style, for example:

body.cms-ui {
  .public-ui {
    font-size: 18px;
  }
}

and wrap your components with .public-ui.

Building a product:

  • Base common package for every customer: https://github.com/RedTurtle/design-volto-theme
  • New intermediate layer for SemanticUI
  • New config layer for razzle/customizations
  • Template for actual projects: design-volto-kit, with a Yeoman generator: create-italia-volto-app

Lightning talks Monday

Posted by Maurits van Rees on December 09, 2020 10:19 AM

Alec Mitchell: WYSIWYG problems be gone

A new add-on to vastly* improve your content editing experience.

(* size of improvement may vary, no warranty implied. The following is (un)paid free software promotional content)

Adding images to a document is so hard! At least nine steps! (Difficulty be exaggerated for marketing purposes). Why can't you drop an image in? You can, with our new, super special add-on krcw.tinymce_imagedrop.

Can you drop two? Yes!

Can you drop more than two? No, because browsers are weird.

But it fails gracefully.

Steve Piercy: Deform and friends

How I learned to stop worrying and love web forms. We must have a good interface, data structure, validation, security. Deform (form library), colander (de/serialization), peppercorn (data structure), bootstrap forms for design. We have a looooong list of widgets. In deform 3.0 we will use bootstrap 5. See https://github.com/pylons/deform

Christopher Lozinski: Forest Wiki

The Forest Wiki is a modern version of Zope 2. Biggest difference: it uses Pyramid's security and views. Modern JavaScript enabled ZMI: reorder, sort, rename, etc. Both WYSIWYG and MarkDown pages. Advanced types like JSON, CoffeeScript, pug. Pug is the leading template engine for Node.

Jens Klein: RelStorage

Plone relational database backend storage. It is a drop-in replacement for FileStorage of ZEO. You can use PostgreSQL, MySQL, Oracle, SQLite. It has been around for about 13 years, grown old, but in recent years development has picked up, driven by Jason Madden, including Python 3 support. It is much more performant. Latest release 3.4.0 is form October 2020.

PostgreSQL is the cloud database, kind of industry standard, well supported by all big cloud providers. Easy to install in Docker.

Advantages of RelStorage: fast, parallel commits, better concurrency, shorter locks. Optimized per process caching. Blobs in database. Optionally you can use it in history free mode. You lose the Undo functionality, but you don't need to pack so often.

plone.recipe.zope2instance supports it with the rel-storage option.

You can use additional client side caches, shared between all threads of a process.

With the zodbconvert tool you convert from ZEO to RelStorage, or the other way around, including converting blobs if needed.

ZODB keeps old transactions, so packing is needed, even in history free mode. RelStorage has a fast zodbpack.

Blobs:

  • RelStorage 3.x is Python 3 only and runs with Plone 5.2+. Here, blobs should be stored in RelStorage.

  • RelStorage 2.x is for Plone 5.0, 5.1, and blobs should **not**  be stored in the database, except for Oracle backends, otherwise you should still use a shared blobs filesystem directory.

    System Message: WARNING/2 (<string>, line 69); backlink

    Inline strong start-string without end-string.

I use RelStorage today for all my live deployments. I have used it since version 1.6 with Plone 4.3 and never had problems. Always blazing fast. Dev/ops and sysadmins love it: it is a standard solution, nothing special, just works.

Maik Derstappen: Add-on catalog for Plone

We want to bring back an add-on catalog for Plone. You can look on PyPI, but it is hard to find packages.

We worked on a tool for this. You can search on named, filter on Plone versions and add-on types.

Components:

We only aggregate packages that have classifier Plone :: Framework. We will probably work on this during the sprints and are happy to onboard you.

David Bain: Plone and Webflow

Both platforms are for building websites, but they approach things in different ways. I hope this may inspire. Keep in mind the motivation of the two platforms, which may account for some strengths and weaknesses.

Webflow is visual web design, less content management. Strong design tools. Designer friendly layout tools. You can design a page with what you could call blocks.

Plone is enterprise content management, focus on security. Linking to an attachment is standard, where it is tricky in Webflow. Forms are way more flexible.

We have also built a website in Webflow and based it on Plone.

Miu Razvan: Volto grid block

  • Created by Eau de Web team
  • Dependencies: Volto blocks form
  • Similar component: Volto columns block
  • Use it to organize other blocks.
  • Demo showing lots of configuration options, including for different screen sizes
  • See https://github.com/eea/volto-grid-block

Maik Derstappen: Custom elements

Custom elements are an extension to normal native html elements, for example <flag-icon>.

The promise of web components: write once, use anywhere. See https://custom-elements-everywhere.com/

How do you use this in Plone? Use plonecli add svelte_app to create a small app. Run yarn. Install in Plone add-ons control panel. Edit a page. Replace html source with <my-svelte-app />. And your component is there and working. The size is less than five  kilobytes.

Tiberiu Ichim: volto-slate

volto-slate is a drop-in replacement for the standard rich text editor in Volto. Volto turns an HTML document into a modern document.

Why another text editor instead of improving the existing one?

  • With Slate we get a better plugin framework. Plugins are just wrappers around the editor. The standard Draft.js is meant to be integrated directly by an application, no concept of plugins out-of-the-box.
  • Slate has simple DOM-like storage for its values, making it easier to render the ersult.

Current status:

  • No migrations of any kind.
  • Right now not possible to completely remove or replace Draftjs out of Volto.

Alin Voinea: Volto Dexterity Schema and Layout Editor

Posted by Maurits van Rees on December 08, 2020 05:40 PM

Through the Web Dexterity Content-Types with Schema Editor and Blocks Layout Editor

How do we define content types schemas in Plone?

  • TTW schema editor
  • GenericSetup profile
  • Behaviors, schemas in Python

Why do we need them, we have Volto blocks, right? You still need metadata, a title, etcetera. Certainly for larger institutions you need a structure, a schema. Volto itself has schema-based components.

Layout editor. Blocks have properties, like a placeholder, a position. You can type text in a block: "Published on date by author". Then select "date" and link this text to the published date metadata, and select "author" and link it to the author. Save this as a layout for a content type. You can export this to a JSONField in a custom behavior, so you can save it in version control for production.

List of add-ons and other packages that make Volto awesome: https://github.com/collective/awesome-volto

Jens Klein: Performance, Profiling, Power-Consumption

Posted by Maurits van Rees on December 08, 2020 04:27 PM

I want to focus on Python performance, so not caching or database performance.

Tools:

  • py-spy: Overall mix of the whole live application, top-like.
  • repoze.profile: WSGI middleware, slows down application. Profile single request and analyse its call stack by count, call time, etc.
  • dis: disassembler for Python at the bytecode level.

Improvements Plone 5.2.0-5.2.3:

  • Avoided early providedBy calls
  • __getattr__ early exit on common attributes
  • zope.interface: some functions are called hundreds of thousands of times when you reindex an index, so a tiny improvement helps a lot. I found various places that could use improvements, and that landed in the package, together with memory improvements by Jason Madden.

Live demo. I call py-spy with sudo because I need to connect to an existing process id.

Future Todo's:

  • plone.restapi has optimization potential, all navigation related, but currently it still supports even Plone 4.3. This will likely wait for a 5.2-only or Python3-only branch.
  • plone.registry is called too often
  • Use python: expressions in all page templates. They are way faster than standard Tales expressions.
  • More introspection.
  • Move more logic from page templates to Python code

Advice: start introspecting the performance of your application.

Alex Clark: The State of Pillow

Posted by Maurits van Rees on December 08, 2020 04:26 PM

The Plone Conference account tweeted that a State of Plone talk would be awesome and that the Plone community missed me. I miss the Plone community too, so I am here.

I will state it clearly: Pillow would not exist if not for Plone.

In July 2010 I announced Pillow as "friendly" fork of PIL. The mailing thread and future answers are interesting to read.

Some history:

  • 1991: Python 0.9.1
  • 1995: PIL started
  • 1998: Zope
  • 1999: Zope2
  • 2000: Python 2.0 with distutils
  • 2001: Plone
  • 2005: Buildout
  • 2006: I attended my first Plone Conference, in Washington
  • 2006: setuptools was born

PIL had an issue, or Plone had an issue with PIL:

  • PIL used distutils.
  • Plone 3.2 used Buildout and setuptools
  • PIL was not installable in Buildout and setuptools
  • Specific problem: import Image could mean the Image module from PIL, or the Image module from Zope.

Various ways of repackaging PIL started, for example PILwoTk. You can still find various PIL derivatives at https://dist.plone.org/thirdparty/

PIL 1.1.6 from 2006 is still the last version on PyPI. I got maintainership of this page this year, actually. Pillow 1.0 is basically the same, except that it uses setuptools. This worked in buildout. I was happy.

Couple years, nothing really interesting happened. But some contributors came along. Pillow 2.0.0 in March 2013 had Python 3 support.

An important milestone in 2015: we added release notes.

Release schedule: in the beginning of every quarter.

We get some money from Tidelift for maintenance.

Fred van Dijk: collective.collectionfilter as a Light-weight Faceted Navigation or a 'compare' Console

Posted by Maurits van Rees on December 07, 2020 09:10 PM

I want to talk about some categorisation and classification options in Plone, next to the folder structure.

Faceted navigation: drill down on 'facets' when you search for items. It was popularized by online shopping. Facets in Plone for developers is: whatever is in the ZCatalog, and for users: what you can search on in Collections. Gold standard is eea.facetednavigation, developed for the European Environment Agency. Examples: EEA, and on two sites by Zest: Vaquums and Minaraad, where it replaces the standard search.

collective.collectionfilter is a much leaner, meaner, but also more limited version of faceted navigation. Demo with standard Plone News Items with some tags (also known as categories, also known as Subject). Add a Collection that filters on News Items. Now add collection filter portlets.

eea.facetednavigation takes over your complete page. In an action you enable or disable it.

Now a demo of collectionfilter in SGBP, a documentation website for water management planning in Belgium/Flanders. The customer wanted to take some graphs and compare them. We did that with collectionfilter and collective.classifiers. With the last one we added structured categories: one for water basins and one for parameters of the graphs. Now we use collectionfilter to query a parameter and show the graphs for all water basins.

You can adapt several things in the collectionfilter UI, for example change how search options are displayed. This is documented, but took me a while to get right.

Collectionfilter also works with Mosaic, because the portlets are also mapped to tiles.

Asko Soukka: Plone and Volto in a Jamstack project

Posted by Maurits van Rees on December 07, 2020 05:54 PM

Here are the slides.

I am a software architect at University of Jyväskylä. I have been using Plone since 2004 and GatsbyJS since 2018. The university wanted one student information management system to rule them all, but... every organisation shall do their own integrations, using granular REST API with deep JSON responses. And there should be branded study guides, which we crafted with GatsbyJS. But this was not enough for the Open University part. They really needed a CMS.

We use Plone 5.2, Volto, GatsbyJS, and have 6000 html pages, times two languages, out of which 760 are Volto pages. With Plone we could extend content types without needing to do any coding, in the content types field editor. In volto we added auto-complete widgets with custom vocabularies. On the GatsbyJs side, we query the connected pages with GraphQL. We render Volto layouts with React components, rendering individual blocks.

Why did we choose GatsbyJs? It is a ReactJS-based site generator. Being static, it is very fast. You can use multiple sources as input, using a plugin architecture. Data lookup is done with GraphQL. It is easy to get started, with comprehensive documentation.

I mentored two Google Summer of Code projects for the gatsby-source-plone plugin. It supports default types and most TTW types, also Volto blocks. You can do incremental updates by modification date, so it is really fast.

Not everything is easy. The full "GatsbyJs experience" requires practice. You want to replace inline images and links with GatsbyJs images and links, replace file links with direct downloads.

Using @plone/volto as dependency to render blocks seemed like a good idea, but it required webpack overrides to be impartable, and could not be used for images and links.

The ugly parts of GatsbyJs:

  • The GraphQL source plugin cannot cache.
  • The build may take hours, and gigabytes of memory.
  • The build result in readonly.
  • For me it is hard to follow GatsbyJs development, especially individual plugins, because they use a monorepo.

Editors can work on the site during the day, and then wee rebuild the result during the night

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'xxx@xxx.de': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'xx@xxx.de': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone xxx@xxx.de'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

moniviestin workflow menu

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

moniviestin workflow

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

moniviestin example

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

moniviestin content rule

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

moniviestin rpa poll

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

  1. Video author selects “Publish with subtitles”.
  2. Plone content rule calls Jenkins webhook to schedule a new robot.
  3. Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
  4. Jenkins schedules a new job to poll pending ASR jobs.
  5. Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
  6. Plone content rule notifies the video author with email that automatically generated subtitles are now available.
  7. ???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

study guide tabs

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

hasura view

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

hasura relationships

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

volto contents

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

  1. Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

plone dexterity editor , 2) Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

volto sisu connector

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

study guide edit

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

study guide image

Something old, something new and something blue.

The perfect match.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.

  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.

  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.

  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.


All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.


If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM
Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM
I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.