Planet Plone - Where Developers And Integrators Write

Buschenschanksprint 2022 Report

Posted by PLONE.ORG on June 17, 2022 01:27 PM

This was the Buschenschanksprint 2022

stand up

After two years of working from home, sprinting remotely, and meeting via video conferences the Buschenschanksprint was for many the first occasion to see each other in person.

And meeting in person was amazing. We had many discussions, demos, and stand-ups which seemed just more effective than what we had over video conferences. Not that video conferences are ineffective, but it was high time to meet in person again.

The beautiful landscape of the "Südsteirische Weinstraße", the Buschenschankjause, and a glass of wine or two in the evening was also a fine thing to have.

We were at most 18 people directly at the sprint and two remote sprinters. The weather was fine, we could work outside and inside, on the terrace or in the arbor, depending on preference.

The space was big enough and the internet connection absolutely sufficient. We had some people in tents, 8 people in the "Ferienhaus List" and the rest at the Apartment "Sonnenberg".

For dinner, we went to nearby Buschenschenken, to a Restaurant, and did some BBQ ourselves.

This year Buschenschanksprint was a strategic sprint for the second time. We had a lot of plans for progressing Plone and related projects, and we did achieve a lot.

group pic

Progress during the sprint

Here is but a short list of headlines of sprint achievements.

To read the progress in full detail, check the community forum post: https://community.plone.org/t/this-was-the-buschenschanksprint-2022/15347

  • Barceloneta LTS improvements
  • Webpack Module Federation
  • Documentation and Training
  • Migrate to Volto
  • Work was done on an automatic migration path from Classic UI to Volto.
  • Relations in Volto
  • Mosaic
  • Datagridfield
  • Images
  • Mxdev/Mxmake
  • YAFOWIL
  • plone.api
  • Control Panel User Group Membership in Volto
  • Various other topics
  • Future of Classic Plone discussion

buschen3.jpeg

Participants of the Buschenschanksprint

  • Johannes Raggam (Organiser)
  • Alessandro Pisa
  • Manuel Reinhardt
  • Robert Niederreiter
  • Jens Klein
  • Maik Derstappen
  • Philipp Auersperg
  • Guido Stevens
  • Peter Mathis
  • Christine Baumgartner
  • Peter Holzer
  • Markus Hilbert
  • Alexander Pilz
  • Maurits van Rees
  • Philip Bauer
  • Georg Bernhard
  • Leo Auersperg
  • Katja Süss
  • Steve Piercy (Remote)
  • Fred van Dijk (Remote)

That was the Buschenschanksprint. We're already looking forward to the next one!

Plone Conference 2022 - Call for Papers Open!

Posted by PLONE.ORG on June 15, 2022 06:00 PM

Plone Conference 2022 takes place in Namur, Belgium on October 10-16, 2022. We have now opened call for papers, looking for all kinds of different talks related to Plone.

The topics can range from Plone, Zope, Volto, and Guillotina to Python and Pyramid or from fancy JavaScript to cool case studies and beyond. Project management, process improvement, open-source, design, and community aspects are valued talk topics. Plone 6 cases are of much interest this year.

There are many kinds of talk slots, from 5 min lightning talks to 30 and 45 minutes long, and you can target your talk to different audiences.

If you have any questions, please contact conf@plone.org. No matter if you are an experienced speaker or a beginner, we are happy to help you!

Submit your talk!

More info

ITONBOARD - IT community onboarding for young people

Posted by PLONE.ORG on May 30, 2022 01:46 PM

At the Plone conference in Ferrara 2019, after a talk by Christine Baumgartner and her two interns Ilvy and Nils, a bunch of companies discussed the future of on-boarding of young people into IT. Three companies, Abstract, Interaktiv and Klein & Partner, followed up on the idea and looked for funding to improve IT on-boarding with focus on free and open source software companies.

Erasmus-Logo.pngAfter further meetings we decided to apply for an Erasmus Plus project of the European Commissison. We expanded the scope from internships only to the whole process of onboarding. With Talentbrücke (Cologne) we found a partner in the field of the transition from school to job; with International Formation Center (Madrid) a company specialized in designing and managing training and development programs.

At the Plone Tagung in Dresden, very shortly before the first COVID19 lockdowns in Europe, we prepared the first application, but it failed. After adjusting our strategy to a broader community, and with remote work in focus, the second application of our project got accepted. In March 2021 we had our kick off!

logo-ITONBOARD-hires-plain.pngITONBOARD's goal is to introduce young students to career paths in open source web development. With that goal in mind we are developing five distinct products to guide students from a gamified experience to first-hand experience in schools; from e-learning opportunities to internship options. Our target groups are students, teachers and IT companies. The project is still ongoing. Some of our five outcomes are almost finished, others are still in development:
  1. an Escape Game for vocational orientation for students from 12 to 16 years old. Within the game, typical job situations and knowledge about IT professions can be experienced and acquired. Professions in web development as well as the concept of open source are presented to the players.
  2. a guide for teachers with a detailed concept for a project week at schools on the topic of IT career orientation. The central and associated learning content and materials were designed during the course of the project so far.
  3. online courses (MOOCs), which can be used as an alternative method of vocational orientation. The approach within e-learning formats is completely different from the classic and more theoretical information materials (flyers, texts, etc.).
  4. recommendations and guidelines for blended remote internships with short on-site phases in companies whose focus is on community-based, open source culture. Remote work and learning have had deep roots in the IT industry, and especially in the open source area, for decades.
  5. an internship exchange facilitating the arrangement of internships between interns and companies. For example, without the Plone Community the ITONBOARD project would never have happened. In the beginning the community was the initial backbone to discuss ideas.

We got in contact with other companies and collected their needs and experiences with interns, got lots of implicit information, personal feelings, and learned about failures and problems. In the Plone community we know each other and have many experts in different fields. Thus, many community members were keen to help with interviews, participating at workshops or reviewing material. Thanks to everyone donating their valuable time!

From the broader Plone community we interviewed Alexander Pilz (Syslab), Timo Stollenwerk (Kitconcept), Nejc Zupan (Niteo Web and project advisory board), Paul Roland (former president of the Plone Foundation and project advisory board) and Chrissy Wainwright (Sixfeetup and former president of the Plone Foundation) about their experience with internships and remote work. Furthermore we got valuable scientific insight into the field by our project advisory board member Univ.-Prof. Dr. Bernd Gössling (Uni Innsbruck). This gave us plenty of input for the blended remote internship guidelines and beyond!

20210825_143505(1).jpgFrom the community Katja Süss (Rohberg), Paul Roeland and Fred van Dijk (Zest Software) joined us at our multiplier event at the Schatzbergalm and provided valuable, decades long experience in remote work, teaching, diversity in IT and in general, and communities.

Additionally, Paul and Fred gave us a short interview. We also got support from other FOSS communities, here with a special mention of the OpenEDX community! And thanks to everyone we forgot to mention here, helping us with the project by sharing minds!

The project will continue for almost another year. We appreciate any feedback! If you want to learn more about ITONBOARD visit our website, send us an email or talk to Stefania, Christine, Jörg or Jens.

Plone Welcomes Students for Google Summer of Code 2022

Posted by PLONE.ORG on May 29, 2022 06:37 PM

For 2022, the Plone Foundation has been granted *six* Google Summer of Code student project slots.

Google Summer of Code is a global program focused on bringing more student developers into open source software development. Students work with an open source organization on a months-long programming project starting over the summer.

After careful selection of the many project proposals presented, we are pleased to announce that the following students will be working on various aspects of Plone:

Read more about their projects

To our new students: welcome to the Plone community – we wish you a great learning experience over the coming months. Congratulations!

Mentoring these students will be:

To our mentors: thank you for introducing new developers to open source and our community!

Beethoven Sprint Recap - Progress Towards Plone 6

Posted by PLONE.ORG on May 20, 2022 08:38 PM

The most effective (and fun) ways to move things forward in Open Source development are sprints - after years of remote sprinting due to COVID, we were finally able to hold a sprint in two locations simultaneously, with more than 25 people around the community gathering together.

The Beethoven Sprint was held in Bonn, Germany, and Bucharest, Romania.

Sprint Topics

These were the topics that we worked on:

Check the official sprint page for more details.

Bonn

Main goals were achieved, and we are now closer than ever to make Plone 6 a reality. The main blocker, Slate in core, is on its way. Other nice-to-have features to be included in Plone 6 include:

  • "base_view" default for custom view-less content types
  • initial iteration of the core Block Style Wrapper
  • extensive work on the new images srcset for frontends
  • initial design of the infrastructure to support them on the backend
  • rethought and refactored user and groups control panel
  • migrations, classic-to-Slate, draftJS-to-Slate, classic-to-Volto in general
  • DnD support for nested structures
  • Schema driven stock Blocks support
  • Containers for backend and frontend, nightly for buildout.coredev
  • and fixed tons of bugs... 

just to name the most important... the PR list is huge! 

Thanks to: Andrea Cecchi, Balazs Ree, Bernd Wolber, Érico Andrei, Fred van Dijk, Ion Lizarazu, Jakob Kahl, Jens Klein, Katja Süss, Maurits van Rees, Mikel Larreategi, Philip Bauer, Piero Nicolli, Rob Gietema, Steffen Ring, Thomas Kindermann, Timo Stollenwerk, and Victor Fernandez de Alba

Bonn

Bucharest

The Plone Beethoven 2022 sprint "two legs" summary from Romania:

  • The first Plone sprint ever organized in Romania!
  • Around 12 attendees in Bucharest and a few more remote from all over Romania
  • Thanks to Maik Derstappen for joining us so we can call it an international sprint 🙂
  • Facts: youngest Plone contributor for the sprint: 18 year old Ionuț Dobriceanu. Contributions merged!

Achievements:

  • volto-slate editor being merged into Plone6/Volto core advanced integration
  • "URL management"/"Aliases" control panel and contextual implemented for both backend and frontend
  • "Content rules" control panel and contextual implemented for the backend (API endpoints)
  • "Link Integrity for content objects" for Plone6 frontend implemented
  • ... and a couple of other generic issues solved!

Thanks to: Valentina Bălan, Laszlo Cseh, Maik Derstappen, Ionuț Dobriceanu, Valentin Dumitru, Krisztina Elekes, Alec Ghica, Andrei Grigore, David Ichim, Tiberiu Ichim, Răzvan Miu, Daniela Mormocea, Iulian Petcheși, and Alin Voinea.

Bucharest

Also, check out The Plone Newsroom podcast episode 8, where Philip, Fred and Victor talk about the Beethoven sprint, Plone 6 and Volto 16: https://plone.org/news/2022/the-plone-newsroom-podcast-episode-08

Upcoming sprints

Do you want to join the effort or meet new people? The friendly Plone community welcomes you to sprints this May:

5.2.8 Released

Posted by PLONE.ORG on May 14, 2022 08:15 PM

Some highlights of this release

  • Zope: Enhance cookie support. For details, see issue 1010
    For more changes see Change log — Zope documentation 4.6 documentation
  • waitress is updated to version 2.1.1 to mitigate a vulnerability in that package. As waitress no longer supports Python versions less than 3.7 it is not advised to run Plone 5.2 on Python 2.7 or 3.6 any longer, even though they are still supported by Plone itself. You get an older waitress version then. If you must use an old Python version, please switch to a different WSGI server. See the recommendations in the Zope documentation.
  • WARNING: the new waitress 2.1.1 does seem to suffer from a possible race condition leading to the process quitting. If you are affected by this, you can downgrade to 2.1.0 (which has a known security vulnerability, as mentioned above) or use a different WSGI server.
  • plone.app.linkintegrity: Track link integrity of referenced PDFs and other site objects in IFRAME SRC references.
  • plone.outputfilters: Resolve UIDs in SRC attribute of of SOURCE and IFRAME elements.
  • plone.app.querystring: Add lazy attribute to vocabularies to prevent fetching any results.
  • plone.schema: Use indent in json.dumps to make JSON readable in the widget.

Download Plone 5.2.8

The Plone Newsroom Podcast Episode #08

Posted by PLONE.ORG on May 12, 2022 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. They cover technical and non-technical topics including Plone, the Plone community, and whatever else they come up with to keep us informed!

Episode #08 on 12 May 2022 was all about Plone sprints - sprints in general and the recent Beethoven sprint in particular, which took place simultaneously in Bonn and Bucharest. It focused on the Volto 16 frontend that is coming in Plone 6. Philip, Fred and special guest Victor Fernandez de Alba discussed what got accomplished and what is planned after Volto 16. Go to the Newsroom page to view other episodes.

World Plone Day 2022 a Great Success

Posted by PLONE.ORG on April 29, 2022 10:15 PM

World Plone Day is an opportunity for members of the Plone community to celebrate and promote their favorite open source CMS. The 2022 event took place on April 27th and featured in-person events plus streaming and recorded content. There were:

  • 58 talks on a wide range of topics from beginner to advanced
  • 20 hours of video in 10 languages from 14 countries
  • 6 local events - in Italy, Brazil, India, Switzerland, and 2 in Romania

There were talks from long-time community members and from people who are relative newcomers. Here are a few highlights for a general audience:

Here are some highlights for developers:

And here are a few of the many talks in languages other than English:

You can find all the World Plone Day videos, and much more on the PloneCMS YouTube channel and the WPD 2022 playlist.

Progress on the New plone.org Built on Plone 6

Posted by PLONE.ORG on April 13, 2022 02:00 PM

The New Plone.org Will Be Plone 6

At the 2021 Plone Conference, the marketing team presented a plan was for how to improve and renew plone.org, the one and only place for Plone-related information:

The main ideas at the time were:

  • Plone.org will be migrated to Plone 6
  • Content needs refreshing, especially the frontpage and main landing pages
  • Visual theme will be out-of-the-box Plone 6 with Volto frontend, using available add-ons (many of which we expect will be added as core features)
  • Goal is to publish the new site by World Plone Day 2022 (April).

Great Progress at Sprints Plone org example

Since December 2022, we have organized 4 monthly sprints, working on:

  • Content and navigation structure
  • Other content work
  • Migration from Plone 5 to Plone 6
  • Visual design
  • Translation, default page content, add-ons
  • Plone 6 documentation
  • Volto improvements
  • Volto Blocks and installation
  • And more

We have updated the original plan to include a new visual theme and additional features that will eventually be available for others to use. A more ambitious scope has moved the release date somewhat further this year.

Special Thanks

The Plone Marketing and Communications Team is organizing the effort and working on the content overhaul, but we also want to express special thanks to:

  • Philip Bauer, from Starzel.de - Full content migration from Plone 5 to Plone 6, including the transition to Volto blocks
  • Érico Andrei, from Kitconcept - Foundation Member content type, installation, deployment, content type renewal, etc.
  • Steve Piercy and Katja Süss - Documentation effort resulting in new Plone 6 docs and organization around the work
  • Massimo Azzolini, Irene Capatti, Massimo Weigert, and Gianantonio Vecelli, from Giallocobalto - For massive content and structure redesign, visual design and layout, and UX design
  • Stefano Marchetti, Andrea Cecchi, Giulia Ghisini, and Andrea Baglioni, from Redturtle - For the new site's buildout which includes needed Volto blocks, curated add-ons, visual theme, and more: https://github.com/collective/volto-plone-org
  • Kim Nguyen, Brian Davis - For lessons learned from the previous renewal, content and structure discussion, theming mockups, etc. 
  • Victor Fernandez de Alba and Jakob Kahl - For Volto knowledge
  • And everyone else who has pitched in ideas, discussions, and questions!

This effort would not be possible without help from the awesome Plone community!

Want to Join the Effort?

The plone.org work is not done - our next sprint will be organized in May 2022. Feel free to join! There will be an event page, but meanwhile join the plone.org Discord channel for discussion. By the next sprint we hope to have a fully running site with migrated content available online for testing and content work.

If you have any questions or comments, please contact marketing@plone.org.

Stay tuned for more plone.org discussions and presentations at World Plone Day on April 27th 2022.

The Plone Newsroom Podcast Episode #07

Posted by PLONE.ORG on April 13, 2022 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. They cover technical and non-technical topics including Plone, the Plone community, and whatever else they come up with to keep us informed!

Episode #07 on 13 April 2022 featured news about the progress towards the Plone 6 Classic release, updates about Volto 15 development, and information about events (GSoC, World Plone Day, sprints (Zope, Beethoven, Buschenschank), and add-ons (collective.easyform, collective.honeypot and captcha). Go to the Newsroom page to view other episodes.

Plone 6.0.0a4 Released

Posted by PLONE.ORG on April 09, 2022 10:00 AM

Download and installation: https://plone.org/download/releases/6.0.0a4

The fourth alpha release of Plone 6 contains various fixes and improvements. Read more about the upcoming Plone 6.

Thank you to everyone involved!

Highlights

Changes since 6.0.0a3:

  • Use zc.buildout 3.0.0rc3 and setuptools 62.0.0 by default.
  • Update waitress to version 2.1.1 to mitigate a vulnerability in that package.
  • Zope 5.5.1: Enhance cookie support.
  • plone.staticresources: The big one: Updated JavaScript for Plone Classic, using ES6 modules. No more through-the-web compiling of JavaScript. See PLIP 3211.
  • Products.CMFPlone:
    • Remove RequireJS.
    • Remove default resource jQuery. It is added to the global namespace via the bundle.
    • Remove support for conditional comments in script and style tags. It's not supported since IE10.
    • Remove dependency on mockup. Mockup is now a npm package only and as such a dependency of plone.staticresources.
    • New resource registry to simplify CSS/JS registration.
    • Only "bundles" are registered - support of "resources" and "bundle resources" is removed.
    • Removed TTW compilation of bundles via r.js and less.js.
    • Property merge_with is no longer needed in HTTP/2 times and merging here unsupported.
    • Unique key for delivery is based on hash of bundle file, last_compilation property is deprecated.
    • PLIP #3279: Implement modern images scales. Add huge (1600px), great (1200px), larger (1000px), teaser (600px). Amend preview and mini (remove height constraint).
    • Add TinyMCE template plugin to the plugins vocabulary.
    • Add TinyMCE alignment classes, to avoid style usage.
  • plone.volto is now a dependency of the Plone package.
  • PLIP 2780: Move features of collective.dexteritytextindexer to core.
  • plone.app.dexterity:
    • Remove JavaScript from this package and move it to Mockup.
    • Modeleditor: Use pat-code-editor from Patternslib instead ACE. Make the model editing form usable without JavaScript. Allow editing the form even with XML errors to be able to fix the problem.
  • plone.recipe.zope2instance: by default do not create a temporary storage.
  • plone.scale: Removed deprecated factory argument from scale method.
  • plone.app.linkintegrity: Track link integrity of referenced PDFs and other site objects in IFRAME SRC references.
  • plone.outputfilters: Resolve UIDs in SRC= attribute of of SOURCE and IFRAME elements.
  • plone.app.querystring: Add lazy attribute to vocabularies to prevent fetching any results.
  • plone.app.theming:
    • Deactivate copy button and modal in theming control panel.
    • Remove all thememapper functionality from theming control panel, including Inspect/Modify theme and the Preview.
  • plone.app.users: Show unfiltered member fields for manager in user profile page.
  • plone.app.widgets:
    • Remove implicit dependency on Mockup. Mockup is no longer a Python package, only an npm package.
    • Update datetime pattern options for Patternslib pat-date-picker/pat-datetime-picker.
  • plone.autoform:
    • Fixes for latest z3c.form.
    • Reimplementation of ObjectSubForm and ISubformFactory, backported from older z3c.form.
  • plone.app.z3cform:
    • Use better types for inputs.
    • Use browser native date and datetime-local input together with patternslib date-picker.
    • Implement TimeWidget which renders <input type="time" />.
    • Use pat-validation in forms.
    • Fixed for latest z3c.form
  • plone.z3cform: compatibility with latest z3c.form.
  • plone.namedfile: Register AnnotationStorage as IImageScaleStorage multi adapter, both from plone.scale. Use this adapter in our scaling functions when we store or get an image scale.
  • Products.PlonePAS: Add separate GenericSetup profile to switch the Zope root /acl_usersto use a simple cookie login form. Useful when Zope root login and logout need to synchronize authentication state between multiple plugins, which is not possible with HTTP Basic authentication.
  • plone.app.layout:
    • Restructure global sections and searchbox markup for mobile navigation as offcanvas sidebar.
    • LiveSearch with support for images in search results.
  • plonetheme.barceloneta: sticky footer.

Note that changes may be mentioned only once, even when they involve multiple packages.

Plone 6

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Try Plone 6 Alpha 4

Download and installation:
https://plone.org/download/releases/6.0.0a4

Read more at the community forum:
https://community.plone.org/t/plone-6-0-0a4-released/15126

Volto 15.0.0 Released

Posted by PLONE.ORG on March 08, 2022 10:10 AM

The Plone community, in particular the Volto team, is happy to announce that Volto 15 is ready and shipped!

Volto is Plone's snappy, modern React front end powered by the RestAPI, and the default for Plone 6.

Volto 15.0.0

New feature highlights

Volto grid block

  • Add `cookiesExpire` value to config to control the cookie expiration @giuliaghisini
  • DatetimeWidget 'noPastDates' option: Take widgetOptions?.pattern_options?.noPastDates of backend schema into account. @ksuess
  • Add a new type of filter facet for the Search block. Heavily refactor some searchblock internals. @tiberiuichim
  • Add date range facet to the search block @robgietema
  • Introduce the new `BUILD_DIR` runtime environment variable to direct the build to run in a specific location, different than `build` folder. @sneridagh
  • Handle redirect permanent calls from the backend in the frontend (e.g. when changing the short name) @robgietema
  • Added id widget to manage short name @robgietema
  • Refactor language synchronizer. Remove it from the React tree, integrate it into the Api Redux middleware @sneridagh
  • Add blocks rendering in Event and NewsItem views (rel plone.volto#32) @nzambello @ksuess
  • Complete Basque translation @erral
  • Complete Spanish translation @erral

Breaking changes

  • Upgrade `react-cookie` to the latest version. @sneridagh @robgietema
  • Language Switcher no longer takes care of the change of the language on the Redux Store. This responsibility has been unified in the Api Redux middleware @sneridagh
  • Markup change in `LinkView` component.
  • Rename `core-sandbox` to `coresandbox` for sake of consistency @sneridagh
  • Extend the original intent and rename `RAZZLE_TESTING_ADDONS` to `ADDONS`. @sneridagh
  • Lazyload draftjs library. See the upgrade guide on how that impacts you, in case you have extended the rich text editor configuration @tiberiuichim @kreafox
  • Deprecating `lang` cookie in favor of Plone official one `I18N_LANGUAGE` @sneridagh
See https://6.dev-docs.plone.org/volto/upgrade-guide/index.html for more information about all the breaking changes.


Bugfix

  • Fix the `null` error in SelectAutoComplete Widget @iFlameing
  • Prevent the MultilingualRedirector to force 4 content load when switching the language @reebalazs
  • Fix the upload image in contents view @iFlameing
  • add "view" id to contact-form container for main content skiplink @ThomasKindermann
  • Fix loading indicator positioning on Login form submit @sneridagh
  • Fix redirect bug with URLs containing querystrings @robgietema
  • Fixed id widget translations @robgietema
  • Contents Rename Modal, use `id` Widget type @sneridagh
  • Fix overflow of very long file name in `FileWidget` @sneridagh
  • Fix overflowing issue in the toolbar @kreafox
  • Overwrite current block on insert new block. @robgietema
  • Fix hot reload on updates related to the config object because of `VersionOverview` component @sneridagh
  • Fix error when lock data is gone after an invariant error. @robgietema


Internal

  • Change prop `name` -> `componentName` in component `Component` @sneridagh
  • Add new RawMaterial Volto websites in production @nzambello
  • House cleanup, remove some unused files in the root @sneridagh
  • Move Webpack related files to `webpack-plugins` folder @sneridagh
  • Remove unused Dockerfiles @sneridagh
  • Update Docker compose to latest images and best practices @sneridagh
  • Improve flaky test in coresandbox search Cypress tests @sneridagh
  • Better implementation of the add-on load coming from the environment variable `ADDONS` @sneridagh
  • Turn `lazyLibraries` action into a thunk. Added a conditional if the library is loaded or in process to be loaded, do not try to load it again. This fixes the lag on load `draftjs` when having a lot of draftjs blocks. @sneridagh
  • Use `@root` alias instead of `~` in several module references. Most of the Volto project code no longer needs the root alias, so it makes sense to phase it out at some point @tiberiuichim
  • Alias `lodash` to `lodash-es`, as this will include only one copy of lodash in the bundle @tiberiuichim

Documentation

  • Upgrade Guide i18n: Make clear what's project, what add-on. @ksuess
  • (Experimental) Prepare documentation for MyST and importing into `plone/documentation@6-dev`. @stevepiercy
  • Fix broken links and redirects in documentation to be compatible with MyST. @stevepiercy
  • Update add-on internationalization. @ksuess
  • Add MyST and Sphinx basic configuration for rapid build and comparison against MkDocs builds. @stevepiercy
  • Fix many MyST and Sphinx warnings. @stevepiercy
  • Remove MkDocs configuration. See https://github.com/plone/volto/issues/3042 @stevepiercy
  • Add Plone docs to Intersphinx and fix broken link. @stevepiercy
  • Get version from `package.json` @sneridagh
  • Remove legacy folder in docs @sneridagh
  • Backport docs of RAZZLE_TESTING_ADDONS environment variables. See https://github.com/plone/volto/pull/3067/files#diff-00609ed769cd40cf3bc3d6fcc4431b714cb37c73cedaaea18fe9fc4c1c589597 @stevepiercy
  • Add missing developer-guidelines/typescript to toctree @stevepiercy
  • Add Netlify for a preview of Sphinx builds for pull requests against `master` and `plone6-docs`. @stevepiercy
  • Clean up toctree errors by removing obsolete files, adding `:orphan:` field list, and reorganizing some files. @sneridagh and @stevepiercy
  • Switch to using netlify.toml to configure Netlify Python environment. @stevepiercy
  • Convert admonition syntax from Markdown to MyST. @sneridagh
  • Make links build both in Volto and Plone documentation. See https://github.com/plone/volto/pull/3094 @stevepiercy
  • Fix StoryBook build @sneridagh

What's Next

Where do we go from here? Plone 6! Right now, the only major feature missing is content rules and the new Slate editor. The rest of Plone’s features are covered in Volto 16.

So the work is not over yet. We still need helping hands and contributors to continue the effort to make Plone 6 a reality. Everybody is welcome!

Thanks!

We would like to thank all the people involved in creating Volto 15. It is amazing how much we were able to accomplish as a team, and as a community, during the last months.

See https://6.dev-docs.plone.org/volto/upgrade-guide/index.html for more information about all the breaking changes.

Try Plone 6 today

Feel free to try out Plone 6 with Volto 15:

Plone selected for 2022 Google Summer of Code

Posted by PLONE.ORG on March 07, 2022 08:41 PM

Once again this year members of the Plone open source community will mentor students and other young people interested in learning how to develop their skill and to contribute to an open source project. Plone has benefited greatly from the contributions of GSOC students since it began participating in the program in 2006. Many of those GSOC students have gone on to become core contributors to Plone as well as productive members of other open source software communities.

Students whose applications are accepted will work with a senior Plone programmer on one of a list of projects with the goal of making a contribution to the Plone code base or a related project. The Plone Foundation supports successful GSOC students by providing support for attending the annual Plone conference and for meeting other members of the diverse Plone / Python / Zope community.

Visit the GSOC home page and learn how to apply for the program

Check out our list of potential student projects for Plone in 2022

Student applications will open on April 4, 2022.

Join us !

Programmers report out at sprint in Sorrento, Italy

The Plone Newsroom Podcast Episode #06

Posted by PLONE.ORG on February 28, 2022 12:40 PM

The Plone Newsroom is a monthly podcast brought to you by Philip Bauer and Fred van Dijk. They cover technical and non-technical topics including Plone, the Plone community, and whatever else they come up with to keep us informed!

Episode #06 on 28 February 2022 featured news about new Foundation members and the 2022 Plone Conference dates, a discussion of the plone.org relaunch work and issues encountered, Volto, Classic and collective.exportimport development news, and a follow up to the composite page builders discussion in episode #05. Plus Philip and Fred issued a call for Volto translators and a call for World Plone Day talks. Go to the Newsroom page to view other episodes.

Welcome to Three New Members of the Plone Foundation

Posted by PLONE.ORG on February 23, 2022 10:25 PM

The Plone Foundation welcomes two new members after unanimous confirmation by the Foundation's Board of Directors on February 17, 2022.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Giulia Ghisini turning world upside down

Giulia Ghisini

Giulia is a frontend developer who works on the Volto development team. She has sprinted at Sorrento once and has participated in Plone conferences since 2019.Giulia works at Red Turtle and lives in Ferrara, Italy.

Piero NicolliPiero Nicolli

Piero also works with Red Turtle in Ferrara. He has been using Plone since 2015 and has made occasional contributions since then. Piero's focus has shifted to Volto after beginning with contributions on plonetheme.barceloneta and plone.staticresources. He is currently part of the Volto Team.

Piero attended Plone Conferences 2018, 2019 and 2021, and he helped organize the 2019 Ferrara conference. In addition to several sprints, Piero gave a talk about frontend development at PloneConf 2019 and about a use case for a repeatable Volto themes at PloneConf 2021. He is active on Discord from time to time and answer questions there when available.

Alessandro Pisa at SorrentoAlessandro Pisa

Alessandro works remotely for Syslab from Ferrara (Italy) and has been
a Plone user and developer since 2008. He has been a Framework Team member since 2017. He is a Plone core developer and contributes and maintains some packages in the Plone ecosystem.

Alessandro is a long-time contributor to the community and helps to manage the collective organization on GitHub. He has attended many sprints and makes sure to reach out to newcomers when conference time comes around.


The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Custom Volto configuration to fix Babel problems with react-leaflet

Posted by PloneExpanse on February 06, 2022 04:05 PM
I’ve started working on a new Leaflet-powered Volto map block and the first thing that happened while loading react-leaftlet was an error reported by the browser: Module parse failed: Unexpected token (10:41) in @react-leaflet/core/esm/path.js ... const options = props.pathOptions ?? {}; ... The problem is that is, for some reasons, the transpiled JS bundle includes code using the nulish coalescing operator This is already a problem reported in react-leaflet and it happens because the distributed transpiled library includes that code.

Questions for the Feburary Steering Circle?

Posted by PLONE.ORG on February 02, 2022 06:05 PM

As described in the Foundation's July 2020 discussion of Plone governance, a series of Steering Circle meetings is being held to discuss our organizational structure and processes, and any hot topics of the moment. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be Feburary 15th at 14:00 UTC (15:00 CEST). Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Plone 6.0.0a3 and Plone 5.2.7 Released

Posted by PLONE.ORG on January 31, 2022 07:50 PM

Important security fix applied

See forum post for more details and easy workaround:
https://community.plone.org/t/security-fix-for-image-view-fullscreen-cache-poisoning/14757

See also CMFPlone security advisory on GitHub.

Plone is vulnerable to reflected cross site scripting and open redirect when an attacker can get a compromised version of the image_view_fullscreen page in a cache, for example in Varnish. The technique is known as cache poisoning.
Any later visitor can get redirected when clicking on a link on this page. Usually, only anonymous users are affected, but this depends on your cache settings.

All Plone versions are vulnerable. It depends on your Plone version and the Image content type which package is vulnerable: Products.CMFPlone, plone.app.contenttypes or Products.ATContentTypes.

The Plone Security Team has released fixes for Plone 5.2:

  • plone.app.contenttypes 2.2.3 (see advisory)
  • Products.ATContentTypes 3.0.6 (see advisory. If you are on Python 3 you will not be using this package.

and for Plone 6:

  • plone.app.contenttypes 3.0.0a9 (see advisory)

Today, Plone 5.2.7 and 6.0.0a3 have been released with these updated packages. Separate announcements will follow.

If you have any questions or comments about this advisory, email us at security@plone.org. This is also the correct address to use when you want to report a possible vulnerability. See our security report policy.

Highlights of the Plone 6 Alpha 3 Release

https://plone.org/download/releases/6.0.0a3

Changes since 6.0.0a2:

  • plone.app.contenttypes: Security fix: prevent cache poisoning with the Referer header.
    See security advisory.
  • Updated the versions of the build requirements: setuptools to 59.6.0, zc.buildout to 3.0.0rc1, pip to 21.3.1.
  • Zope 5.4:
    • Add support for Python 3.10 (Plone does not have this yet).
    • WebDAV fixes.
    • https://zope.dev is now the canonical Zope developer community site.
  • plone.volto: Removed collective.folderishtypes dependency.
  • Products.CMFEditions:
    • Got rid of the skins directory. Most items in here have been moved to browser views. Some were no longer used, or had an alternative, and were removed.
    • The VersionView class is deprecated because it contained just one method that is now part of the @@plone view.
  • plone.app.linkintegrity: Track integrity of video and audio files in HTML source tags.
  • plone.app.uuid: Speed up uuidToPhysicalPath and uuidToObject.
  • plone.namedfile:
    • Make DefaultImageScalingFactory more flexible, with methods you can override.
    • Drop support for Python 2.7. Main target is now Plone 6, but we try to keep it running on Plone 5.2 with Python 3.
  • diazo: Removed FormEncode test dependency.
  • Pillow updated to 9.0.0
  • plone.app.content: Deprecate the human_readable_size method of the ContentStatusHistoryView class because the one from the @@plone view should be used.
  • plone.app.layout: Improved the Global section viewlet:
    • Catalog based navigation.
    • Allow more customization by adding methods as hooks.
    • Various performance optimizations.
    • Deprecate now unused navtree_depth property.
  • plone.app.layout: Removed deprecated methods.
  • plone.app.layout: Add viewlet to display customizable favicon. See the Site Settings.
  • Various packages: No longer use deprecated property types ulines, utext, utoken, and ustring, but their non-unicode variants, without a u at the beginning. See issue 3305.
  • plone.restapi:
    • Enhance @addons endpoint to return a list of upgradeable addons.
    • Add support for DX Plone Site root in Plone 6. Remove blocks behavior hack for site root in Plone 6.
  • Products.CMFPlacefulWorkflow: Removed the CMFPlacefulWorkflow skin layer.

Plone 6 editing experience combines the robust usability of Plone with a blazingly fast JavaScript frontend

Plone 6 editor

Plone 5.2.7 released

Specific release notes for Plone 5.2.7:

Some highlights of this release are:

  • `plone.app.contenttypes` and `Products.ATContentTypes`: Security fix: prevent cache poisoning with the Referer header. See security advisory.
  • `plone.app.linkintegrity`: Track integrity of video and audio files in HTML source tags.
  • `plone.app.z3cform` and `plone.app.textfield`: Enable multiple wysiwyg editors (use default editor registry setting).
  • `plone.namedfile`: Make `DefaultImageScalingFactory` more flexible, with methods you can override.
  • `plone.app.layout`: Improved the Global section viewlet:
    • Catalog based navigation.
    • Allow more customization by adding methods as hooks.
    • Various performance optimizations.
    • Deprecate now unused navtree_depth property.
  • `diazo`: Removed `FormEncode` test dependency.
  • `plone.restapi`: Be permissive when testing the schema of the querystring endpoint.

https://plone.org/download/releases/5.2.7

On The Road to Plone 6 - Volto 14 Released

Posted by kitconcept GmbH on December 22, 2021 09:24 PM

Volto continues to innovate at a fast pace towards Plone 6. Today we released another major milestone on our road to Plone 6: Volto 14.

Volto 14 was in the making since September 2021 and it is in active use in various projects at Eau de Web, Red Turtle, Rohberg, kitconcept, and others.

Volto 14 comes with a set of new exiting features: a new search block that supports faceted search, locking support, a new seamless mode that makes deploying Volto easier, a new mobile navigation and support for Node 16.

Faceted Search Block

The new search block allows editor to create sophisticated faceted searches through the web without writing a single line of code.

Editors can define criteria for the content that is listed, like in the existing listing block in Volto.

Then editors can then choose arbitrary facets that are displayed to the users to choose from to narrow down the search.

Locking

Locking in a Content Management System is a mechanism to prevent users from accidentially overriding each others changes.

When a user edits a content object in Plone, the object is locked until the user hits the save or cancel button. If a second user tries to edit the object at the same time, she will see a message that this object is locked.

volto14 locking Content object in Volto 14 that is locked by another user

Locking requires at least plone.restapi 8.9.0 or plone.restapi 7.4.0 to be installed.

Seamless Mode

The new “seamless mode” allows zero configuration deployment by avoiding hardcoded environment variables in builds involved, and establishing good sensible defaults when setting up deployments (and also in development). So the developer/devOps doesn’t have to overthink their setups.

These are its main features:

  • Runtime environment variables
  • Unified traversal ++api++
  • Use Host header to auto-configure the API_PATH

and these immediate benefits:

  • Avoid having to expose and publish the classic UI if you don’t really need it
  • If possible, avoid having to rewrite all API responses, since it returns paths that do not correspond to the original object handled and “seen” from Volto, so you have to adjust them (via a code helper) in a lot of call responses.
  • Simplify Docker builds, making all the configuration via the runtime environment variables

Seamless Mode requires at least plone.rest 2.0.0a1 to be installed.

New Mobile Navigation

We polished the mobile navigation for Volto 14 to improve the user experience on mobile.

New mobile navigation in action on the Plone Conference 2021 website

Node 16

Volto 16 will support Node 16 in addition to Node 14 and Node 12. Node 16 will receive security updates until April 2024.

node 16 Node 16 release schedule from nodejs.org

Plone 6

Volto 14 is an important step towards Plone 6. Volto 15 will switch the default editor from DraftJS to Slate editor and it is planned to be ready in Q1/Q2 2022. This is the last big step for Volto before Plone 6 can be released.

David Ichim: What's new in Plone 6 frontend for developers

Posted by Maurits van Rees on November 08, 2021 11:18 AM

Distilled from the latest work done in Volto, we're showcasing some patterns, features, or enhancements that have landed in Volto from the last year to the present. We will also have a glimpse of what is ahead in the future of Volto with the new features roadmap.

In the past year we had four major releases, 40 minor releases, 36 alpha releases, 25 patch releases, for a total of 105 releases. Plus some new tooling and tool releases, like plone i18n and plone generator.

New Volto config, dubbed as Volto's Configuration Registry, introduced to fix circular import dependency problems. Hot Module Reloader was fixed. Read the upgrade guide.

New i18n (internationalisation) infrastructure. This is now a separate package. Same is used for generation of i18n in add-ons.Read the upgrade guide.

Forms as schema. Forms should be constructed from schemas instead of by hand. InlineForm allows us to create forms for blocks from a schema. Blocks can have variations, or we can extend it. Read the edit component documentation.

New widgets:

  • Object List Widget. Similar to the original DataGridField. Used in core by the Search Block facets.
  • Object Browser Widget is now a separate widget, instead of part of a block, and now allows the addition of external content.
  • Querystring Widget. Behaves like its counterpart from plone.app.querystring. Allows to create search criteria, used by the search block.
  • URL Widget. Used on text inputs, it knows to validate their value as a url, both internal and external.
  • Vocabulary Terms Widget for a JSONField, acts as a source for a SimpleVocabulary or Choice field. Play with it in the storybook.

New core blocks:

  • Search block
  • Teaser block (ongoing work).

Pluggable framework, similar to React's Portal component. See the talk by Tiberiu and see the Pluggables development recipe.

Storybook provides a sandbox to build and test visual components in isolation. Currently it is only setup to be used by Volto core. We need help with work to have storybook setup with adding. See the Storybook talk held by Victor, and see the storybook itself.

Critical CSS: inline the critical CSS for improved performance. Run critical-cli to output critical.css. This is then inlined in the headers, while regular CSS is moved to the bottom of the body. See the deployment documentation.

Lazy Loading utilities. Introduced injectLazyLibs HOC wrapper to inject lazy-loaded libraries as properties to your components. These components are only loaded once all your main libs are loaded.

Express.js middleware. Volto uses this for SSR and static resources serving. You can now write custom middleware and add it to settings.expressMiddleware.

API expanders allow the expansion of different API endpoints from Volto with calls from your custom endpoints. Avoid adding too many expanders if they are not critical to the initial page.

External routes: useful when another application is published under the same top domain as Volto. If Volto would fetch the content for that path, it will break. You can disable this path in config.settings.externalRoutes. You can also use regular expressions.

Seamless mode, introduced in Volto 13, enhanced in Volto 14, which has already seen a lot of alpha releases. Originally, we tried to unify both frontend and backend servers under the same path, but this was tricky, causing various problems. We settled on a new ++api++ traversal for getting information from the backend. Also, to come closer to zero config, you can now pass environment variables at runtime instead of build time. This means you can generate one build, and use this in all environments (testing, production). Read the deployment documentation.

Context navigation component. This is a navigation portlet component, similar to Classic Plone. The view is there, but you need to enable it. See the development recipe.

There is some work in progress:

  • Slots are Volto's answer to portlets, see the Volto Slots talk by Tiberiu Ichim for more details.
  • Image proxy: image scale generation done by a middleware instead of plone.scale.
  • Authentication from backend.
  • Replace Draft.js editor with Volto Slate. What is missing is a migration tool from one to the other. But work has started on a block conversion tool.
  • Async blocks that work with SSR.
  • Defaults in blocks form.

Future work:

  • Defaults in all widgets
  • Enable blocks enhancers in all blocks
  • Storybook in add-ons
  • Use newest react-intl package
  • Refactor the folder contents component
  • Form editing text enhancements, making it easier to modify text inputs.
  • A "group block" included with Volto
  • Quanta toolbar

I did nothing, I just brag about what others have done. So thank you Volto early adopter community!

Tiberiu Ichim: Volto Pluggables

Posted by Maurits van Rees on November 08, 2021 09:55 AM

This presentation is an introduction to the new Volto developer-targeted feature, the Pluggables framework. It is more an argument for extensibility in CMS UI and in Volto.

Basically, with Pluggable a central component provides a pluggable slot that other components can fill, like this:

<div className="toolbar">
<Pluggable name="toolbar-main" />
</div>
// ...
<Plug id="quicklinks" pluggable="toobar-main">
<Button />
</Plug>

This is a Volto port of https://github.com/robik/react-view-slot.

The big picture:

  • I work with Eaudeweb Romania
  • Our client EEA is an early adopter of Volto
  • The strategy is: move everything to Volto
  • But the EEA sites are less brochure, the CMS side is really strong.
  • We build powerful UIs for power users.
  • The EEA already has 91 Volto repositories on GitHub. How can we scale that? Can we write an add-on to make it easier to write an add-on?

In React, data flow is top to bottom. A parent component passes properties to children and children communicate with the parent by emitting events. This makes sense and works well. For "out of tree" data you need Redux. There is no protocol for add-hoc communication between components.

UI state is fluid. Extensibility means reusability and scalability. This is hard. You need to design upfront. Plone backend uses the Zope Component Architecture, which means pluggability is baked in, it is very natural. You can view Pluggables as viewlets-on-demand, but they are really not. But yes, you can think about a Pluggable as a viewletmanager and a Plug as a viewlet.

You can overwrite a Plug with a Plug, by registering it with the same id. So if the original Plug gives you color blue, you can overwrite it with color red.
You can do custom rendering of Plugs within your Pluggable, by iterating over all Plugs and for example wrapping each in a div with a class name.

Showcase: volto-workflow-progress (main toolbar plugin) and volto-editing-progress ("sidebar" for plugin).

Limitations:

  • No Server Side Rendering
  • Watch out for dependency lists.
  • Limited adoption for now.

Implementation in pseudocode:

First the context:

<PluggablesProvider>
<Pluggable name="top" />
<Plug name="delete" />
</PluggablesProvider>

Then the dumb version of the Plug:

const Plug = ({id, children}) => {
const { register } = useContext(PluggablesProvider.Context);

React.useEffect(() => {
register(id, () => children);
});

return null;
}

Takes a bit of study, but in the end it is not so hard.
The Pluggable becomes a bit simpler:

const Pluggable = (name) => {
const { getPlugs } => useContext(PluggablesProvider.Context);
return getPlugs(name).map(f => f());
}

Site updated from Plone 2.5 to Plone 6.0

Posted by Maurits van Rees on November 04, 2021 11:34 AM

I am one of the Plone Release Managers, and have been working on Plone 6, which is now in alpha stage. But my personal website was still using the ancient Plone 2.5:

Screenshot maurits.vanrees.org Plone 2.5 Screenshot of my website end of October 2021: still Plone 2.5

Often I have made plans to update my site to:

  • Plone 3
  • Grok
  • Plone 4
  • Plone 5
  • Plone 5.2
  • finally Plone 6

Long ago it was clear to me that an inline migration would not be practical. It would take too many steps: update the code to Plone 3.3, migrate the data, to Plone 4.3, migrate data, to Plone 5.2 Python 2.7, migrate data, Plone 5.2 Python 3, migrate data, Plone 6, migrate data.

Additionally, the question was how to handle the weblog, which is the main content. This was using Products.Quills, a Plone add-on for blogs. Updating to Plone 3 could have worked at the time, but this was made harder by some some custom code I added. This enabled me to use it as a podcast. I used this to enrich some of my summaries of sermons from my church with the actual audio of the sermon. I doubted whether to even include this content in Plone 6, as the last sermon was from 2008. I hate breaking links, so I kept it, although a bit hidden.

Another point that needed some extra attention, was that most, if not all, blog entries were not written in html, but in ReStructuredText. I make a lot of summaries of talks when there is a Plone Conference. The html editor on Plone 2.5 did not work anymore, or I disabled it to a simple textarea. I always open up the same text editor that I use for programming (previously Emacs, currently VSCode), and type the summary there. I much prefer writing ReStructuredText there, especially when I simply need text without any markup. I then paste it in Plone, without fear of losing all my work when my internet connection dies.

Lastly, I have an RSS/atom feed which is used by planet.plone.org and maybe others to stay updated when I add a new blog entry. I did not want this url to change.

Anyway, about six years ago I decided that I would use collective.jsonify to export my site, and then import it using transmogrifier. But time passed without any progress. This year, collective.exportimport was shaping up to be the preferred way to import the data. For export you can use it in Plone 4.3 as well, but definitely not in Plone 2.5.

At the beginning of this week I looked at jsonify. Didn't I have a local copy of my website on my laptop, with collective.jsonify installed? No! And it was not installed on the live site either. And installation would be hard, because the site uses Python 2.4, and currently you cannot even reach PyPI with older versions of Python 2.7.

Mildly panicked, I started on a custom script to export the content, still as json. Some details aside, this actually was not so hard. I looked at what collective.exportimport expected, and created the Python list of dictionaries accordingly. Then do a simple json.dumps() call and you are done. Except that this gave an ImportError: the json module is not available in Python 2.4. Okay, install simplejson package. But you need PyPI access for that, which does not work. Workaround:

  • Manually download an egg of Python 2.4-compatible simplejson 1.7 and save it in the buildout directory.
  • cp bin/instance bin/instance-json
  • Edit the new script and add the simplejson egg to the system path.
  • bin/instance-json run export_mvrsite25.py

After that, it was not too hard anymore. I used plonecli to create a new add-on with Plone 6.0.0a1. I actually do not yet use the add-on code, except for loading a minor patch that I added, but this gave a reasonable, modern Plone 6 buildout. Create a Classic Plone site, tweak a few settings (let Folder use folder_workflow, allow English and Dutch, navigation depth 1, enable syndication, configure caching and mail), import the content with collective.exportimport, edit a bit, and done.

The weblog now consists of standard Folders and Pages. To improve the view, I added a Collection, showing the latest pages first, and showing all content of the last seven blogs. I enabled syndication here, so it has an RSS/atom feed.

The weblog has always advertised two atom feeds:

  1. One for all weblog entries, at https://maurits.vanrees.org/weblog/atom.xml
  2. One for weblog entries with keyword 'plone' at https://maurits.vanrees.org/weblog/topics/plone/@@atom.xml

In the new site, the first one kind-of worked out of the box, but it only showed the items that were directly in the weblog folder, and this is not where my weblog entries are. I solved this with a patch to Products.CMFPlone.browser.syndication.views.FeedView: when atom.xml is viewed on a folder, check if it has a default page, and show the atom.xml of this default page instead. In this case, the default page is a Collection with the correct settings. So the general feed will keep working.

For the second one, my first idea was to create a folder 'topics' and within it a Collection with id 'plone'. Problem: 'plone' is a reserved word and cannot be used as id of a content item, so the id became 'plone-1'. Solution here: create the 'plone-1' collection directly in the weblog folder, and do a redirect in the frontend server (nginx):

 rewrite ^/weblog/topics/plone/@@atom.xml /weblog/plone-1/atom.xml last;

And that's it! My website is now on Plone 6.0.0a1:

Screenshot maurits.vanrees.org Plone 6.0 Screenshot of Site overview in new site.

There are some more details that I could go into, like splitting up the buildout into multiple parts, with tox as main way to build everything, in preparation for moving more and more parts to pip-only. But that will have to be for another time.

Meanwhile: have a look around, and enjoy the fresh look!

Steering Circle

Posted by Maurits van Rees on October 31, 2021 02:39 PM

Volto 14 alpha 23 is out. So still in alpha, but companies are using it in production. Should be final soon. Some plans for Volto 15. Created plone.volto integration package, where we try to give an easy transition from earlier company-specific versions. plone.restapi as always is pretty boring, stable. Erico worked on Docker integration.

Plone 6 alpha 1 is out. Eric sent an email for some coordination, like docs, training, accessibility, installers. If you want to be involved, let me know.

Franco has stepped out of the Framework Team, thank you for all your work. There is discussion about the role of the Framework Team. Plan is to keep it running, some more people have been asked.

Membership team: we have some people in sight as new members. Erico is stepping down as team lead, Victor is stepping up.

Security: Plone 6 will have 5 years security support. Synchronizing with Zope team. Some new members may be coming.

Marketing: busy with conference, also after the conference. Busy with plone.org renewal.

Installers: see the talk by Jens Klein earlier. Plone 6: no more installer, but we do have tooling. There are Docker images. We may want to reduce the role of buildout, and focus more on pip.

Plone Conference: you are looking at it. Some tasks to do afterwards. If anyone is interested in getting a certificate for following a training, ask us, and we can send it.

Internationalization: new branches for Plone 6, so Plone 5 uses different branch. New releases for 5 and 6. Updating po files, looking at i18n in Mockup.

Admin/Intrastructure: servers are still running. Cat herding sysadmins for doing stuff, keeping things up to date.

Trainings: relaunch is complete. We have three new trainings: Theming Plone Classic, Deploying Plone 6, Getting Started with Plone 6 (that one only in video). Various have seen major updates. Need to work on landing pages (we have two), remove the number 5 from the url, update some more trainings. Maybe Mastering Plone Classic training, but hard with navigation due to duplicate section targets when copying. Migration training would be good. We need to prune and tag some.

Plone Classic: We did polishing on Barcelonate, it is pretty ready. Focussing on bobtemplates before the trainings, making theming easier. JavaScript/ES6 is the remaining big issue. Plan is to finish it this year, we are quite far. We need other people helping us out.

Documentation: will be releasing a new Plone 5 branch today. For the new stuff, the tech stack is ready. New version of automated screen shot is about ready. We don't want a duplicate of the training, but we can automatically include code of it, so there is only one source of truth. The style guide is not always followed, seeing what we can do about that. Biggest point is missing documentation. There are now branches where the various teams can add and edit their content. We may change things, but we take this as input for the final structure.

Fred van Dijk: 7 things that can surprise you when you start customising or developing for Plone

Posted by Maurits van Rees on October 29, 2021 09:06 AM

This is a basic rundown and summary of our beloved subjects like ZODB persistence. Traversal. The view/viewlet/portlet trinity. How is a call handled in Plone. The differences between zcml and generic setup. Utilities and the ZCA. restrictedTraverse. The Plone catalog.

These are surprises that I have encountered myself, or that I have seen on faces of people I have trained or worked with during the years.

  1. Everything can or could be done through the web (TTW).

Zope vision from the nineties. Why don't we use this dynamic language called Python so we can change things TTW? It's so easy.

  1. The learning curve.

It starts easy, but then you hit what we call a Z-shaped learning curve. Dynamic Python makes things easy, and then it makes things hard, at least when you put Zope on it, then CMF, then Plone. Plone the product on top of a CMS on top of a framework on top of a language. We have a product, a framework, a stack, so it is hard.

  1. Five levels of conceptual complexity.

It helps to teach all the levels. Give new users a drawer for each level, so they have a box that they can put some info in.

You have:

  • browser: html, css, js
  • frontend: transforms, templates, zpt, metal, diazo
  • application logic: views, viewlets, portlets, adapters, Zope Component Architecture
  • dynamism: GenericSetup, zcml, xml, zope schema
  • programming language: Python, buildout, pip
  • package WorkManager (OS): apt, rpm

4 Same language/formats on different levels:

  • XML is used for the ZCA, GS, zope.schema, Diazo rules
  • package manager: buildout, pip, setuptools, GS

But: there is no magic. It is just advanced technology.

Startup:

  • Python uses sys.path modules to start bin/instance
  • ZCA loads site.zcml, package includes, other zcml to change configuration.
  • Then we have a runtime environment with objects and classes, the ZODB. GenericSetup is then some XML that you can use to change the ZODB.

So the ZCA overrides components in runtime. The alternative is to edit core files, maybe compile them, and restart. Much less nice and not sustainable.

So now we have a Plone process running.

  1. Zope is not that complicated.

Over HTTP Zope gets a request, does traversal, finds an object in the ZODB, loads it in memory. Then on top of this object we show a browser view / template. The template hooks into a main template, maybe does some sub requests, some Diazo, and we end up with some html and we are done.

This is all still 'lies to children'. It is simplified until we are able to understand more. With increment exposure to these concepts, it will stick more. It is complicated, but there is no magic.

  1. Acquisition.

It is traversal in the wrong direction.

  1. When you try to explain things, you improve your own understanding.

There is so much Plone content online: training, docs, youtube, github, discourse. We all learn in different ways, with own preferences, reading, viewing. There are so many repositories on github that you can explore for new ideas. Just yesterday Philip did a talk about exportimport and afterwards I did it, but from a different angle. It helps.

The community is our biggest asset.

Annette Lewis and Will Gwin: From Zope to Plone: Thinking User-First During Migration

Posted by Maurits van Rees on October 28, 2021 02:39 PM

Migrating a site is always a challenging task, but when you have dozens of subsites with specific brand standards and custom user functionality, the challenge becomes mammoth. Six Feet Up worked hand-in-hand with Purdue's College of Engineering to migrate their existing Zope site and its subsites into a new Plone instance running on Plone 5.2 / Python 3. Throughout the migration process, we considered the project scale, timelines, and limiting the impact on end users, all while managing the balance between user needs and best practices. During this presentation, you will learn:

  • why it matters to think user-first during migration,
  • about creative solutions for translating content and functionality into Plone, and
  • how to successfully migrate subsites.

An overview of the project:

  • HigherEducation always seems to go a bit slower, certainly with migrations.
  • Our previous CMS was built in Zope and was getting extremely old.
  • We have been using Python since 2001.
  • There were security concerns and modernization issues.

The impact:

  • Only 15+ content editors.
  • 40+ public facing subsites
  • 30,000 total pages.
  • 20+ departments and units

Why did we select Plone as the next CMS?

  • It is a modern CMS solution
  • Python-based, so that fit what we currently have.
  • It is built on top of the Zope web framework that we were already using.
  • We looked at Drupal, Wordpress, and more, but that would have been a too big undertaking.

This was in 2018.

Laying out the solution. From requirements to action. Who are our users and what do they want? Not just our direct client (Will and his team) but their clients/users.

Challenges during development:

  • Purdue University changed its brand look. We had to seamlessly blend subsites into the existing parent site.
  • Convert all content types and templates from Zope to Plone.
  • Keep sight of the user experience in both environments. Could they use the new environment without too much training, or needing to have too much tech knowledge?

Determine the project essence. Distill the requirements down into broad categories: accessibility, usability, flexibility, security.

What is the path to successful collaboration?

  • The absolute best might not be the right answer.
  • It's okay to say no to an idea, but you dhouls have an alternative ready.
  • Aim for the best, avoid the dangerous, end up somewhere in the middle sometimes.

On to our development goals:

  • Do things in a Plone way. Plone uses Zope, but Zope may do some things in a different way than is the best way in Plone.
  • So observe best practices.
  • Make it intuitive and keep it familiar for the editors.

Solutions at a glance

Migrating site content:

  • We wanted to move subsites one by one, as needed.
  • Translate existin content to Plone content types
  • We could re-import content over existing content non-destructively.

Theming: retrofitting Plone into an existing theme. That is what Diazo was made for, bridging the gap between Plone and the theme, especially since the theme is 'living', with subtle changes coming in often.

Each subsite had a browser view named local.css to change some things. Not really what you want in Plone, but they really needed it, as a way to make subsites or sections look different. So we added an action to edit the local css, inheriting from parent folders.

We created a subsite settings control panel. We used lineage.registry for this. All kinds of customizations can be done based on that, for example add extra text and links in the navigation menu. They used to be able to do this in Zope as well, but that was with various properties, and much more code oriented.

We use Mosaic for flexibility of layout. In Zope we had blocks for layout. Mosaic took this a step further into a nice UI with drag and drop. It gives faster site prototyping and development.

The sandbox: we wanted to have safe spaces for content experimentation. Completely separate from production environment. It is used for new user training and testing. It is quick and easy to reset.

This migration project has been a constant collaboration between the Purdue communications office, Engineering Computer Network, and Plone company Six Feet Up. The content editors feel empowered to make complex changes, without constant oversight from my team.

Alan Runyan: Building a Secure Cross Platform Mobile/Tablet Application (Flutter) using Plone as Backend Server

Posted by Maurits van Rees on October 28, 2021 01:44 PM

Enfold has been working on a secure cross platform mobile application the past eight months. Walk through of the Requirements, Security, Flutter framework, Backend configuration of Plone, Authentication and Lessons Learned. Our goal is to have a free public release of a limited version of the application Q4 2021.

It was an adventure for us. The core team never built a large mobile application. We did not know what we did not know.

The big picture:

  • A mix of devices (Android, iOS) needs to synchronize files to and from Plone.
  • All services are self-hosted in GovCloud. So we have no central database or server that we control.
  • If this becomes a success, then future phases may require a lot of certification of the codebase, code reviews.
  • Initially we worked with 15k devices, supporting 40k would be a success, the ceiling that we might support is 300k.
  • Users are completely offline for longer times.

We used Flutter to create a React native app, see https://flutter.dev It is a UI toolkit. Why did we use Flutter?

  • It is cross platform mobile.
  • It uses Dart, which is statically typed, making code analysus much easier.
  • Google seems to be prioritizing developer user experience, it really shows of quite a bit.

Dart has asynchronous code as a first class citizen. Quite different from Python. Runtime reflection (pdb) is unavailable. It has good ergonomics, with generics and closures. It is a driving force behind the Flutter toolkit.

Thoughts on mobile development:

  • It is a lot to take in.
  • You need to understand lots of languages, for us: kotlin/java, swift/obj-c, and Dart.
  • No idea how to test platform integration.
  • Native libraries are managed using Cocoapods/Gradle. Flutter drives those. Setting it up is yet another new thing to learn.
  • There are lots of inconveniences, like how do you read sqlite off a device, because that is how we store some of the info?
  • Also inconsistencies: if the app works on an emulator, that does not mean it works on a device.

On the server side:

  • Plone operating as a Webdav server
  • We need to support OIDC (mod_openids/oauth2)
  • Not many writes, maybe 100-1000 per day, but lots of reads.
  • 20k+ devices daily

Alternatives to the server/protocol could be nice:

  • Honestly, are there any standards other than WebDav?
  • An S3 api would be reasonable.
  • So ideas are welcome, let me know.
  • We have been working on prototypes with guillotina_webdav + openidc.

The good parts of Flutter:

  • UI/UX development is very fast, with lots of widgets. We had two developers who were used to Angular, and they took it up quite fast.
  • The bridge to native code (Pigeon) is straight forward.
  • Drift is an amazing sqlite library.
  • Riverpod for state management
  • Dependency management is good (flutter pub). You can tell that they learned a ton from others, like pip. Except that very occasionally the package cache is broken so you need to clean it.
  • They have a good community, with add-ons.

The not so good parts of Flutter:

  • Inconsistent platform features, like WorkManager (Android) versus NSUrlSession (iOS)
  • Dependency churn: often new versions come in, which you then need to check.

The mobile app:

  • We are still wrapping up the remote file operations.
  • After we deploy into production, we will improve the UI.

Yes, we hope to open source the synchronization framework, and maybe the foreground/background transferring subsystem. Yes, we have built a Flutter web-app of this, but it looks just like the mobile app currently. Needs a separate layout really. No, we have not done a Desktop app.

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released!

Posted by kitconcept GmbH on April 12, 2021 09:24 AM

Two weeks ago we cut two important releases on the road to Plone 6. Plone REST API 7 introduces a new link integrity feature for blocks-based pages. Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

stephen leonardi 5CH1TNfcZoo unsplash Photo by Stephen Leonardi on Unsplash

Plone REST API 7

Linking pages is one of the core idea of the world wide web. Keeping links within a website intact is therefore one of the core features that any Content Management System needs to provide.

One of the core features of Plone has always been that editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unigue IDs (UUIDs) instead of relative or absolute paths when adding a link to another page. Plone uses “portal transforms” internally to rewrite those links in RichText fields on save operations.

Plone 6 (aka Volto) introduces blocks-based page layouts, that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout allows more complex page layouts. Though, because of this change, the existing portal transforms mechanism that rewrites links to UUIDs did not work any longer.

volto blocks edit mode Blocks-based Volto page in edit mode (with kitconcept-blocks-grid)

Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020, since they needed that for a client project.

I started to draft a possible solution and wrote a first prototype. Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy.

Andrea Cecchi from RedTurtle joined our efforts and refactored the resolveUID algorithm into a blocks transformer, that we started to use for other use cases and that made the resolveUID transformer more generic and flexible.

After that, we cut a first plone.restapi 7 alpha release and entered a longer period of quality assurance and testing.

Two weeks ago, after we tested the new feature in multiple projects at kitconcept, I cut a final release of Plone REST API 7.

After the first alpha release, the ResolveUID transformation was added to links and images. In addition plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

We also added a “smart fields” concept that allows integrators to mark a blocks field as searchableText field.

A new @contextnavigation endpoint was added that allows for local navigations. We enhanced the navigation endpoint to expose an optional navigation title (“nav_title”) field.

Since the final 7 release, we cut six more releases and REST API 7 was included in Plone 5.2.4 release.

Volto 12

gaelle marcel vrkSVpOwchk unsplash Photo by Gaelle Marcel on Unsplash

Tiberiu and Victor worked on a new Volto configuration registry which is the new central point to store and retrieve Volto configurations. The configuration registry is a singleton that ensures a setting only exists once. With more an more Volto add-ons that started to depend on each other, we started to run into circular dependencies issues that we can avoid now with the new configuration registry.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Plone 6

Plone REST API 7 and Volto 12 were two very important releases on our road to Plone 6.

We plan to cut another Plone REST API 8 branch and release for Plone 6, which will support Python 3 only.

Volto will continue to move at very high pace towards Plone 6. We started to organize a series of Plone 6 “Micro-Sprints” to push things further.

Check out the Volto Roadmap on github for more details.

Implementing user epics with BPMN

Posted by Asko Soukka on March 21, 2021 12:00 PM

Many user stories start simple. For example: “A user self registers into a course”. The final version, of course, tend to be more like: “A new user, after being verified to be an authentic real person, creates their new user account, immediately forgets and resets their new password, finally logs into the system, finds the course again, completes payment for the registration, and is then eventually enrolled into the course.” This is no longer a user story, but a series of stories supposed to be completed in a specific sequence. Let’s call these user epics.

An epic is a sequence a user stories

Individual user stories, by their definition, should be relatively straightforward to implement. For example, a stereotypical user story may require interaction with specific user interface element to start the story, following one or more views to gather the required user input to continue the story, finally ending with user’s goal for the story. Where did the user come from, or where does the user continue afterwards, is simply not part of the story. On purpose.

But user epics are different.

Epics must know the consequences of users' actions

An implementation of a user epic must know where the user came from, and must be able to decide, which stories the user should be guided to complete next. Epic should also cover the consequences of user failing to complete a story, or completely abandoning their fate. Therefore implementing an epic is not only about managing state, but also about programming the often complex business logic. Both being among the least fun parts of software development.

As an insult to injury, less and less are users doing their work alone, more and more they are being accompanied by various forms of automation. So, not only should an implemented epic be aware of the actions of its user, but it should also be able to react on known results or side effects of automation – anything from synchronous function calls to asynchronously executed software robots.

With just code, implementations for this kind of epics could get dirty and hard to understand, fast.

Epics must know the consequences of users' actions

Offloading state and logic to BPMN engine

It’s not a coincidence that my example user epics above are drawn as BPMN diagrams. That’s what I have been doing for the last couple of months in our latest project. We had to modernize a process, where users were found to require a lot of hand-holding to be able complete all the required steps and in the right order.

Also, with the previous implementation, when users got lost in the middle of their epics, or an unexpected system error blocked them, it was often surprisingly hard to figure out what really had happened.

This time all this should be different.

The active user task may be anywhere in the current process tree

Whenever a user starts a new epic, a new BPMN modeled process is being started. The model may be as long and complex as needed. It may even be a composition of multiple nested models, resulting in an arbitrarily deep process tree when executed. But this is not a problem, because this is exactly what dedicated BPMN process execution engines have been built for.

The fun part: With a BPMN engine executing our user epics directly from their BPMN models, we don’t really need to “implement” them at all. Instead, we:

  1. Implement user interface routes and views for individual user tasks.
  2. Make the views to read their preconditions from the process engine state.
  3. Eventually complete the task at the engine (with user input when required).
  4. Poll the engine for the next available task and redirect the user there.

Now our user task views are standalone, with no magical dependencies between them to code and maintain.

Plone, Volto and Camunda FTW

I would not be writing this, unless all the above could not be implemented by using open source software only:

  • Because every application seems to need a CMS (sooner or later), we build on top of Plone, the open source CMS we trust. Thanks to its powerful and complete REST API, we can do all the integrations we need with just HTTP requests – no need to know about Plone internals.

  • For modern user experience, we use Volto – ReactJS based web experience on top of Plone REST API (with Razzle based server side rendering support). Every part of Volto is easily customizable and extensible. For example, custom server side routes, like the ones we need to authorize and proxy calls to BPMN engine, are common and well documented use case.

  • Finally, for the BPMN engine, we choose Camunda BPM Community Edition. While the open source version of Camunda is probably missing countless convenient features of their enterprise offering, its BPMN engine and REST API come uncompromised.

Thanks to all above, it’s not a (technical) problem at all that users tend to forget their passwords just after sign up. Now the process engine remembers were they came from, and what they were supposed to do, just after resetting their password one more time…

With a reusable sub process, password change can part of any epic

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'xxx@xxx.de': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'xx@xxx.de': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone xxx@xxx.de'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

moniviestin workflow menu

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

moniviestin workflow

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

moniviestin example

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

moniviestin content rule

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

moniviestin rpa poll

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

  1. Video author selects “Publish with subtitles”.
  2. Plone content rule calls Jenkins webhook to schedule a new robot.
  3. Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
  4. Jenkins schedules a new job to poll pending ASR jobs.
  5. Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
  6. Plone content rule notifies the video author with email that automatically generated subtitles are now available.
  7. ???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

study guide tabs

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

hasura view

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

hasura relationships

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

volto contents

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

  1. Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

plone dexterity editor , 2) Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

volto sisu connector

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

study guide edit

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

study guide image

Something old, something new and something blue.

The perfect match.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.

  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.

  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.

  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.


All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.


If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM
Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM
I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.

Plone Beethoven Sprint 2019

Posted by kitconcept GmbH on September 25, 2019 09:00 AM

beethoven sprint group image

21 developers from nine different countries gathered in Bonn, Germany between June 20th and 24th to work on implementing the upcoming Plone 6. The sprint at the office of the kitconcept GmbH has been declared a “strategic sprint” by the Plone framework team. Sprint topics included working on Volto, the Plone REST API, and Guillotina.

Volto

Volto is a new ReactJS-based frontend for Plone. It was started by Rob Gietema in 2017 and it is actively developed since then. Volto will become the default frontend for Plone 6. It implements a complete new user interface called Pastanaga UI, which was developed by Albert Casado.

Victor Fernandez de Alba and Rob Gietema led the efforts to further enhance Volto together with Paul Roeland, Nicola Zambello, Piero Nicolli, Janina Hard, Jakob Kahl, Thomas Kindermann, Maurizio Delmonte, Stefania Trabucchi, Rodrigo Ferreira de Souza, Fred van Dijk, and Steffen Lindner.

Pastanaga UI Toolbar

The new editing toolbar has been part of Volto right from the start. Though, so far we did not fully implement every detail that Albert has imagined for the Pastanaga UI toolbar. Victor has been working on this for quite a while already. During the sprint, he integrated his work and polished it. Victor and Rob also worked on a new sidebar that shows up on the right side of the Volto user interface that holds additional information like page metadata and controls the settings of the individual tiles.

volto toolbar The new Volto edit toolbar

New Tiles: Collection, Slider and Table

At the Plone conference in Tokyo, we came up with a list of tiles for the new Volto composite page editor that we would like to implement. On top of that list were the listing/collection tile, the slider tile and the table tile. Piero, Rodrigo, and Rob started to work on the listing/collection tile that is supposed to become a replacement for the Collection Type in Plone.

Jakob and Janina worked on copying over the code of a slider tile that was developed by kitconcept. Rob implemented the table tile. Before you could ask Rob about the progress, he already finished his work, like always…

volto table The new Volto table tile

Accessibility

Accessibility has always been a first class citizen in the Plone community. Plone is used by many government and public websites where accessibility is a basic requirement. Paul, Thomas, Timo, and Victor worked on improving the accessibility of Volto. Our goal is to fully support the WCAG 2.1 standard with the “AA” level of conformance.

Timo set up ESlint-based static code analysis checks and Cypress-based axe accessibility tests. Paul, Thomas and Victor then started to fix the reported accessibility issues. At the end of the sprint, we were able to fix all the reported violations. Our CI build now fails if new accessibility violations are committed.

Documentation

Sven Strack, the Plone documentation team leader, joined our sprint remotely from Amsterdam to discuss how to maintain and structure the Volto documentation in the future. We all agreed that we will use Markdown as format for our docs, since this is the de-facto standard in the JavaScript community as well as in most other Open Source communities. As much as we like Restructured Text, it seems that Markdown won that battle.

We currently use MKdocs for the Volto documentation and Styleguidist to document our React components. There are lots of different tools to choose from, Docz, MKDocs, MDX, Gatsby to just name a few. We agreed that we need to do more research to make an educated decision regarding our doc toolchain.

There will be a Google Season of Docs project this year where a technical writer will help us to enhance the Volto docs further. More Volto Features Nicola added a feature to display the currently used Volto version in the Plone control panel, he added animations for page transitions and added an Italian translation for Volto. Together with Victor, he worked on a toast component for user notifications.

More Volto Features

Piero worked on improving and fixing the event type view and added styling for the listing tile.

Victor and Steffen worked on the image sidebar.

Rodrigo added a feature that allows the user to use SHIFT+RETURN to create a new line in a Volto text. By default Volto creates a new text tile on RETURN. Rodrigo also looked into the Add-ons control panel work that Eric Steele started some time ago.

Nilesh joined us remotely from India and continued his work on the users and groups control panel for Volto.

Steffen fixed a bug in the subjects search. Rodrigo and Thomas looked into our Cypress-based acceptance tests.

Fred looked into what consequences it would have for us to get rid of the description tile and how we could customize the domain in Plone to allow to send portal emails via the Volto frontend.

Stefania and Maurizio worked on a product definition of Volto and an elevator pitch for clients and users. Together with Paul, Fred and Timo, they had a longer discussion about how to position Volto and Plone 6 in the market.

Plone REST API

The Plone REST API builds the foundation for the new Volto frontend and Plone 6. It has been under active development in the last four years and seen more than 80 releases in that time period. The REST API is stable, battle-tested and production-ready and part of the Plone core since the release of Plone 5.2.

Thomas Buchberger and Lukas Graf led the REST API efforts during the sprint and worked with Carsten Senger, Roel Bruggink, Janina Hard and Timo Stollenwerk to further enhance the REST API, fix bugs, improve the documentation, and the error handling.

API Team in the graden Plone REST API discussion in the garden

Querystring Endpoint for the Collection Type

One of the most important new features, that has been developed during the sprint, is the new querystring endpoint. This endpoint will allow us to implement the Collection type for the Volto frontend. Lukas implemented this essential feature with help from Rob, who took care of the frontend implementation in Volto.

New Features, Bugfixes, and first time contributions

Thomas added support for the retrieval of additional metadata fields in summaries in the same way as in search results. Lukas made the @types endpoint expandable and the @users endpoint easier to customize. Janina fixed setting the effective date on workflow transitions. This was her first contribution to the Plone core. On the same day she signed the contributor agreement, got her fix merged and released, which is quite an accomplishment for a single day. Way to go Janina!

In addition to that, the team fixed lots of smaller bugs (#780, #729, #764, #755, #777, #732, #738) during the sprint. REST API Error handling Carsten, Nathan, Thomas, Lukas and Timo had a longer discussion about error handling in the Plone REST API and about how we can improve things. We considered and discussed implementing RFC7807 and decided against adopting it.

Developer in office Sprint discussions…

The RFC did not see much adoption and it only slightly differs from what we are already doing. A breaking change does not seem to be worth the effort. We decided to create a separate error reporting component unifying the error reporting and ensuring consistency. We also started to fix a few inconsistencies right at the sprint.

API Documentation

Guillotina is using Swagger to document its API. Nathan explained the approach he took for that and Carsten started to integrate Swagger to generate dynamic api docs in plone.restapi.

REST API Releases

Most of the work we did on Plone REST API during the sprint has been released in Plone REST API 4.1.4, 4.2.0, 4.3.0, and 4.4.0.

Dexterity Site Root

Roel Bruggink joined us on Saturday and Sunday to continue his work on turning the Plone site root into a Dexterity content object. He fixed some complex recursion and acquisition problem when migrating the site root to Dexterity. His Plone Improvement Proposal (PLIP #2454) has been accepted by the Plone framework team and it will build the foundation for a new Plone 6 branch that we plan to cut this year.

Guillotina

Guillotina is a new AsyncIO REST data API. It has been written from the scratch by Ramon Navarro Bosch and Nathan van Gheem. It can easily scale billions of objects via its REST API. With the CMS addon, it is compatible with the Plone REST API. Therefore you can run Volto on a Guillotina backend and Guillotina might be able to replace our existing Plone backend in the future.

Guillotina Developer The Guillotina master thinking hard. :)

Nathan and Ramon worked on Guillotina 5. The Guillotina 5 release brought Python 3.7 support and the use of the context vars APIs. They refactored quite a bit, moved parts of addons into core(swagger, caching, pg catalog), implemented OOTB PostgreSQL catalog, added a Helm/Kubernetes/Docker configurations for the Guillotina CMS addon that can be configured to act as a Volto backend, worked on improving caching and filestorage.

Websockets for Plone

At the Plone Open Garden 2019, Asko Soukka started to work on bringing websocket support to Plone. Websockets allow bi-directional communication between the server and the browser. With websockets, the server can send messages and notifications to the browser. This can be used to implement notification systems or collaborative editing to just name a few possible applications.

Asko continued his work during the Beethoven sprint and created an example where comments that are added to Plone automatically pop up as toast notifications for other users. He published a Docker image with a Twisted-based ZServer and implemented the foundation to use that feature for gatsby-source-plone. This will allow us to create an extension that will allow editors to live-preview changes they do in Plone in a GatsbyJS site.

Developer in Office

Karaoke, Barbecue and Garden Parties

A Plone sprint is much more than just programming. Collaborating, meeting old friends, talk, discuss and having a good time together are the ingredients that make a Plone sprint. We started our sprint on Thursday with a free React and Volto training for newbies. At the evening when everybody arrived, we went to a local Karaoke bar for a revival of the epic Karaoke sessions we had in Tokyo last year.

Karaoke Karaoke at the “Nyx”

On Friday evening we ordered Pizza and hacked the night away at the office and our garden. On Saturday we went to the “Arithmeum”, a mathematics museum owned by the Research Institute for Discrete Mathematics at the University of Bonn.

Arithmeum Our tour at the “Arithmeum”

Afterwards we had a garden party with a barbecue and we spend the afternoon sitting in the garden and the office and continued to drink and hack on Plone.

Developer in garden Barbecue in the garden of the kitconcept office

Summary

The Beethoven Sprint 2019 was a major success. We were able to start working on the main missing pieces for Volto. The new edit toolbar is in place and the foundation for the new tiles/metadata toolbar on the right side of the UI has been finished. We also implemented a new table tile.

We implemented a proof-of-concept for the new listing / collection tile and implemented the required querystring endpoint in Plone REST API. Both Volto and the Plone REST API got lots of enhancements and bugfixes.

Both the Plone REST API and Volto are stable and used in production. We will use the next month to further enhance Volto and implement the missing bits and pieces. Our plan is to present Volto 4 at the Plone conference in Ferrara at the end of this year and cut a Plone 6 branch afterwards that contains the Dexterity Site root, folderish content types and other enhancements that are needed make Volto the default frontend for Plone 6.

Guillotina and the Websockets support for ZServer together with Gatsby as another frontend for Plone are amazing projects by the smartest folks in our community. Our diversity is one of our main strength and we are looking forward what the future will bring!

We had a great time at the sprint. Thank you to everyone who attended! We thank the Plone and Python community for their ongoing support. The Plone Foundation, the Python Software Foundation, and the German Python Software Verband made this event possible with their sponsorship as well as our company sponsors Abstract, Starzel, Werkbank and Zest.

We are looking forward to see you all in Bonn next year!

Developer in garden The “Hofgarten” near the kitconcept office

Digital Signage management with Plone and GatsbyJS

Posted by Asko Soukka on September 01, 2019 09:00 AM

Sometimes complex problems have simple solutions, and this was one of those times. At work, we are piloting a new digital signage solution for our university. Unfortunately, the new system lacks in permission management, and for a while, it looked like we were unable to safely delegate digital signage content management to departments.

Lucky us that we knew a system that is very good in permission management: Plone.

Plone – An enterprise open source content management system

Plone is an open source content management system based on hierarchical object database. That’s why Plone excels in managing many kind of content and related permissions in hierarchies. That’s also why Plone has been a good fit for such hierarchical organizations where content management follows the hierarchy of the organization itself.

Plone excels in managing content and related permissions in hierarchies

So, how we are going to solve the shortcomings of our digital signage system with Plone and GatsbyJS? In brief, we

  • let our users to manage their digital signage content on Plone in the simples possible way: every display group has their own folder on Plone and the responsible users have required rights to manage content in their folders

  • we use GatsbyJS project to fetch that content from Plone and build simple digital signage content players hosted as static web pages

  • we use our on-premises GitLab CI and GitLab pages to build and host those web pages; a Plone feature called content rules is used to trigger a new updating build after each content update.

With this setup, instead of letting the content managers to directly manage the digital signage system (where we were unable to restrict their permissions to their own display groups), we configure the system to display those GatsbyJS built sites and let the users manage their digital signage content on Plone – on the content management system they are already familiar with.

Simple digital signage content loop can be populated with just dropping images into a Plone folder

And that’s just the beginning. Thanks to Plone’s customizable structured content types, we already added new content field for setting the display time of each content page. And in the future we can use this to add more configuration options to ensure that the content is displayed as intended by the content managers.

Plone supports customizing structured content fields directly form the browser

Also built-in content types like first class Image type can be enhanced with custom fields

gatsby-theme-ds-player

Show the code or it didn’t happen!

To be honest, our current solution is so simple that there is not that much to show.

Anyway, I put together a simple open source version where similar GatbyJS built web site is created from given Plone folder so that each subfolder has its own digital signage player to loop through the images dropped into that folder.

I packaged all this into a GatsbyJS “theme” plugin gatsby-theme-ds-player preconfigured to be best compatible with a Plone source using gatsby-source-plone. There is also a live demo of the result, automatically built from the theme package repository at Travis-CI:

Using Glitch for Pyramid Learning

Posted by Jazkarta Blog on May 09, 2019 08:02 PM

Screenshot: Pyramid Quick Tutorials showing pyramid-view-decorators, pyramid-view-predicates, pyramid-view-renderers and pyramid-asset-specifications

At this year’s Jazkarta sprint we decided to invest some time in creating Pyramid learning resources. We are fans of the Pyramid web framework, and we felt that it would be nice to work on something that would make it easier to get started with, especially for novice developers.

Enter Glitch. This is a platform that is designed for letting users explore web apps created by other Glitch users. When you see an application that does something that you want to do, you can “remix” it, which means that an exact copy, with all its code and resources, is created in your own workspace (which is some sort of Docker container). You can then change it in any way you like. This is very powerful, because you get a working application that is hosted for you, and every code change that you make is immediately reflected in the web app page.

This turns out to be really useful for teaching web application development concepts, because students can see the code for the example, read an explanation, and then right away remix the code to try out their changes, which they can see working instantly. No need for setting up a workspace, checking out code, or running a web server. Just try out an idea and see the result right there, or share it with other people. Like “view HTML source” but with code, which is one of Glitch’s goals.

We decided to create a Glitch collection that would showcase a few of Pyramid’s most distinctive features, along with a simple Hello World style tutorial. The first thing was to verify that Pyramid could be installed. Glitch’s main development target is Node.js, but its containers install Python and they allow the user to run pip, so it’s possible to install most PyPI packages including Pyramid. (For those who might be wondering: sorry, Plone is not installable, because it needs system dependencies that are not installed by Glitch.)

Glitch supports a setup file named glitch.json, which is what allowed us to install Pyramid:

{
 "install": "pip3 install --user pyramid",
 "start": "python3 pyramid_app.py"
}

In the json above, you can see that we install Pyramid using pip, and we can add any other dependencies for our project there. We can then use the “start” key to tell Glitch how to start our application.

Once we did that, the only other thing needed was to create our Pyramid app, which in this case uses code directly lifted from the trypyramid.com site:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response

def hello_world(request):
    return Response('Hello Glitch')

if __name__ == '__main__':
    with Configurator() as config:
       config.add_route('hello', '/')
       config.add_view(hello_world, route_name='hello')
       app = config.make_wsgi_app()
    server = make_server('0.0.0.0', 3000, app)
    server.serve_forever()

That’s it. Glitch automatically runs the application (the only caveat is that it needs to use port 3000). You can configure it so that any code change automatically restarts the app. After this, if a potential new user sees your project, she can just use the remix button and have a copy made in her workspace. Any change that she makes, like changing the hello text, will be immediately visible on the page.

Overall, we found Glitch to be a very intuitive and easy to use service. You can see the collection of simple tutorials that we created here:

Pyramid Quick Tutorials

 

Jazkarta Goes To LA

Posted by Jazkarta Blog on May 09, 2019 07:47 PM

The Jazkarta team at LAX-C, a city block sized Asian food distributor and restaurant supply

This year Jazkarta’s annual sprint was held in the city of Los Angeles, graciously hosted by Alec Mitchell who lives a block away from LA’s City Hall.

View of the Los Angeles City Hall from Los Angeles Street

We had fun exploring the city – Union Station, Chinatown, LAX-C (pictured above, a gigantic Asian food distributor and restaurant supply), Grand Central Market, Griffith Observatory, and the Hollywood Farmers’ Market where we bought lots of delicious fruit.

Citrus and strawberries from the Hollywood Farmers' Market

We attended an improv comedy performance at the Upright Citizens Brigade Theater, ate lots of tacos, had original Phillipe’s French Dip sandwiches, visited brew pubs and drank lots of local beer.

The Jazkarta team in front of the Highland Park Brewery.

We even had a private tasting of cheese, beer, and wine pairings.

Plate with five different cheeses and a variety of fruits and nuts

Oh, and we also sprinted on a variety of topics!

Alec and Jesse Snyder ported several of our Plone add-ons to Python 3 and Plone 5.2:

Carlos de la Guardia created a set of Pyramid Quick Tutorials on Glitch – more about that coming in another post.

Matthew Wilkes and Jesse researched the best way to set up Pyramid functional tests using pytest and webtest in an efficient manner. They were unhappy with their previous model, which relied on in-memory databases. They settled on a pattern that switched out private variables of the sessionmaker to join a longer-running transaction for each test. This is more self-contained than adapting the session setup machinery in the app itself, and works transparently on different database backends. Matthew made a gist of the relevant code.

Alec made a prototype of an app to provide faceted search of bibliography references kept in Zotero. The app has a Elasticsearch backend and a React front end. It uses the Zotero API to pull the references into Elasticsearch for querying.

Carlos investigated what it would take to move the CastleCMS quality checking features into a Plone add-on. It’s a work in progress.

Left to right: Carlos, Sally, Matthew, Alec, Jesse and Nate in front of a painting of the Griffith Observatory

It’s wonderful to be able to work together in person, instead of remotely from our home offices spread across 3 countries and 2 continents. We try to do it once a year and always have a fantastic time. This year was no exception.

Sprinting in Sorrento

Posted by Jazkarta Blog on May 08, 2019 10:44 PM

Sunset over the island of Capri, as viewed from the sprint venue

This year I helped organize the annual sprint in Sorrento called the Plone Open Garden. Plone is a highly secure and feature rich open source content management system written in Python. Plone has no single company behind it, its IP is held by a non-profit foundation. Progress on the software happens through the efforts of a self-directed community, whose members sponsor sprints to work on particular problems.

PLOG took place April 12-19 and it was a great success, attended by 18 people from 7 countries. There were two training classes:

  • Porting Plone sites and add-ons to Python 3
  • Creating a web application using the restapi and Volto, the new React front end for Plone

We also accomplished a lot of work:

  • Python 3 ports of over a dozen Plone add-ons
  • Accessibility improvements for classic Plone and Volto front ends
  • New Volto features and documentation
  • Marketing discussions
  • Documentation discussions
  • …and various other Plone improvements

I posted a news item on plone.org that describes the details of what we accomplished. But Plone sprints have a social side as well as a technical side. Friendships are renewed, meals and news are shared, and a good time is had by all. This year’s PLOG was true to form. Here are some of the things we enjoyed.

1) Breakfasts at the Hotel Mediterraneo, the sprint venue since 2007. Pictured below is the pastry table; there are also tables for cheeses, meats, fruits, breads, cereals, eggs, and more. You can squeeze your own orange juice and optionally add some Prosecco.

Breakfast buffet at the Hotel Mediterraneo - the pastry table

Dinners, also included, are a wonderful way to end a day of sprinting. First and second course, plus dessert.

A typical dessert at the Hotel Mediterraneo

2) Happy hour at the Mediterraneo. We book the rooftop for happy hour at least once during the sprint, where we enjoy the sunset over Capri (pictured at the top of this post) and the extensive cocktail menu. That’s a Black Mohito at the top and a Negroni at the bottom.

 Drinks from the Hotel Mediterraneo cocktail menu

3) Excursions. One day we walked to Sorrento’s old harbor and had lunch with a view of Mount Vesuvius.

A view looking down at Sorrento's old harbor

4) Being with old friends and meeting new ones. Always the best part.

Sprinters on the rooftop of the Hotel Mediterraneo

 

Four options to try Plone 5.2 on Python 3

Posted by Starzel.de on January 24, 2019 08:30 AM

Demo pages

There are nightly build of the current Plone 5.2 coredev (the development version) with Python 2 and 3:

Minimal Buildout

Here is a minimal buildout to run Plone 5.2rc1 on Python 3:

[buildout]

parts = instance

extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg



[instance]

recipe = plone.recipe.zope2instance

eggs =

Plone

Pillow

You set it up like this:

$ python3.7 -m venv .

$ ./bin/pip install -r https://dist.plone.org/release/5.2rc1-pending/requirements.txt

$ ./bin/buildout

And start it as usual with ./bin/instance fg

Standalone Development buildout

[buildout]

extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg



parts =

    instance

    zopepy

    packages

    test

    robot



eggs =

    Plone

    Pillow

    collective.easyform



test-eggs =

    collective.easyform [test]



auto-checkout =

    collective.easyform



extensions =

    mr.developer



show-picked-versions = true



[instance]

recipe = plone.recipe.zope2instance

user = admin:admin

eggs = ${buildout:eggs}

debug-mode = on

verbose-security = on



[zopepy]

recipe = zc.recipe.egg

eggs =

    ${buildout:eggs}

interpreter = zopepy

scripts =

    zopepy

    plone-compile-resources



[packages]

recipe = collective.recipe.omelette

ignore-develop = False

eggs = ${buildout:eggs}

ignores = roman



[test]

recipe = collective.xmltestreport

eggs = ${buildout:test-eggs}

defaults = ['--auto-color', '--auto-progress']



[robot]

recipe = zc.recipe.egg

eggs =

    ${buildout:test-eggs}

    Pillow

    plone.app.robotframework[reload,debug]



[sources]

collective.easyform = git git@github.com:collective/collective.easyform.git branch=python3



[versions]

Starzel buildout

The buildout that we at Starzel.de use supports Plone 5.2rc1 with Python 2 and 3.

https://github.com/starzel/buildout

It has some nice features:

  • It extends to config- and version-files on github shared by all projects that use the same version of Plone.
  • It allows to update a project simply by changing the version it extends.
  • It allows to update all projects of one version by changing remote files (very useful for HotFixes).
  • It is minimal work to setup a new project.
  • It has presets for development, testing, staging and production.
  • It has all the nice development-helpers we use.

Quickstart:

$ git clone -b 5.2rc1.x https://github.com/starzel/buildout <SOME_PROJECT>

$ cd <MY_PROJECT>

Remove all files that are not needed for a project but are only used for the buildout itself.

$ rm -rf linkto README.rst README.txt .travis.yml secret.cfg_tmpl VERSION.txt local_coredev.cfg CHANGES.rst

If you're not developing the buildout itself you want a create a new git repo.

$ rm -rf .git && git init

Add a file that contains a passwort. Do not use admin as a password in production!

$ echo -e "[buildout]\nlogin = admin\npassword = admin" > secret.cfg

Symlink to the file that best fits you local environment. At first that is usually development. Later you can use production or test. This buildout only uses local.cfg and ignores all local_*.cfg.

$ ln -s local_develop.cfg local.cfg

Create a virtualenv in Python 2.7 or Python 3.7 (Plone 5.2 only).

$ virtualenv .  # for Python 2.7

$ python3.7 -m venv .  # for Python 3 (Plone 5.2 only)

Install and configure Plone

$ ./bin/pip install -r requirements.txt

$ ./bin/buildout

Wrapup

I hope these options help you get started with Python 3. For serious projects you will likely create you own buildout.

Update (4.2.2019)

You should use the Plone coredev for developing Plone itself:

$ git clone git@github.com:plone/buildout.coredev.git coredev
$ cd coredev
$ git checkout 5.2
$ python3.7 -m venv .
$ ./bin/pip install -r requirements.txt
$ ./bin/buildout
$ ./bin/instance fg

Update 2 (24.2.2019)

Since Plone 5.2b2 wsgi is the default, even when running Python 2. See https://github.com/plone/Products.CMFPlone/issues/2763

This means you do not need to enable wsgi = on since it will be enabled automatically. You can still choose to switch wsgi = off in Python 2 if you have a good reason to do so (e.g. you need FTP or WebDAV). I updated the examples accordingly.

Update 3 (8.3.2019)

Release candidate 1 of Plone 5.2 is pending. See https://community.plone.org/t/plone-5-2rc1-soft-released/8163. I updated the examples accordingly.

Plone finally supports Python 3!

Posted by Starzel.de on November 29, 2018 09:21 AM

At the Plone Conference 2018 in Tokyo I gave a plenary talk in which I told the story how Plone got to support Python 3. Ten minutes before my talk Plone 5.2a1 was released. That release was the culmination of a 3-year long journey (or even 5 years if you start in Brazil). If you want to learn how we got there you should watch the talk.

Where are we now?

Plone 5.2 supports Python 3.7, 3.6 and 2.7. It is currently in alpha but the plan is to release a final version in February 2019 right after the Alpine City Sprint. You should seriously consider joining that sprint if you plan to migrate your own projects. Working together with the people who ported Plone to Python 3 will give you the know-how to succeed in your own migrations!

But in fact you should not wait until February to start working with Plone on Python 3 but start right now so you'll have enough time to migrate to Python 3 before January 1st 2020 (End of Life of Python 2).

To make testing and working with Python 3 easier some tools and addons were already working on Python 3.

The following essential development-tools are already ported:

Also the three addons that achieved the highest ranking during the latest poll at Ploneconf now work on Python 3:

You can also run the new frontend Volto with Plone on Python 3.

Open tasks

Before the final release of Plone 5.2 scheduled for February 2019 there are still a lot of things to do.

Here are the six most pressing issues:

  1. The upgrade- and porting-guide need to be completed.
  2. ZODB migration needs to be finished and documented.
  3. Much more addons need to be ported and released.
  4. We require performance tests.
  5. People who really need a replacement for FTP and/or WebDAV need to figure out how to do that.

There is still a lot to do. You can help Plone by coming to a sprint, testing it and fixing or reporting a bug.

Migrating to Python 3

From 1.1.2020 on you'll have to inform your clients that you are running their super-secure system on an unsupported version of Python. That might be ok if you're running a small website for a friend but for serious projects this simply will not do. Instead you should start to adopt Python 3 in your Plone projects now. You need to start early and plan ahead!

Here are the six steps you have to follow to upgrade to Python 3. Do not attempt to do all steps at once, instead you should work iteratively and even deploy to production in between steps.

1. Upgrade to Plone 5.2

You should still use Python 2.7 for this step. The changes between Plone 5.1 and Plone 5.2 are not huge but you will gain some great features including a new and much faster dropdown-navigation and a bootstrap-based ZMI.

The Upgrade-Guide for 5.2 still needs much more love. So if you encounter any issues please update that document and/or create a ticket in github.

2. Drop Archetypes and only use Dexterity

Archetypes is still supported on Plone 5.2 but only when running with Python 2.7. It is now officially deprecated and is not ported to Python 3. There has been a built-in migration for default types from Archetypes to Dexterity since 2013, and since 2015 there have been helpers and even a form that allows you to migrate your custom content.

3. Migrate your code to Python 3

Follow the documentation in to migrate your custom code and any addons that you need in your packages.

  • Make sure your code works in Python 2 and Python 3. If you are certain that you will not publish that code as a addon or reuse it in another project you can also drop support for Python 2. That will make porting and testing easier.
  • For most addons: Use python-modernize as described in the documentation.
  • For small addons you could even simply try to startup on Python 3 and and fix whatever fails (e.g. relative imports or invalid syntax)
  • Manually fix whatever python-modernize misses (e.g. relative imports)
  • Do the same with tests. You'll need a test-setup with tox and a test-matrix in travis that tests different Python versions. The setup in collective.ifttt is almost done and might be a good example.
  • If you have complex doctests consider migrating them to python tests since these are easier to get to pass in both Python 2 and 3. If you do want to keep them as doctests, change them so that the output for Python 3 is the default and use a Py23DocChecker to make the tests pass in Python 2 as well. See the example in plone.indexer.
  • Once all seems to work you need to test every single feature manually because even a high test coverage is never a guarantee that your code actually does what it should.

4. Migrate Addons and Dependencies

Now migrate any addons you use to Python 3 as explained above. The main difference is that now everyone who uses this addon benefits. You shoud not drop support for Python 2 for those!

I listed the addons that already run on Python 3 above. I'm sure that list will be outdated very soon. Before you start migrating an addon you need to check if there already is a branch that supports Python 3 and check if there is a ticket like the one for collective.easyform. If not create such a ticket and start the process.

5. Migrate your Database

ZODB itself is compatible with Python 3 but a database that was created in Python 2.7 cannot be used in Python 3 without being modified first. I will not go into details about that. You should read the documentation and see David Glick's talk.

The migration will probably require some downtime and in rare cases you might have to write your own mappings to tell zodbupdate how to handle your data.

6. Deploy on Python 3

Plone 5.2 will use WSGI with waitress as the default WSGI server. Alternatively you can use uwsgi or gunicorn. ZServer can only be used with Python 2.7.

Some deployment tools may not ported yet, others like ZRS or RelStorage already support Python 3. You should test your production setup early. There may be changes that you don't want to struggle with a week before a scheduled launch.

Summing up

Start that whole process as early as possible because you may need to deal with unexpected problems and addons that you did not know still used Archetypes. 2019 might be much busier for you than you expect.

Do not wait for a final release of Plone 5.2! Start the process of planning and testing a migration. Consinder hiring experts to help you if necessary.

Again: You can and you should work iteratively. That means you do not have to do everything in one step and one night:

  • Upgrade to 5.2 with Dexterity on Python 2 and deploy that early in 2019.
  • Then take some time to migrate and test your code and addons.
  • You can then deploy the Python 3-compatible code while still running Python 2.7.
  • Then you can migrate your existing database and deploy on Python 3.

FAQ

Here are some questions you might have:

What about Python 4?

That will not be a big issue since 4.0 will be a upgrade like from 3.7 to 3.8. The Python community simply don't want to release a Python 3.10.

When will Plone drop support for Python 2?

Not before we have a good reason. Maybe Plone 7? It would allow us to remove and clean up some code but we would not gain a lot.

What about Zope 5?

There are no real plans for that yet.

Will Archetypes be migrated to Python 3?

Please don't try to postpone the inevitable. We are too small a community to maintain two Python versions, two frontends and two content type frameworks.

How do we replace FTP/WebDAV?

I don't know, for WebDAV you might be able to use a middleware like WsgiDAV.

Why did you make all that effort and not 'simply' migrate to Guillotina?

Guillotina does not aim to be a replacement for Plone. Also it has no equivalent addon ecosystem, feature equality or migration path.

Can I now use async/await?

You can but it won't really help you since Plone and Zope are still fundamentally synchronous.

Why doesn't the Plone Foundation simply buy support for Python 2.7 from RedHat/IBM for all of us?

Are you kidding me?

Does Python 3 make Plone run faster?

Maybe. We still lack performance tests but my guess is that Plone 5.2 will run faster on Python 3 than on 2.

Can I run the same Database in Python 2 and 3?

No.You can neither use a DB create with py2 in py3 nor use one created with py3 in py2.

Is addon xyz already ported?

Most likely not.

Plone supports the latest two major version. Does that extend to Python?

Plone's security update policy supports the latest two major versions (currently, that means the 5.x series and the 4.3.x series) with security updates. That guarantee does not extend to the programming language, the operating systen or the hardware you are running Plone on.

Will this work on Windows?

Very likely. Someone will have to test it.

I need help with migrating our projects to Python 3. Who can help me?

Talk to us at Starzel.de or ask other professionals with experience.

Warm words

Migrating Plone to Python 3 was quite a ride and a great community effort. I had to cope with some brain-melting changes in Zope, fight heartbreaking test-isolation issues and toil through an endless list of packages and tests. But I had the immense privilege and pleasure to do all of that together with some of the smartest and nicest people I know. Thanks to all of you!

Watch the full talk if you want to learn more about how Plone finally landed on Python 3.

Here are all videos from Ploneconf 2018: Day 1, Day 2, Day 3.

Help Oshane get to the Tokyo Plone Conference

Posted by David "Pigeonflight" Bain on October 24, 2018 03:38 AM

Summary

Oshane Bailey, a talented Plone developer with loads of Plone experience has been selected as a presenter for the 2018 Tokyo Plone Conference. His Japanese visa was just approved.
He will share a streamlined approach to Plone development that he is applying on a Plone project targeted at Jamaican Developers. At the time of writing he has raised about 17% of the funds he needs to get to Tokyo. You can help him get to Japan by contributing to his crowd-funding campaign.

Oct 29, 2018 update
Thanks to generous contributions, Oshane's trip is now 70% funded. You are welcome to join the crowdfund and cover the rest of his trip.

Oct 30, 2018 update

Oshane's trip is now 89% funded. The plane ticket and conference ticket have been purchased. you can still pitch in by joining the crowdfund to cover the rest of his trip.

Since at least 2015, Oshane has worked on Plone projects for teams around the world and in the process has been exposed to varied approaches to the development and ongoing management of Plone sites. Over recent months he has poured his, hard earned, experience, into a side project -- the Jamaican Developers site.  Through this project he has refined a continuous development pipeline based on some of the best techniques used in the Plone community and enhanced with some of his own innovations.

Last year Oshane participated as a Plone Google Summer of Code student and presented his work at the Barcelona conference. He also participated in the after-conference sprints, contributing to efforts to port Plone to Python 3 and also looking into the WebSauna project.

Supporting his trip to Tokyo will serve to enrich PloneConf 2018 in many ways. Here are three that spring immediately to mind:
1) As part of his talk he will share the techniques he is using on the Jamaican Developers site
2) He plans to participate in the after conference sprints.
3) He will bring an important perspective to discussions influenced by constraints common to Jamaican developers.

How to Support Oshane

Appropriately, his crowdfunding campaign is running on the Jamaican Developers site that he built with Plone. His goal is to raise enough to cover his travel and expenses related to the Japan trip.

As we say in Jamaica... "Follow back a me" as I support Oshane's trip to PloneConf2018 in Tokyo.

Python 3, Plone 5.2, Zope 4 and more

Posted by Starzel.de on October 19, 2018 09:14 AM

With over 30 attendees the SaltLabs Sprint was among the bigger sprints in the history of Plone and Zope.

Fortunately, our host Gocept just moved into a new office and had plenty of space to spare. Stylish as we are, the Plone crowd quickly congregated in the palace-like salon with red walls and renaissance paintings. Even better: Throughout the week the cafe Kaffej downstairs was open for us to eat breakfast, lunch and for work. We tried to keep the barista busy producing excellent Cappuccino. Thanks!

Zope 4

The sprint split in two groups, one working on Zope and one working on Plone. A short report on about what happened at the Zope can be found at https://blog.gocept.com/2018/10/11/beta-permission-for-earl-zope-extended . Zope 4.0b6 was released just after the sprint. The new Beta 6 includes the new bootstrap-styled ZMI, support for Python 3.7 and a metric ton of bug fixes compared to 4.0b5 which was released in May this year. By the way: Zope can now be installed without buildout using pipenv

Python 3

The Plone crowd was huge and worked on several tasks. The biggest task - and the main reason to join the sprint - was continuing to port Plone to Python 3. Jens Klein, Peter Mathis, Alessandro Pisa, David Glick (remote) and Philip Bauer as lead worked mainly on fixing all remaining failing tests and getting the jenkins-builds to pass. We dove deep into crazy test-isolation-issues but also added a couple of missing features to Dexterity that were so far only tested with Archetypes.

During the sprint we got very close to a green build but a couple of really nasty test-isolation issues prevented a green build. A week after the sprint we finally got that done and all tests of Plone now pass in Python 2.7, Python 3.6 and Python 3.7 with the same code-base!

With the test passing we're now merging that work and preparing the release of Plone 5.2a1. The tasks for this are discussed in this ticket

Deprecate Archetypes

Archetypes will not be ported to Python 3 but still has to work when running Plone 5.2 in Python 2.7. Since all tests that used PloneTestCase were still using Archetypes Philip changed that to use Dexterity now and created a new testlayer (plone.app.testing.bbb_at.PloneTestCase) that is now used by all the packages using the Archetypes-stack. See https://github.com/plone/plone.app.testing/pull/51. You could use this layer in your archetypes-based addon-packages that you want to port to Plone 5.2.

Addons

Yes, Plone is now able to run on Python 3. But without the huge ecosystem of addons that would be of limited real-world use. Jan Mevissen, Franco Pellegrini (remote) and Philip Bauer ported a couple of addons to Python 3 that Plone developers use all the time:

Being able to run tests against Plone in Python 3, and with these development-tools now available also, it is now viable to start porting other addons to Python 3. To find out how hard that is I started porting collective.easyform. The work is only 97% finished since some tests need more work but it is now useable in Python 3!

ZODB Migration

Harald Friesenegger and Robert Buchholz worked on defining a default migration-story for existing databases using zodbupdate. To discuss the details of this approach we had a hangout with Jim Fulton, David Glick and Sylvain Viollon. They solved a couple of tricky issues and wrote enough documentation. This approach seems to work well enough and the documentation points out some caveats but the ZODB migration will require some more work.

Relevant links:

ZODB Issues

Sune Broendum Woeller worked on a very nasty and complex issue with KeyErrors raising while releasing resources of a connection when closing the ZODB. That happened pretty frequently in our test-suites but was so far unexplained. He analyzed the code and finally was able to add a failing test to prove his theory. Then Jim Fulton realized the problem and wrote a fix for it. This will allow us to update to the newest ZODB-version once it is released. See https://github.com/zopefoundation/ZODB/issues/208 for details.

WSGI-Setup

Thomas Schorr added the zconsole module to Zope for running scripts and an interactive mode. Using WSGI works in Python 2 and Python 3 and will replace ZServer in Python 3.

Frontend and Theming

Thomas Massman, Fred van Dijk, Johannes Raggam and Maik Derstappen looked into various front-end issues, mainly with the Barceloneta theme. They closed some obsolete tickets and fixed a couple of bugs.

They also fixed some structural issues within the Barceloneta theme. The generated HTML markup now has the correct order of the content columns (main, then left portlets, then right portlets) which allows better styling for mobile devices. Also in the footer area we are now able to add more portlets to generate a nice looking doormat. See the ticket https://github.com/plone/plonetheme.barceloneta/pull/163 for screenshots and details.

They also discussed a possible enhancement of the Diazo based theming experience by including some functionality of spirit.plone.theming into plone.app.theming and cleaning up the Theming Control Panel. A PLIP will follow for that.

New navigation with dropdown support

Peter Holzer (remote) continued to work on the new nativation with dropdown-support which was started on collective.navigation by Johannes Raggam. Due to the new exclude_from_nav index and optimized data structures the new navigation is also much faster than other tree based navigation solutions (10x-30x faster based on some quick tests).

Static Resource Refactoring

Johannes Raggam finished work on his PLIP to restructure static resources. With this we no longer carry 60MB of static resources in the CMFPlone repository, it allows to use different versions of mockup in Plone, enable us to release our libraries on npm and will make it easier to switch to a different framework. Skin scripts removal Katja Süss, Thomas Lotze, Maurits van Rees and Manuel Reinhardt worked hard at removing the remaining python_scripts. The work is still ongoing and it would be great to get rid of the last of these before a final Plone 5.2 release. See https://github.com/plone/Products.CMFPlone/issues/1801 for details. Katja also worked on finally removing the old resource registry (for js and css).

Test parallelization

Joni Orponen made a lot of progress on speeding up our test-runs by running different test-layers in parallel. The plan is to get them from 30-60 minutes (depending on server and test-setup) to less than 10 minutes. For a regularly updated status of this work see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225

Documentation and User-Testing

Paul Roeland fought with robot-tests creating the screenshots for our documentation. And won. A upgrade guide to Plone 5.2 and Python 3 was started by several people. Jörg Zell did some user-testing of Plone on Python 3 and documented some errors that need to be triaged.

Translations

Jörg Zell worked on the german translations for Plone and nearly got to 100%. After the sprint Katja Süß did an overall review of the german translation and found some wording issues with need for a discussion

Plone React

Rob Gietema and Roel Bruggink mostly worked on their trainings for React and Plone-react, now renamed to “Volto”. Both will be giving these trainings at Plone Conference in Tokyo. On the second day of the sprint Rob demoed the current state of the new react-based frontend for Plone.

Assorted highlights

  • In a commit from 2016 an invisible whitespace was added to the doctests of plone.api. That now broke our test in very obscure ways. Alessandro used some dark magic to search and destroy.
  • The __repr__ for persistent objects changed breaking a lot of doctests. We still have to figure out how to deal with that. See https://github.com/zopefoundation/zope.site/issues/8 for details.
  • There was an elaborate setup to control the port during robot-tests. By not setting a port at all the OS actually takes care of this to makes sure the ports do not conflict. See https://github.com/plone/plone.app.robotframework/pull/86. Less is sometimes more.
  • Better late than never: We now have a method safe_nativestring in Products.CMFPlone.utils besides our all-time favorites safe_unicode and safe_encode. It transforms to string (which is bytes in Python 2 and text in Python 3). By the way: There is also zope.schema.NativeString and zope.schema.NativeStringLine.
  • We celebrated the 17th birthday of Plone with a barbecue and a generous helping of drinks.