Planet Plone - Where Developers And Integrators Write

On The Road to Plone 6 - Plone REST API 7 and Volto 12 Released!

Posted by kitconcept GmbH on April 12, 2021 09:24 AM

stephen leonardi 5CH1TNfcZoo unsplash Photo by Stephen Leonardi on Unsplash

Two weeks ago we cut two important releases on the road to Plone 6. Plone REST API 7 introduces a new link integrity feature for blocks-based pages. Volto 12 improves the add-on ecosystem by introducing a new configuration registry to avoid circular dependencies.

Plone REST API 7

Linking pages is one of the core idea of the world wide web. Keeping links within a website intact is therefore one of the core features that any Content Management System needs to provide.

One of the core features of Plone has always been that editors can copy and move single pages as well as large content trees without breaking internal links to other parts of the site. This is accomplished by using unigue IDs (UUIDs) instead of relative or absolute paths when adding a link to another page. Plone uses “portal transforms” internally to rewrite those links in RichText fields on save operations.

Plone 6 (aka Volto) introduces blocks-based page layouts, that store a JSON structure internally instead of HTML. This more structured way of storing page content and layout allows more complex page layouts. Though, because of this change, the existing portal transforms mechanism that rewrites links to UUIDs did not work any longer.

volto blocks edit mode Blocks-based Volto page in edit mode (with kitconcept-blocks-grid)

Werkbank, a Plone agency from Bochum, Germany, stepped up to sponsor the development of link integrity in 2020, since they needed that for a client project.

I started to draft a possible solution and wrote a first prototype. Thomas Buchenberger from 4teamwork picked up that work at the Plone Conference sprint in Ferrara, Italy.

Andrea Cecchi from RedTurtle joined our efforts and refactored the resolveUID algorithm into a blocks transformer, that we started to use for other use cases and that made the resolveUID transformer more generic and flexible.

After that, we cut a first plone.restapi 7 alpha release and entered a longer period of quality assurance and testing.

Two weeks ago, after we tested the new feature in multiple projects at kitconcept, I cut a final release of Plone REST API 7.

After the first alpha release, the ResolveUID transformation was added to links and images. In addition plone.restapi 7 comes with a new blocks serialization mechanism and an important fix that makes sure files are opened directly by Plone for anonymous users.

We also added a “smart fields” concept that allows integrators to mark a blocks field as searchableText field.

A new @contextnavigation endpoint was added that allows for local navigations. We enhanced the navigation endpoint to expose an optional navigation title (“nav_title”) field.

Since the final 7 release, we cut six more releases and REST API 7 was included in Plone 5.2.4 release.

Volto 12

gaelle marcel vrkSVpOwchk unsplash Photo by Gaelle Marcel on Unsplash

Tiberiu and Victor worked on a new Volto configuration registry which is the new central point to store and retrieve Volto configurations. The configuration registry is a singleton that ensures a setting only exists once. With more an more Volto add-ons that started to depend on each other, we started to run into circular dependencies issues that we can avoid now with the new configuration registry.

You can find more details about it in the Volto docs and in the Volto 12 upgrade guide.

Plone 6

Plone REST API 7 and Volto 12 were two very important releases on our road to Plone 6.

We plan to cut another Plone REST API 8 branch and release for Plone 6, which will support Python 3 only.

Volto will continue to move at very high pace towards Plone 6. We started to organize a series of Plone 6 “Micro-Sprints” to push things further.

Check out the Volto Roadmap on github for more details.

Plone Training Is Now Available Online

Posted by PLONE.ORG on April 09, 2021 10:08 PM

When the pandemic forced the annual Plone Conference online, the conference training classes went online too. We sure did miss seeing everyone in person, but there's a small silver lining. The online training sessions were all recorded and they have now been posted to the Plone YouTube channel. Now anyone can watch and re-watch them!

The following videos are available. All are about 4 hours long.

Mastering Plone 6 - Part 1

Philip Bauer, Instructor

Learn how to develop custom projects with Plone 6 and Volto, the new React-based frontend. Covers the core technologies involved in Plone 6 programming, including how to write your own add-on package and customize your Plone site by writing Python code and React components. Covers the first half of the material in training.plone.org/5/mastering-plone.

Mastering Plone 6 - Part 2

Philip Bauer, Instructor

A continuation of Part 1. Covers the second half of the material in training.plone.org/5/mastering-plone.

React and Volto - Part 1: React

Jakob Kahl and Alok Kumar, Instructors

Get started with React so you can create your own site using Volto. Learn the basis of React, Redux and React-Router. Covers the material in training.plone.org/5/react/index.html.

React and Volto - Part 2: Volto

Jakob Kahl and Alok Kumar, Instructors

Learn how to how to quickly bootstrap and customize a Volto project, and how create your own website based on Volto. Covers the material in training.plone.org/5/volto/index.html and training.plone.org/5/voltohandson/index.html.

Volto Add-ons - Part 1

Tiberiu Ichim and Víctor Fernández de Alba, Instructors

Learn how to develop Volto add-ons and other useful Volto patterns. Learn how to quickly develop a real world Volto add-on and how to structure your code to make it simple, reusable and provide extensible components. Covers the material in training.plone.org/5/voltoaddons/index.html.

Volto Add-ons - Part 2

Tiberiu Ichim and Víctor Fernández de Alba, Instructors

A continuation of Part 1. Covers the material in training.plone.org/5/voltoaddons/index.html.

Getting Started With Your Plone Site

David Bain, Instructor

Content management principles for Plone. A little background, principles & concepts, logging in & out, preferences and password management, folder management and the basic publication workflow.

Pyramid

Steve Piercy, Instructor

A hands-on, quick tutorial covering "a little about a lot". Practical introductions to the most common features. Fun, fast-paced, and targeted to newcomers to both Python web application development and the Pyramid web framework.

Plone Conference 2020 Videos Now Available!

Posted by PLONE.ORG on April 04, 2021 03:06 PM

Plone Conference 2020 Online

Plone Conference 2020 was a 9-day event for users and developers in the growing Plone ecosystem. Professionals from the Plone, Zope, Volto, Guillotina, and Pyramid communities (see below) participated in trainings, talks, and sprints, at the conference held in December 2020. Speakers illuminated some history of one of the world's foundation open source CMS platforms, built in Python, and dove into demonstrations of the latest future-proof features and visions focused on optimizing speed and agility while retaining Plone's legendary content capacities.

Every training, talk, and presentation was recorded and is now available online on YouTube:

Plone Conference 2020 topics include:

  • Plone - The original, open-source, enterprise-grade, all-in-one content management system written in Python.
  • Zope - The original Python web application server - the foundation for Plone and inspiration for Guillotina.
  • Volto - Plone's snappy, modern React front end powered by RestAPI.
  • Guillotina - A re-imagined asynchronous back end compatible with Plone's RestAPI.
  • Pyramid - Pyramid is a small, fast, down-to-earth Python web framework that continues to gain attention.
  • And more!

All the presentations and info can be found at https://2020.ploneconf.org.

Implementing user epics with BPMN

Posted by Asko Soukka on March 21, 2021 12:00 PM

Many user stories start simple. For example: “A user self registers into a course”. The final version, of course, tend to be more like: “A new user, after being verified to be an authentic real person, creates their new user account, immediately forgets and resets their new password, finally logs into the system, finds the course again, completes payment for the registration, and is then eventually enrolled into the course.” This is no longer a user story, but a series of stories supposed to be completed in a specific sequence. Let’s call these user epics.

An epic is a sequence a user stories

Individual user stories, by their definition, should be relatively straightforward to implement. For example, a stereotypical user story may require interaction with specific user interface element to start the story, following one or more views to gather the required user input to continue the story, finally ending with user’s goal for the story. Where did the user come from, or where does the user continue afterwards, is simply not part of the story. On purpose.

But user epics are different.

Epics must know the consequences of users' actions

An implementation of a user epic must know where the user came from, and must be able to decide, which stories the user should be guided to complete next. Epic should also cover the consequences of user failing to complete a story, or completely abandoning their fate. Therefore implementing an epic is not only about managing state, but also about programming the often complex business logic. Both being among the least fun parts of software development.

As an insult to injury, less and less are users doing their work alone, more and more they are being accompanied by various forms of automation. So, not only should an implemented epic be aware of the actions of its user, but it should also be able to react on known results or side effects of automation – anything from synchronous function calls to asynchronously executed software robots.

With just code, implementations for this kind of epics could get dirty and hard to understand, fast.

Epics must know the consequences of users' actions

Offloading state and logic to BPMN engine

It’s not a coincidence that my example user epics above are drawn as BPMN diagrams. That’s what I have been doing for the last couple of months in our latest project. We had to modernize a process, where users were found to require a lot of hand-holding to be able complete all the required steps and in the right order.

Also, with the previous implementation, when users got lost in the middle of their epics, or an unexpected system error blocked them, it was often surprisingly hard to figure out what really had happened.

This time all this should be different.

The active user task may be anywhere in the current process tree

Whenever a user starts a new epic, a new BPMN modeled process is being started. The model may be as long and complex as needed. It may even be a composition of multiple nested models, resulting in an arbitrarily deep process tree when executed. But this is not a problem, because this is exactly what dedicated BPMN process execution engines have been built for.

The fun part: With a BPMN engine executing our user epics directly from their BPMN models, we don’t really need to “implement” them at all. Instead, we:

  1. Implement user interface routes and views for individual user tasks.
  2. Make the views to read their preconditions from the process engine state.
  3. Eventually complete the task at the engine (with user input when required).
  4. Poll the engine for the next available task and redirect the user there.

Now our user task views are standalone, with no magical dependencies between them to code and maintain.

Plone, Volto and Camunda FTW

I would not be writing this, unless all the above could not be implemented by using open source software only:

  • Because every application seems to need a CMS (sooner or later), we build on top of Plone, the open source CMS we trust. Thanks to its powerful and complete REST API, we can do all the integrations we need with just HTTP requests – no need to know about Plone internals.
  • For modern user experience, we use Volto – ReactJS based web experience on top of Plone REST API (with Razzle based server side rendering support). Every part of Volto is easily customizable and extensible. For example, custom server side routes, like the ones we need to authorize and proxy calls to BPMN engine, are common and well documented use case.
  • Finally, for the BPMN engine, we choose Camunda BPM Community Edition. While the open source version of Camunda is probably missing countless convenient features of their enterprise offering, its BPMN engine and REST API come uncompromised.

Thanks to all above, it’s not a (technical) problem at all that users tend to forget their passwords just after sign up. Now the process engine remembers were they came from, and what they were supposed to do, just after resetting their password one more time…

With a reusable sub process, password change can part of any epic

Plone 5.2.4 Released!

Posted by PLONE.ORG on March 05, 2021 10:50 AM

General notes:

Plone 5.2.4

Plone 5.2.4 is a release of Plone 5.2.

Download Plone 5.2.4

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.4/versions.cfg.

  • Linux/BSD/Unix users: Use the Unified Installer. It is a configuration and setup kit with build scripts.
  • Windows 10 users: Use the Unified Installer. See Windows-specific installation instructions. Consider using the Unified Installer within the Windows Subsystem for Linux (WSL).
  • OS X users: use the Vagrant kit or install XCode command-line tools and use the Unified Installer.
  • Automated provisioning: See Plone's Ansible Playbook for a full-stack installation kit.
  • Cross-platform Docker: install Docker and use the Plone Docker image.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

Some highlights of this release are:

  • Products.PluggableAuthService: security fix for open redirect and missing access control.
  • Zope: security fix for missing access control in some XML-RPC requests.
  • GenericSetup/CMFQuickInstallerTool: security fixes for possibly seeing information from installation logs and snapshots.
  • plone.recipe.zope2instance: Windows fixes
  • Products.MailHost: Use standard conforming ``\r\n`` line endings.
    If you use Microsoft Exchange to send mails, this should prevent empty mails.
  • mockup / plone.staticresources: various fixes in folder contents.
  • plone.app.caching: Restored ``resourceRegistries`` ETag, but now for Plone 5 resource registries.
  • Fixes warning "Could not find value adapter for ETag component resourceRegistries".
  • plone.app.contenttypes: Various fixes for restoring references during migration.
  • plone.app.users: Fix setting "Use site default" for wysiwyg_editor.
  • plone.restapi 7.0.0 introduces new features, which should be backwards compatible:
    • Add ResolveUID functionality for Volto blocks, allowing Volto to preserve internal links when content is moved.
    • Add root element to the @breadcrumbs endpoint.
    • Mark restapi 7 with a zcml feature flag: plonerestapi-7
    • Add new @contextnavigation endpoint.
    • Refactor navigation endpoint, add new nav_title attribute
    • Add "smart fields" concept: if block has a searchableText field, this will be indexed in Plone

For detailed changelog, go to https://plone.org/download/releases/5.2.4

Great Progress at the Plone Relations Sprint

Posted by PLONE.ORG on February 19, 2021 02:18 PM

At the 2020 Plone Conference, Philip Bauer gave a Day 3 lightning talk titled "Why Relations are Weird" (at minute 12:13 of the video). In the talk he explained how Plone relations code is not straightforward to work with, and he described a package he wrote to help: collective.relationhelpers. Inspired by lots of buzz on the conference Slack channel afterwards - including a number of calls to "put that relation helper into plone.api!" - a group including Philip, Alec Mitchell, David Glick, Jans Klein, and Alec Ghica held a conference Open Space to discuss possible changes to how Plone relations work. Ideas for major improvements (though appealing) were dismissed as simply too much work, but all agreed that the following changes were both desirable and feasible to accomplish in a short sprint.

  • Move calls from collective.relationhelpers to the Plone API.
  • Add most of the other collective.relationhelpers code (including rebuild_relations) into CMFPlone.
  • Fix the data managers for relation fields to make it easier to use other kinds of widgets with them.

A remote sprint to work on these changes took place on January 30th and 31st and was attended by Philip Bauer, Alec Mitchell, Fred van Dijk, Maurits van Rees, Iulian Petchesi, Alexander Loechal and Sally Kleinfeldt. All agreed to put this improvement in core and to put documentation in training.plone.org. We also agreed to only work on Plone API and save the RestAPI work for others who know it better.

  • Philip, Alec and Fred worked on data managers and converters, better widgets for relation fields, and an easy way to define a catalog query in the schema for any type of select widget.
  • Maurits and Iulian worked on additions to plone.api, including relation create, delete, and get methods and tests for them.
  • Alexander worked on the tox setup to make testing Plone API easier, and on control panels for Plone relations - one to inspect and one to rebuild.

By the end of the sprint nothing was finished (i.e. merged and released), but a lot was accomplished and the code produced is production quality. A follow on sprint was scheduled to polish off loose ends (plone.api methods, control panel for Plone Classic and Volto, documentation, tests) and review and merge PRs.

Everyone at the sprint was excited to have made a significant improvement to a gnarly part of Plone core code.

Three Days, Lots of Progress, the "Not an Alpine City Sprint" Sprint

Posted by PLONE.ORG on February 19, 2021 10:45 AM

It was the time of the year for an Alpine City Strategic Sprint, but you know, COVID19. Because of the special spirit of this Sprint, it is not possible to just go online and proceed. On the other hand some people out of our Plone community asked me to organize something: there is work to be done. We decided to sprint online for three days, at a Not-an-Alpine-City-Sprint Sprint. 

This happened from Wednesday 10th to Friday 12th of February.

We had twelve participants covering seven topics. Using Jitsi, Discord and Github with its issues and project boards the remote coordination worked very well. We had two “standup” meetings over Jitsi each day, with reports, planning and overall coordination of the tasks.

It was great to meet online at least, but we are all looking forward to meet in person as soon as all are vaccinated and it’s safe again!

The tasks we worked on with people and outcomes:

  • Plone Classic: Update Barceloneta to Bootstrap 5, including cleanup.
    Work done by Peter Holzer, Peter Mathis, Jens Klein, Stefan Antonelli, Robert Kuzma.
    Progress: Lots of tests fixed, all are green now. Many UI details were fixed. All branches are merged to master now.
  • Mockup: Update to ES6.
    Work done by Maik Derstappen, Johannes Raggam
    Outcome: Prototypical integration of ES6 Mockup/Patternslib in Plone 6 done, see PLIP-3211. Removed RequireJS from the resource registry in CMFPlone. https://github.com/plone/Products.CMFPlone/pull/3247. Integrated new Mockup in plone.staticresources and clean up (removed resources.xml registry entries, stripped down bundles.xml, removed components subdirectory. Upgrade steps are not done yet!). https://github.com/plone/plone.staticresources/pull/120. TinyMCE update to version 5 and pattern migration to ES6, 90% done. Working on ES6 Pattern integration to make more of them work in Plone 6. Webpack build optimizations. Toolbar pattern is functional. Structure pattern is almost functional. Tooltip pattern from Mockup replaced with the one from Patternslib. Allow to customize pattern options from Patternslib in Mockup. E.g. the tooltip pattern default value for the trigger option was changed from click to hover. Created https://github.com/Patternslib/pat-code-editor as replacement for pat-texteditor. Created https://github.com/Patternslib/pat-tinymce as replacement for Mockup’s pat-tinymce. Experiments with replacing Mockup’s pat-inline-validation with pat-validation from Patternslib.
  • Plone Installer: Fix the old one and plan a new installation story for Plone 6.
    Work on the old installer done by Jens Klein, Allessandro Pisa, Fred van Dijk.
    Outcome: Old installer works again on Linux and MacOS; Windows works but tests are failing. We had a Zoom discussion with a broader circle of persons about a new installer for Plone 6. We discussed the production and development requirements for the back end (Python) and front end (JS). There will be a separate announcement about the outcome soon.
  • Using Dexterity for Plone Site Root
    Work done by Alessandro Pisa.
    Outcome: Code cleanup, some hacks were removed, and for this reason we have less packages checked out. Restored the ZMI for the Plone site root, added an upgrade step to make a Plone site DX ready. For real world testing some databases with simple add-ons were successfully migrated. During the sprint Plone was upgraded to use the latest Zope 5.1.
  • Relations: Add a control panel and helpers for management.
    Work done by Philip Bauer.
    Outcome: Continued work on relation control panels in CMFPlone and wrote some tests for it. https://github.com/plone/Products.CMFPlone/pull/3232. Added support for broken relations in https://github.com/collective/collective.relationhelpers and a new control panel; released collective.relationhelpers 1.3.
    Work will continue at the relations sprint sequel on February 27th.
  • plone.autoinclude: A full pip compatible modern rewrite of z3c.autoinclude.
    Work done by Maurits van Rees, Thomas Schorr.
    Outcome: Added lots of tests and a setup with tox; this now runs with GitHub Actions. Test coverage over 90%. Worked on integration tests: the repo contains several test packages with zcml and entry points, and two integration packages that mimic how Products.CMFPlone would use this. Worked on unit tests and a wrapper to install the test packages in the working set when needed. See PR 1. Added a “plone.autoinclude.plugin” entry point for when we cannot handle a  “z3c.autoinclude.plugin” entry point. This is the case when the project name (setup.py) differs from the Python module name. We load both entry points. Apart from the new entry point, the original code was hardly touched: it is holding up quite well under all the new tests. 
  • eea.facetednavigation: A feature to add custom CSS per widget.
    Work done by Robert Kuzma
    Outcome: Add custom_css field, branch https://github.com/collective/eea.facetednavigation/tree/custom-css-widget. Update tests. Check in with Alin Voinea about the PR and release https://github.com/collective/eea.facetednavigation/pull/58

3rd Annual Python Web Conference

Posted by PLONE.ORG on February 11, 2021 01:05 PM

The 3rd annual Python Web Conference is back. This interactive, thought-provoking virtual experience will take place from Monday, March 22nd through Friday, March 26th, 2021 (8am-12pm ET). The event will feature international tech experts presenting on 48+ topics such as Django, machine learning, Big Data, AI, deployment best practices, serverless architecture, GraphQL, Docker, Plone, and much more. Talks will be split into 4 engaging tracks: “App Dev”, “PyData”, “Cloud” and “Culture”. Each track will be accessible via LoudSwarm, Six Feet Up’s virtual event platform. The full list of speakers can be found on the Python Web Conference site, where you can also grab your ticket for the event.

WHEN:
Tutorials: March 22, 2021, from 8am ET to 2pm ET (UTC -4h)
Talks: March 23-26, 2021, from 8am ET to 2pm ET (UTC - 4h)

WHERE: Online via LoudSwarm


REGISTER TODAY: Professionals: $199 | Students: $99

Questions for the Steering Circle?

Posted by PLONE.ORG on January 29, 2021 08:47 PM

As described in the Foundation's July discussion of Plone governance, a series of Steering Circle meetings is being held to discuss issues with our organizational structure and processes. This is part of the Foundation's initiative to solicit ideas for changes that will better serve the needs of our community, our projects, and our teams. The meetings will be held every two months, and the next one will be February 16th at 3:00 PM UTC. Each Plone team will send one or two representatives, including the Zope, Volto, RestAPI and Guillotina teams.

The Steering Circle meetings will include a discussion of questions from community members. Please use this form to submit any questions you have and we will put them on the agenda.

Thank you for being an awesome community and for helping to move Plone forward into its third decade!

Open Source Week 2020 Recap

Posted by PLONE.ORG on January 05, 2021 08:50 PM

Open Source Week 2020 was held from December 1 to December 4. During the event, several topics about the Open Source world were discussed. Four storytelling days about Open Source through the most reliable voices in this world, the customers' feedback, technical workshops and focus on products and solutions.

https://opensourceweek.it.

There was talk about enterprise business, the complexity of development models and business, the ethical value, and the legal and security challenges from the perspective of the top insiders. The first entirely online RIOS conference dedicated to the Open Source world dealt with Cybersecurity, Collaboration, Machine Learning, and Artificial Intelligence, Cloud Native, DevOps+Sec, Development, IoT, Augmented Reality, Industry 4.0, BigData, Analytics, all in an Open Source key.

As part of World Plone Day on December 3rd, we talked about Volto and the new developments of Plone.

Plone 6 example

The event in its 4 days has brought the participation of about 500 people and the sale of more than a thousand tickets. Through sponsored posts on social media, the participation of the event was maximized, in addition to targeted posts during the conference on the main social channels. Through the social campaigns, Plone's posts were seen by more than 18 thousand people.

The Plone logo has been mentioned as an Official Sponsor on the event website https://opensourceweek.it.

Philip Bauer: Growing Pains: PosKeyErrors and Other Malaises

Posted by Maurits van Rees on January 05, 2021 10:33 AM

This talk is about the issues that you face when your project grows, the code base grows, the database grows, the problems grow. This is about the causes and some of the remedies.

Symptom 1: huge database

Cause 1: a huge number of revisions or versions.

Remedies:

  • Remove all versions and pack the database. When you migrate to a new Plone version, and you ask your client, they will usually be okay with this.
  • Manage or limit revisions. Easiest is to use collective.revisionmanager for this. Especially, revisions may have been left behind for content that no longer exists. You can easily remove it with this tool.
  • Disable versioning of Files. It is disabled by default, but maybe someone has switched it on.
  • Enable manual versioning instead of automatic. Then the editor needs to check a box when they make a major change that they want to be able to rollback.

Cause 2: no packing.

Remedy: just pack it. Use the zeopack script, which part of plone.recipe.zeoserver. Add a cronjob for this, weekly seems best for most sites.

Cause 3: unused content.

Remedy: delete it. You have to find it first. Of course no code can tell you which content is safe to delete. You could use statistics.py from collective.migrationhelpers to get an idea of where which content is.

Cause 4: the SearchableText index is huge

Remedies:

  • Use solr or elasticsearch and possibly remove the SearchableText index.
  • Don't index files. They are converted to text, but this may not be needed for your site.

Cause 5: large blobs For example, plone.de had a Linux iso image, which was huge.

Remedies:

  • Limit the upload size. You could do this in nginx/apache. Archetypes had something, you can likely do this in Dexterity too.
  • Get stats and remove or replace too large items.

Cause 6: aborted uploads (rare)

Remedy: check IAnnotations(portal).get('file_upload_map').

Symptom 2: slow site

Cause 1: unneeded full renders of content

Remedy: use Python in page templates. By default, page templates use path expressions like this: tal:define="foo context/foo". But this tries to render  foo as html if possible. Use foo python:context.foo instead.

Cause 2: wake up many objects

Remedies:

  • Always try to use brains and metadata. The difference is huge, also with Dexterity.
  • Listing 3000 brains: 0.2 seconds
  • Listing 3000 objects: 2 seconds
  • Same is true for Volto when you use the search-endpoint with fullobjects.

Of course most page templates in Plone will not list thousands of objects, but will be paginated. Still: just use brains, they are so much tastier.

Cause 3: no caching

Remedies:

  • Switch on the built-in caching
  • Add varnish
  • Manage the zeocache (that is a bit of science, ask the community)
  • Use memoize in your code.

Cause 4: hardware

Remedies:

  • Don't be cheap.
  • Buy enough ram to keep the database in memory.
  • Remember that your consulting time probably costs more than buying better hardware would.

Cause 5: slow code

Remedies:

  • Learn and use profiling. A very handy toy for that is py-spy. Sample use: sudo py-spy top --pid 12345
  • Do not call methods multiple times from templates. Call them once, store the result, and use this.

Cause 6: slow data sources

Remedies:

  • decouple, for example using redis or celery
  • Use your choice of async implementations
  • Use lazyloading of images if they come from outside of your Plone site.

Symptom 3: conflict errors

Conflict errors happen when two requests work at the same time and both change the same object. This is complicated, but Zope and the ZODB have built-in conflict resolution.

Cause 1: conflict resolving is not enabled. The zeoserver needs access to the same code that your zeoclient has, otherwise conflicts cannot be resolved and the transaction will be aborted.

Remedy: add all application code to the zeoserver:

[zeoserver]
eggs = ${buildout:eggs}

Cause 2: long running requests change data

Remedies:

  • Prevent writes.
  • If it takes long, do intermediate commits when possible.
  • Prevent crossfire: disable cronjobs and editors when a long request needs to run.
  • Use async. Talk to Asko about that probably.

Symptom 4: PosKeyErrors

Cause 1: missing blobs

Remedies:

  • Copy all blobs of course.
  • Use experimental.gracefulblobmissing in development to create dummy blobs where needed.
  • Find and delete afflicted content in a browser view.
  • There can be cases when you have two zeoclients and the syncing does not work well. Talk to Alessandro about that.

Symptom 5: broken data

Now for the really interesting part. These are errors like:

ModuleNotFoundError
AttributeError
ImportError
PostKeyError
BrokenObject

I could read you my whole blog post about zodb debugging.

Cause 1: code to unpickle som data is missing

Remedies:

  • Ignore the errors, if normal operation still works, and the site only has to stay up for a limited time, because zeopack probably also fails.
  • Fix it with a rename_dict. See zest.zodbupdate for some examples that are actually really useful. [Thanks! MvR]
  • Work around it with an alias_module patch, like plone.app.upgrade does in several cases. Then imports can work again.
  • Find out what and where broken objects are and then fix or remove them safely. Use zodbverify.

Steps for the last one:

  • Call bin/zodbverify -f var/filestorage/Data.fs to get all broken objects.
  • Pick one error type at a time, with an oid (object id) that has a problem.
  • Call bin/zodbverify -f var/filestorage/Data.fs -o <oid> -D to inspect one object and find out where it is referenced.
  • For the extra options, you should use the branch from my pull request, which I still have not finished yet, but it runs fine.
  • Remove or fix the object.
  • Important: make notes, write upgrade steps, keep the terminal log, because you will forget it and need it again.

To remove or fix the object, it helps to start the actual Plone site with some special zodbverify sauce:

./bin/instance zodbverify -f var/filestorage/Data.fs -o <oid> -D

Then you can use your debugging skills to try and fix things. Note that after you fixed it, you need to commit the changes explicitly:

import transaction
transaction.commit()

Note that the bad object is still in the database, until you pack it.

Frequent culprits are IntIds and Relations, especially if you migrated from Archetypes to Dexterity. Using collective.relationhelpers you can clean this up:

from collective.relationhelpers.api import cleanup_intids
from collective.relationhelpers.api import purge_relations
from collective.relationhelpers.api import restore_relations
from collective.relationhelpers.api import store_relations
# store all relations in a annotation on the portal
store_relations()
# empty the relation-catalog
purge_relations()
# remove all relationvalues and refs to broken objects from intid
cleanup_intids()
# recreate all relations from a annotation on the portal
restore_relations()

Symptom 6: bad code

Unreadable, untested, unused, undocumented, unmaintained, complicated, overly complex, too much code. If you can convince a client to not want a feature because they will only use it once, that is a win. Every line of code that is not written, is a good line of code.

Plone Conference 2020 Recap

Posted by PLONE.ORG on December 20, 2020 04:54 PM

Plone Conference 2020 Online

Plone Conference 2020 was a 9-day event for the Plone, Zope, Volto, Guillotina & Pyramid communities and consisted of training, talks, and sprints. The conference provided insight into the long history of Plone CMS as well as the latest, future-proof features and visions.

  • Plone - The original, open-source, enterprise-grade, all-in-one content management system written in Python.
  • Zope - The original Python web application server - the foundation for Plone and inspiration for Guillotina.
  • Volto - Plone's snappy, modern React front end powered by the RestAPI.
  • Guillotina - A re-imagined asynchronous back end compatible with Plone's RestAPI.
  • Pyramid - Pyramid is a small, fast, down-to-earth Python web framework.

With 295 attendees from 30+ countries from over the world, from Jamaica to Japan, USA to the UK, Germany to Finland, and Austria to Australia the conference was a definite success. We had newcomers to the community, as well as long time contributors since the early days of Plone. 

With the help of our sponsors, like iMio and Syslab, and many others, the conference was held 100% Online, using LoudSwarm platform by Six Feet Up. LoudSwarm nicely combined video streaming, recordings, schedules and chat discussion.

Every training, talk, and presentation was recorded and will be available online on YouTube later.

Group Photo 2020

Group Photo of Plone Confercence 2020.

Training

The conference contained, like in previous years, over 30 hours of free training, with topics including Mastering Plone, Volto Addons, Pyramid, and Guillotina.

https://2020.ploneconf.org/schedule

The training videos are already publicly available on YouTube.

Talks and Open Spaces

4 days of talks included almost 60 professional presentations, ranging from different use cases to building and managing projects into very technical talks about some aspect of a certain technology. Many of the talks focused on Volto, Plone's new React based frontend, and there was also lots of discussion about the future of Plone 6.

With Open spaces and 5 min lightning talks, it was possible to bring into focus many other aspects and topics.

Some highlights of talks: 

And many more! https://2020.ploneconf.org/schedule

Keynote Day 4

Day 4 Keynote.

Sprints

Every Plone Conference has contained 2 days of sprinting so after the conference talks people still gathered together and improved and developed Plone ecosystem forward.

Sprint topics included e.g.

  • Newbies Introduction to Plone Development
  • Plone Foundation Sponsorships
  • Plone Classic UI
  • ES6/Mockup/Patternslib
  • Volto
  • Unified Field for Images and Files (PLIP 2968)
  • Pylons Project projects (Pyramid, Deform, and others)
  • plone.importexport
  • Plone.org improvement

We will publish a separate sprint report on Plone.org to cover further details.

Community

Plone community aims to be the most welcoming and friendly community towards new and old people alike. We are happy to report that this conference had many new faces and we sincerely hope, that everyone felt the love.

On the communal aspects, check out the presentation Oh, the Places we've been! Plone 2001 - 2020

Every conference has a party, and this was no different, except beside the fact that everyone was online this time.

Thank you, everyone, sponsors, organizers, trainers, presenters, and especially participants for making this year's conference one to remember!

Community activity

Discussion during the conference.

Next year: Plone 20-year birthday, Plone Conference 2021 in Belgium!

This year's conference was in many ways very special, and next year even more so - Plone is celebrating its 20-year birthday, and if the situation allows, the conference will be held in Namur, Belgium!

So mark your calendars and stay tuned for Plone Conference 2021 news! Follow @ploneconf and #ploneconf2021 on Twitter and Instagram.

In Memory of Max Jakob

Posted by PLONE.ORG on December 12, 2020 02:01 PM

Every community has its silent heroes, people who contribute a lot to the community and put their heart and soul into it. In many open source projects, only the amount of code a person produces is what counts, but in the Plone community it is the person and each of their contributions that matters, be it enthusiasm, community work or the ability to motivate. Yesterday, the Plone community lost one of its silent heroes: Max Jakob.

Max was an unassuming yet integral part of the Plone community. After his first career, as a control systems electrical engineer in steel mills in remote parts of the world, he became an administrator at the Institute of Computer Science at Ludwig Maximilians University in Munich in the 1990s, turning to web technologies. Around 2000, he heard about Zope for the first time and co-founded the local Python and Zope User Group, which he hosted beginning in 2003. In the years that followed, he worked extensively with Zope and then Plone, using them for a wide variety of applications. He was the driving force behind the Munich and emerging German Plone and Python community.

Even though Max never directly made large code contributions to Plone or Zope, without him, Plone would not be what it is today. He created and fostered a vital and very active Plone community in Munich, extending to all the German-speaking countries. Many members of the Plone community were initially attracted to and eventually became integral to the community thanks to Max and the example he set. Two such members are former Plone Foundation board members Philip Bauer and Alexander Loechel, who were instrumental in bringing Plone to Python 3. Max was the rock without whom the Deutsche Zope User Group (DZUG) conferences in Munich, the PloneKonf 2012, and the Plone Tagung 2019 would not have been possible. He hosted many sprints and all of the World Plone Day events in Munich. Through his commitment and support, Max made possible the Munich Zope/Plone, Python and PyLadies groups. He was also a founding member and longtime board member of the German Python Software Association.

His application for Plone Foundation membership more than reflected his understatement. In it, he wrote that he would not have wanted to step into the limelight except that his friends had insisted he apply simply so there would be an official list of his contributions to Plone.

Max was a role model for the Plone spirit. He built a community for Plone and a better world led with trust, empathy and hope.

Max attended every Plone conference since 2007, in Naples. Plone community members from around the world have had the pleasure of meeting him at conferences, the Plone Open Gardens in Sorrento, or one of our many sprints. All of us who had the privilege of getting to know Max will carry him in our memory: a random meeting at a gas station in Brasilia, where Max, Alan Runyan (co-founder of Plone), and Alexander Loechel stayed chatting for hours; walking through the cobbled streets of Ferrara trying to talk Max into hosting a conference...and being surprised by a quiet smile and a “Yes”.

Max Jakob passed away on December 11, 2020. He fought an unwinnable fight against cancer and lost it during one of his favorite events, the Plone Conference 2020. The Plone community has lost a notable and beloved Plone ambassador and Foundation member. He will be cherished and honored by the community. May his spirit live on.

We raise a glass of his beloved IPA to wish him a good farewell. Rest In Peace, Max!

The Plone Foundation Board of Directors thanks him for his many services on behalf of the community and expresses its deepest condolences to his widow and family. 

max-jakob-collage-01.jpgmax-jakob-collage-02.jpgmax-jakob-collage-03.jpgmax-jakob-collage-04.jpg

Maik Derstappen: Plone 6 Theming with Diazo

Posted by Maurits van Rees on December 10, 2020 10:16 PM

How does Plone theming look in classic UI?

  • html5 theme plus a mapping configuration
  • deploying themes as ZIP-file for shared hosting is possible
  • With Diazo you can map any Plone html to a static theme layout.

Separating frontend and backend theme. Don't reinvent the backend views! You could theme the backend, so for content editors and admins, but it looks fine, not needed. You should focus on the frontend layout for visitors. To use the default backend layout you can include backend.xml in your rules, with some conditions.

Diazo is not for everything! If the backend markup differs from what you need, do not try to solve it with Diazo or XSLT. Instead, fix the backend templates directly, most likely with a z3c.jbot override.

You can use theme fragments or browser views to add new templates. The theme fragments can also be used as tiles in Mosaic.

I like using SASS mixins. Say you have a <div class="main-wrapper container">. Cleaner would be <div class="main-wrapper">. You can do this with mixins with @include in your selector.

For more information on Diazo, see https://diazo.org.

New features in Plone 6

From the backend we get Bootstrap 5 compatible html. Result is that Bootstrap themes are easier to integrate in Plone 6.

You have custom CSS in the theming control panel, for small changes. This actually sneaked into Plone 5.2.2 as well.

We have simplified Diazo rules.

Create a theme with plonecli:

  • pip install plonecli
  • plonecli create addon plonetheme.yourtheme
  • cd plonetheme.yourtheme
  • plonecli add theme
  • plonecli build

Theme from this presentation will be published as collective.bunapisicuto when it is ready for you to inspect.

Stefan Antonelli: Plone 6 Theming from Scratch

Posted by Maurits van Rees on December 10, 2020 05:08 PM

How to create a theme for Plone 6. Quite easy, because the templates use Bootstrap 5 classes. We build a theme from scratch, no Barceloneta, no Diazo.

First step is to create an empty plone_addon package with plonecli or mr.bob. For the questions you can answer: use Plone 5.2.1. We will switch later. My theme is plonetheme.munich.

I recommend to cleanup the standard package a bit. I remove tests, constraints for Plone 4 and 5. Check it out in the commits.

Now switch to extend Plone 6 and run the buildout.

You can add theme structure with a bob template, but I prefer creating my own.

Some interesting files:

  • package.json lists various tasks, especially the watch task.
  • In the theme manifest.cfg we more or less disable Diazo by emptying the rules line.
  • The compiled CSS and JavaScript are registered in registry.xml.

You can compile SASS to CSS using npm or yarn. Do yarn install in the top of your package. Later, with yarn dist you make it ready for production.

After these steps Plone is partially broken, or at least ugly. I do some basic fixes and it looks better.

For templates that you need to change you add z3c.jbot overrides. Personally I always kick out the "search in section" checkbox.

I don't like columns, but for this example I kept them. In most cases I need just one column. Plus maybe a side bar for portlets, but portlets must die.

With plonetheme.tokyo everything is Bootstrap, no columns, so no portlets, really fully responsive. This was the package where we built on Barceloneta Plone 5.2 and introduced lots of template overrides to put in Bootstrap. For Plone 6 we can just remove the overrides.

What about the toolbar? Yes, we dropped it. We bring editing features and navigation together. This is now a few feature: collective.sidebar. It is only one template to override. It works for Plone 5.2 at the moment, and I may work on it for Plone 6 during the sprints.

Question: is TTW still a viable path?

Answer: I like to concentrate on one path. I am not an expert in TTW theming. I switched to file system, except really small customizations. For small CSS customizations there is a field in the theming control panel.

Stefan Antonelli and Peter Holzer: Modernize Plone’s Classic UI

Posted by Maurits van Rees on December 10, 2020 04:02 PM

What was new in Plone 5? We had beautiful new theme: Barceloneta. Diazo theming by default. We switched to CSS compilation with less.

During the Tokyo conference Stefan thought up Tokyo Theme. Clean responsive theme for Plone 5. Tons of overrides to tackle problems in Plone 5. Issue with navigation and editing on mobile we solved with collective.sidebar.

We had community discussions, especially during several Plone events. Everyone tried to use Bootstrap (components). First idea: map variables from Barceloneta to Bootstrap, because they have similar ideas using different terms.

We have PLIP to modernize markup in templates, and another PLIP to modernize the default theme: Barceloneta LTS. Forms using z3c.form are already using the new classes.

Make things easier: UI, development. Creating a modern UI for the web is complex. You need to support different devices, responsiveness. In Bootstrap there are patterns for most useful things.

Developer perspective: expect one way to do things. Developers should not have to worry about design. When busy in the backend, you should focus on Python, not on it looking nice and shiny. Don't think about markup, just use components. The good news: there is documentation. The Bootstrap documentation is our documentation.

What is new in Plone 6?

  • Volto is the default UI.
  • There still is the Classic UI with Barceloneta look and feel, but updated.
  • No TTW (through the web) theming.
  • But there is a textarea to add some simple CSS (already in 5.2.3).
  • Some CSS variables may be changeable TTW.
  • Finally jQuery 3

Bootstrap is still the most popular front-end framework. Well documented, tested and maintained. It is so easy to create stuff, I enjoy it a lot.

What is new in Bootstrap 5?

  • Improved overall look and feel.
  • Updated and extended the color system
  • Custom properties: css variables
  • SVG Icon library
  • Pure javascript
  • Dropped IE10 and IE11 support
  • Bootstrap 5 is currently alpha 3.
  • See https://v5.getbootstrap.com

Features: what do we get from these changes?

  • Core templates use Bootstrap 5 markup. Instead of overrides in plonetheme.tokyo, we have lots of branches for the actual packages.
  • All major templates have been touched already.
  • For the current state, see the unofficial demo at https://classic.plone.de
  • The Bootstrap documentation has lots of snippets than you can copy.
  • You don't need much more CSS on top of it if you paste most examples. We added a little for own components, like navigation.

Barceloneta appearance is fully customizable. It is basically on opinionated set of bootstrap variables. Every aspect can be changed with variables: colors, fonts, sizes, spacings, grid gutters, etc. There are overall properties, like shadowed, rounded, gradients. Just turn on or off.

Theming workflow. plonetheme.barceloneta will also be published as npm package. bobtemplates.plone will have a template for the new theming workflow. You can do quick and dirty customizations through the CSS overrides field in the theme controlpanel.

Diazo will still be there, will work as before. Some optimizations in the rules.xml to make content are customizations easier.

How to deal with icons? What if you want to change the content type icons? Used to be hard. Now we come up with the idea of an icon resolver. We decided to use the bootstrap icons. Icons are registered via GenericSetup, for example with a record name plone.icon.alarm pointing to an SVG. You can then override this in your own GS profile.

Example icon use:

<tal:icon replace="structure python:icons.tag('love', tag_class='custom-class', tag_alt='foobar')" />

You get get back an inline SVG or an image tag.

Note: all z3c form widgets in Plone are now in plone.app.z3cform, and not scattered over lots of packages.

We will restart our weekly Plone 6 Classic UI sprints, starting Januari 13 2021, 10:00 (UTC+1).

Keynote: The User Experience - Editing Composite Pages in Plone 6 and Beyond

Posted by Maurits van Rees on December 10, 2020 02:55 PM

It may be a surprise to non-technical people to learn that pages created in Volto are not currently interoperable with traditional Plone's page editing. If you think about it, the reason becomes obvious. Volto, like Mosaic, creates tiled layouts, and like Mosaic it stores page data in special fields for the individual blocks and their layout. Neither Volto nor Mosaic pages are editable in TinyMCE, which expects just one rich text field.

Is this divergence between sites created in Volto and sites created in traditional Plone a problem? It does make it harder to describe what Plone is, and it might mean that there is no way to mix both approaches - for instance when part of a larger site is available as a Volto-based sub-site. Would it be possible to have one tool and one representation for tiled layouts so that we can avoid this divergence? Is there some other solution? Is it even a problem? Will Plone 6 be backwards compatible with Plone 5 and include a smooth upgrade path?

We will tackle these questions in this strategic panel discussion, moderated by Sally Kleinfeldt. Panelists will include Paul Roeland, Philip Bauer, Timo Stollenwerk, Victor Fernandez de Alba, and Eric Steele.

First, Philip has a message from Max Jacob who is very ill with pancreatic cancer and may not survive this weekend. He wants to thank the Plone community for what they allowed him to do: like organize the German Plone Konferenz. Thanks for all the friendships. Such a pity that this is happening now, he wanted to jump into only doing Plone for the next few years, due to changes at his job, and looked forward to that.

Classic, Mosaic and Volto Pages

We have Classic, Mosaic and Volto Pages. They have different internal representations and are not compatible. Is this a problem? Is there a solution? Is one tool, one representation possible? If we really need three, how to position?

Paul: for me as user this presents a problem. When do you switch over your site? We would like to not write 700 pages from scratch, again, like we did for previous composite pages.

Timo: We migrated quite a few large projects from classic Plone to Volto. One of those had collective.cover (other composite page system). Problem in general with such systems, is that they are pretty specific. They solve specific use cases and come from different eras. After any migration, it will not look the same any more. Whatever you do: the page will initially look ugly. So you put a lot of effort into migration, but then have to put manual effort into every page anyway. It can help: you at least have a start. We created a system where we migrated overview pages, and editors could click to migrate other pages one at a time.

Philip: We have code to migrate from non-folderish to folderish content types. There will be code to migrate to dexterity site root on Plone 6. We can make sure to migrate any standard content types. Mosaic is another story. So for pages you would at least have the text available. Maybe only visible for editors to pick and choose from. You may lose portlets, unless they get implemented in Volto.

Timo: When you go to Plone 6 and do a redesign at the same time, then you can jump on Volto. Otherwise you could stay at Classic Plone for now. There will be an overlap period.

Victor: For Mosaic you could dump all tiles into html and insert it in a block.

How to have a big Classic site with a subsite made in Volto?

Victor: Definitely doable, though we have not done this ourselves.

Sally: But what happens when an editor in Classic goes to the Volto subsite? The Volto page would not be editable then, right?

Philip: You should not offer this. I see no upside, no use case. Split them into separate applications, with shared authentication maybe.

Paul: Use case: large site with several departments. The marketing department may want snazzy new Volto things.

Timo: Just create another site then.

Cost and benefit of upgrading big Classic Site to Plone 6

So you just had a big migration to Plone5, and now what would you get for going to Plone 6.

Philip: We have this discussion every major upgrade. Communicate every upgrade as a relaunch. The relaunch is the reason for the upgrade. "There is a new version so you need to upgrade" does not fly for my clients.

Timo: We became a very developer oriented community, and every develop understands the need and benefits. We should really get back to giving more value at major releases, so clients really want to upgrade themselves. Plone releases should sell themselves.

Eric: It looks like we think Plone 6 + Volto is a costly upgrade with lots of benefits. For Classic 5 to Classic 6 the upgrade is not costly.

Alex Limi's vision for Deco: are we there yet?

Was this fulfilled by Mosaic? Volto? Something still needed?

Paul: Not quite, but slowly getting there. For me it would be Volto, plus some power features that Eau de Web (EEA) adds.

Timo: I think we went beyond what Limi envisioned.

Eric and Victor: What we have seen from Volto, is pretty close to what Alex wanted.

Victor: We gave the users powerful tools, so beware of them.

Philip: Partly yes. Volto is close, and it is for normal users.

Now some questions from users.

Migrations

Philip: Plone 4 to 5.2 was three migrations in one. Plone 6 is less of a problem.

Eric: 5.2 had a lot of backend migrations. A split between backend and frontend with plone.restapi in between makes things easier.

[The question on multiple variations of Volto, especially editors, went a bit too fast for me to write intelligible notes down.]

What's the future of using CT's/behaviors in Plone to design information architecture?

With Volto the trend seems mixing/adding 40 different blocks for every page.

Timo: Blocks are definitely the way to work. But the underlying power of content types and behaviors still exists.

Philip: We need blocks that represent a field or a behavior. That is unavoidable.

Next steps

Timo: We plan to have an open space on page compositions and Volto, and want to sprint on it.

Paul: Good if there is a longer term vision. I would rather have more power that a Site Admin can lock down, than having to choose between three different versions. I don't want choice stress.

Lightning Talks Wednesday

Posted by Maurits van Rees on December 09, 2020 06:21 PM

Lukas Guziel: Continuous Deployment

CD means deploying code automatically. It saves time, reduces human error. It Gitlab you can add gitlib-ci.yml and configure it. Include a base template that you use in multiple projects. End result can be a site that the customer can test.

Erico Andrei: World Plone Day 2021

We are a global community. Almost 300 people from 36 countries are at this online conference. World Plone Day is an annual Plone event. Next year of course online. April 28th 2021.

We want to stream 24 hours live on our YouTube channel. Showcase Plone. Technical talk, use cases, interviews, demo.

It should not all be in English, please use your own language. Talk to your local community.

Please help and join. See https://plone.org/events/wpd

Andreas Jung: collective.contentsync2

Syncing content between Plone sites through plone.restapi. It is a behavior. You have a source Plone site and one or more target Plone sites. You need Plone 5.2 under Python 3.

  • Create a dedicated user account with global role Editor.
  • Configure on the content sync control panel.
  • Automatically creates two content rules to sync content when added or modified.
  • You can enable it on all content types, also Folders.

See https://github.com/collective/collective.contentsync2

Philip Bauer: Why relations are weird

These packages have a part in relations:

  • zc.relation: abstract relation catalog
  • z3c.relationfield: fields and values on objects
  • plone.app.relationfield: converters from field to widget and vv
  • plone.app.z3cform: widgets
  • mockup: actual widget UI

In a schema use a RelationChoice field with vocabulary plone.app.vocabularies.Catalog, and set pattern directives.

It is not straightforward. So I wrote collective.relationhelpers.

See https://github.com/collective/collective.relationhelpers

Maybe use uuid instead of all this code.

[About ten people in the chat want to merge this package into plone.api. Actually, see this issue.]

Christopher Lozinski: Simple JSON Schema GUIs

Create a JSON schema, automatically generate the UI. Search for basic JSON editor library. He shows JSON in ZODB, so you can browse it, if I understood correctly.🙂

Eric Brehault: Second Guessing the Single-Page-App Pattern

Posted by Maurits van Rees on December 09, 2020 05:42 PM

SPA (Single Page App) is about providing an entire app by exposing a single physical web page containing an enormous javascript bundle. It breaks the original web paradigm in many ways. Surprisingly enough, we invest a lot of efforts to mimic the regular web behaviour.

Isn’t it time for modern frontend to reconsider the SPA approach?

[Note: Eric presented by using a projector to show his slides on a black Plone conference T-shirt. :-)]

Why are we doing this? Originally we always requested a whole page and this was considered slow. But we have good bandwidth now. And if you don't have good bandwidth, the super big bundle is not good either.

With SPA we try very hard to bring back the original working of the page, especially the browser history, being able to browse and then share the link to the current page.

To mitigate problems, we created an enormous stack. And we deny the complexity. New tools create new problems, even when their individual creators does not see the complexity.

"SPA isn't stable or efficient." But there is no way back. For example, you cannot create Google Docs with server side rendered pages. Web 2.0 is 15 years old. It is still about content.

SPA is separation of concern, which is a good principle. But we mix the browser layer (how you get and view the page) and the content layer (the page content).

It seems a take it or leave it situation: either use SPA or don't. What do we want? We want proportionate complexity. Do we need 100 percent SPA?

You can use micro components, see for example the demo of Maik Derstappen in the lightning talks on Monday, using Svelte. Micro frontend is bigger than that. It is a part of the application, that you develop separately. For example, you could do the Plone Sharing page like this.

Can we compile each page separately? Then each page is an app.

ES6 native support would be interesting. Combine with HTTP/2 and you need no bundles. Bundling is the most brutal thing ever. Horrible. Get rid of it.

Respect the layers. SPAs are monolithic. Break them down.

We should have a generic browser layer, common to many different use cases, for example for logging in. I don't want to code that, but plug it. Second step: push this layer to the browsers themselves.

Asko Soukka: Deploying Plone and Volto, the Hard Way

Posted by Maurits van Rees on December 09, 2020 04:22 PM

Here are the slides.

How about building Plone without buildout? Running Plone on Python 3 without WSGI? Deploying Plone and Volto with containers without Docker? Building all this in re-usable and safe manner in sandbox with restricted network access with Nix? Welcome to hear about our hipster setup where we lock, build and configure Plone deployments with Nix, insist to keep ZServer running on Python 3 for the love's sake, build software deployments into standalone tarball archives, and run them with Nomad – the simple on-premises-friendly alternative for K8S.

  • The easy, documented way: buildout, WSGI, Docker (if you need containers), Registry.
  • Our way: pip, TxZServer, Nomad, Nix

When you use a container infrastructure, you have multiple containers for running a Plone site, for example zeo clients, zeo servers, load balancer. Nomad helps there, and is much simpler than Kubernetes. We have one job file to rule them all: task groups, instance count, update policy, server resources, volumen mounts, tasks, consul services, vault secrets, environment variables, exec artifacts.

Nomad has "isolated fork / exec driver". No docker image needed. We have a Nix-built artifact, a tarball that we extract in the root of the container.

With Nix, you get 100 percent reproducible artifacts. Production equals development. You have a full dependency graph. The result is a standalone tarball, perhaps 100 MB. Disadvantage is that there are no conventions, no metadata, no shared layers, no documentation. It needs learning and practice. Well, some documentation now: https://nixos.org and https://nix.dev, partially made by people that were using Plone previously.

Some ugly parts from Nix:

  • Every language has their own Nix-conventions
  • dependency generator ecosystem is comples
  • cyclic dependencies are not supported
  • no storage device is big enough for /nix/store

Our (legacy) approach for Plone 5.2.1 without Buildout and with pip:

  • generated requirements.txt with buildout
  • create Python env with pip and nix
  • use pip-branch of z3c.autoinclude
  • disabled <includeDependencies />
  • generate instance skeleton with nix
  • forked plone.recipe.zope2instance

Plone 6 without Buildout should be pip-installable out of the box, but that is hear-say.

We use TxZServer in production, so ZServer using Twisted.

Nicola Zambello: Theming Volto without SemanticUI: Is It Possible?

Posted by Maurits van Rees on December 09, 2020 02:22 PM

We will walk through the process of building a product for Italian Public Administrations using a bootstrap-based theme. I'm presenting io-comune, RedTurtle's first product based on Volto and the strategies we used. We will see the possibilities in Volto for theming without SemanticUI, using bootstrap and sass and what are the next ideas we could work on.

Scenario:

  • We wanted to adopt Volto in our new project.
  • We needed to include Bootstrap.
  • Volto uses SemanticUI instead.
  • Two such frameworks will conflict, for example fighting over the same selector.

We tried. We tried harder. A cheap approach did not seem possible, so we looked for a sane one.

A new theme: pastanaga-cms-ui. Load only the CSS needed for Volto admin UI, see Volto PR 970. And public-ui for public pages. In your src/theme.js do not import the css/less from semantic-ui, but the pastanaga-cms-ui. In theme theme.config also use pastanaga-cms-ui. Also razzle-config.

You should normalize your base style, for example:

body.cms-ui {
  .public-ui {
    font-size: 18px;
  }
}

and wrap your components with .public-ui.

Building a product:

  • Base common package for every customer: https://github.com/RedTurtle/design-volto-theme
  • New intermediate layer for SemanticUI
  • New config layer for razzle/customizations
  • Template for actual projects: design-volto-kit, with a Yeoman generator: create-italia-volto-app

Lightning talks Monday

Posted by Maurits van Rees on December 09, 2020 10:19 AM

Alec Mitchell: WYSIWYG problems be gone

A new add-on to vastly* improve your content editing experience.

(* size of improvement may vary, no warranty implied. The following is (un)paid free software promotional content)

Adding images to a document is so hard! At least nine steps! (Difficulty be exaggerated for marketing purposes). Why can't you drop an image in? You can, with our new, super special add-on krcw.tinymce_imagedrop.

Can you drop two? Yes!

Can you drop more than two? No, because browsers are weird.

But it fails gracefully.

Steve Piercy: Deform and friends

How I learned to stop worrying and love web forms. We must have a good interface, data structure, validation, security. Deform (form library), colander (de/serialization), peppercorn (data structure), bootstrap forms for design. We have a looooong list of widgets. In deform 3.0 we will use bootstrap 5. See https://github.com/pylons/deform

Christopher Lozinski: Forest Wiki

The Forest Wiki is a modern version of Zope 2. Biggest difference: it uses Pyramid's security and views. Modern JavaScript enabled ZMI: reorder, sort, rename, etc. Both WYSIWYG and MarkDown pages. Advanced types like JSON, CoffeeScript, pug. Pug is the leading template engine for Node.

Jens Klein: RelStorage

Plone relational database backend storage. It is a drop-in replacement for FileStorage of ZEO. You can use PostgreSQL, MySQL, Oracle, SQLite. It has been around for about 13 years, grown old, but in recent years development has picked up, driven by Jason Madden, including Python 3 support. It is much more performant. Latest release 3.4.0 is form October 2020.

PostgreSQL is the cloud database, kind of industry standard, well supported by all big cloud providers. Easy to install in Docker.

Advantages of RelStorage: fast, parallel commits, better concurrency, shorter locks. Optimized per process caching. Blobs in database. Optionally you can use it in history free mode. You lose the Undo functionality, but you don't need to pack so often.

plone.recipe.zope2instance supports it with the rel-storage option.

You can use additional client side caches, shared between all threads of a process.

With the zodbconvert tool you convert from ZEO to RelStorage, or the other way around, including converting blobs if needed.

ZODB keeps old transactions, so packing is needed, even in history free mode. RelStorage has a fast zodbpack.

Blobs:

  • RelStorage 3.x is Python 3 only and runs with Plone 5.2+. Here, blobs should be stored in RelStorage.

  • RelStorage 2.x is for Plone 5.0, 5.1, and blobs should **not**  be stored in the database, except for Oracle backends, otherwise you should still use a shared blobs filesystem directory.

    System Message: WARNING/2 (<string>, line 69); backlink

    Inline strong start-string without end-string.

I use RelStorage today for all my live deployments. I have used it since version 1.6 with Plone 4.3 and never had problems. Always blazing fast. Dev/ops and sysadmins love it: it is a standard solution, nothing special, just works.

Maik Derstappen: Add-on catalog for Plone

We want to bring back an add-on catalog for Plone. You can look on PyPI, but it is hard to find packages.

We worked on a tool for this. You can search on named, filter on Plone versions and add-on types.

Components:

We only aggregate packages that have classifier Plone :: Framework. We will probably work on this during the sprints and are happy to onboard you.

David Bain: Plone and Webflow

Both platforms are for building websites, but they approach things in different ways. I hope this may inspire. Keep in mind the motivation of the two platforms, which may account for some strengths and weaknesses.

Webflow is visual web design, less content management. Strong design tools. Designer friendly layout tools. You can design a page with what you could call blocks.

Plone is enterprise content management, focus on security. Linking to an attachment is standard, where it is tricky in Webflow. Forms are way more flexible.

We have also built a website in Webflow and based it on Plone.

Miu Razvan: Volto grid block

  • Created by Eau de Web team
  • Dependencies: Volto blocks form
  • Similar component: Volto columns block
  • Use it to organize other blocks.
  • Demo showing lots of configuration options, including for different screen sizes
  • See https://github.com/eea/volto-grid-block

Maik Derstappen: Custom elements

Custom elements are an extension to normal native html elements, for example <flag-icon>.

The promise of web components: write once, use anywhere. See https://custom-elements-everywhere.com/

How do you use this in Plone? Use plonecli add svelte_app to create a small app. Run yarn. Install in Plone add-ons control panel. Edit a page. Replace html source with <my-svelte-app />. And your component is there and working. The size is less than five  kilobytes.

Tiberiu Ichim: volto-slate

volto-slate is a drop-in replacement for the standard rich text editor in Volto. Volto turns an HTML document into a modern document.

Why another text editor instead of improving the existing one?

  • With Slate we get a better plugin framework. Plugins are just wrappers around the editor. The standard Draft.js is meant to be integrated directly by an application, no concept of plugins out-of-the-box.
  • Slate has simple DOM-like storage for its values, making it easier to render the ersult.

Current status:

  • No migrations of any kind.
  • Right now not possible to completely remove or replace Draftjs out of Volto.

Alin Voinea: Volto Dexterity Schema and Layout Editor

Posted by Maurits van Rees on December 08, 2020 05:40 PM

Through the Web Dexterity Content-Types with Schema Editor and Blocks Layout Editor

How do we define content types schemas in Plone?

  • TTW schema editor
  • GenericSetup profile
  • Behaviors, schemas in Python

Why do we need them, we have Volto blocks, right? You still need metadata, a title, etcetera. Certainly for larger institutions you need a structure, a schema. Volto itself has schema-based components.

Layout editor. Blocks have properties, like a placeholder, a position. You can type text in a block: "Published on date by author". Then select "date" and link this text to the published date metadata, and select "author" and link it to the author. Save this as a layout for a content type. You can export this to a JSONField in a custom behavior, so you can save it in version control for production.

List of add-ons and other packages that make Volto awesome: https://github.com/collective/awesome-volto

Jens Klein: Performance, Profiling, Power-Consumption

Posted by Maurits van Rees on December 08, 2020 04:27 PM

I want to focus on Python performance, so not caching or database performance.

Tools:

  • py-spy: Overall mix of the whole live application, top-like.
  • repoze.profile: WSGI middleware, slows down application. Profile single request and analyse its call stack by count, call time, etc.
  • dis: disassembler for Python at the bytecode level.

Improvements Plone 5.2.0-5.2.3:

  • Avoided early providedBy calls
  • __getattr__ early exit on common attributes
  • zope.interface: some functions are called hundreds of thousands of times when you reindex an index, so a tiny improvement helps a lot. I found various places that could use improvements, and that landed in the package, together with memory improvements by Jason Madden.

Live demo. I call py-spy with sudo because I need to connect to an existing process id.

Future Todo's:

  • plone.restapi has optimization potential, all navigation related, but currently it still supports even Plone 4.3. This will likely wait for a 5.2-only or Python3-only branch.
  • plone.registry is called too often
  • Use python: expressions in all page templates. They are way faster than standard Tales expressions.
  • More introspection.
  • Move more logic from page templates to Python code

Advice: start introspecting the performance of your application.

Alex Clark: The State of Pillow

Posted by Maurits van Rees on December 08, 2020 04:26 PM

The Plone Conference account tweeted that a State of Plone talk would be awesome and that the Plone community missed me. I miss the Plone community too, so I am here.

I will state it clearly: Pillow would not exist if not for Plone.

In July 2010 I announced Pillow as "friendly" fork of PIL. The mailing thread and future answers are interesting to read.

Some history:

  • 1991: Python 0.9.1
  • 1995: PIL started
  • 1998: Zope
  • 1999: Zope2
  • 2000: Python 2.0 with distutils
  • 2001: Plone
  • 2005: Buildout
  • 2006: I attended my first Plone Conference, in Washington
  • 2006: setuptools was born

PIL had an issue, or Plone had an issue with PIL:

  • PIL used distutils.
  • Plone 3.2 used Buildout and setuptools
  • PIL was not installable in Buildout and setuptools
  • Specific problem: import Image could mean the Image module from PIL, or the Image module from Zope.

Various ways of repackaging PIL started, for example PILwoTk. You can still find various PIL derivatives at https://dist.plone.org/thirdparty/

PIL 1.1.6 from 2006 is still the last version on PyPI. I got maintainership of this page this year, actually. Pillow 1.0 is basically the same, except that it uses setuptools. This worked in buildout. I was happy.

Couple years, nothing really interesting happened. But some contributors came along. Pillow 2.0.0 in March 2013 had Python 3 support.

An important milestone in 2015: we added release notes.

Release schedule: in the beginning of every quarter.

We get some money from Tidelift for maintenance.

Fred van Dijk: collective.collectionfilter as a Light-weight Faceted Navigation or a 'compare' Console

Posted by Maurits van Rees on December 07, 2020 09:10 PM

I want to talk about some categorisation and classification options in Plone, next to the folder structure.

Faceted navigation: drill down on 'facets' when you search for items. It was popularized by online shopping. Facets in Plone for developers is: whatever is in the ZCatalog, and for users: what you can search on in Collections. Gold standard is eea.facetednavigation, developed for the European Environment Agency. Examples: EEA, and on two sites by Zest: Vaquums and Minaraad, where it replaces the standard search.

collective.collectionfilter is a much leaner, meaner, but also more limited version of faceted navigation. Demo with standard Plone News Items with some tags (also known as categories, also known as Subject). Add a Collection that filters on News Items. Now add collection filter portlets.

eea.facetednavigation takes over your complete page. In an action you enable or disable it.

Now a demo of collectionfilter in SGBP, a documentation website for water management planning in Belgium/Flanders. The customer wanted to take some graphs and compare them. We did that with collectionfilter and collective.classifiers. With the last one we added structured categories: one for water basins and one for parameters of the graphs. Now we use collectionfilter to query a parameter and show the graphs for all water basins.

You can adapt several things in the collectionfilter UI, for example change how search options are displayed. This is documented, but took me a while to get right.

Collectionfilter also works with Mosaic, because the portlets are also mapped to tiles.

Asko Soukka: Plone and Volto in a Jamstack project

Posted by Maurits van Rees on December 07, 2020 05:54 PM

Here are the slides.

I am a software architect at University of Jyväskylä. I have been using Plone since 2004 and GatsbyJS since 2018. The university wanted one student information management system to rule them all, but... every organisation shall do their own integrations, using granular REST API with deep JSON responses. And there should be branded study guides, which we crafted with GatsbyJS. But this was not enough for the Open University part. They really needed a CMS.

We use Plone 5.2, Volto, GatsbyJS, and have 6000 html pages, times two languages, out of which 760 are Volto pages. With Plone we could extend content types without needing to do any coding, in the content types field editor. In volto we added auto-complete widgets with custom vocabularies. On the GatsbyJs side, we query the connected pages with GraphQL. We render Volto layouts with React components, rendering individual blocks.

Why did we choose GatsbyJs? It is a ReactJS-based site generator. Being static, it is very fast. You can use multiple sources as input, using a plugin architecture. Data lookup is done with GraphQL. It is easy to get started, with comprehensive documentation.

I mentored two Google Summer of Code projects for the gatsby-source-plone plugin. It supports default types and most TTW types, also Volto blocks. You can do incremental updates by modification date, so it is really fast.

Not everything is easy. The full "GatsbyJs experience" requires practice. You want to replace inline images and links with GatsbyJs images and links, replace file links with direct downloads.

Using @plone/volto as dependency to render blocks seemed like a good idea, but it required webpack overrides to be impartable, and could not be used for images and links.

The ugly parts of GatsbyJs:

  • The GraphQL source plugin cannot cache.
  • The build may take hours, and gigabytes of memory.
  • The build result in readonly.
  • For me it is hard to follow GatsbyJs development, especially individual plugins, because they use a monorepo.

Editors can work on the site during the day, and then wee rebuild the result during the night

Plone Conference 2020 Off to a Great Start

Posted by PLONE.ORG on December 06, 2020 04:54 PM

Plone Conference 2020 Online started with two days of training sessions, gathering about 100 participants each day to 9 different sessions!

Training included topics such as:

  • Getting Started with Plone
  • Mastering Plone 6 (2 parts)
  • Volto Addons (2 parts)
  • React and Volto (2 parts)
  • Pyramid
  • Guillotina

Feedback from the training:

"Learned some exciting things about how to develop new addons for Volto/Plone 6. What a great first day at the virtual Plone Conference 2020."

"I'm really impressed about how easy it is to customize a new #Plone site using #Volto."
"I've done for Pyramid training, #ploneconf2020 's first program! It was the first time for my English conference. I could have a conversation because everyone was so kind. Many thanks, @steve_piercy and all attendee! I'm looking forward to Monday's keynote."

The training sessions were recorded and will be available to all Plone Conference participants.

Thank you trainers and thank you participants, see you soon again at the conference talks!

https://2020.ploneconf.org/

How to Participate: Plone Conference 2020 Online

Posted by PLONE.ORG on December 01, 2020 05:41 PM

All this information and more is available to Plone Conference attendees, so please make sure you get your ticket.

What is the conference language?

The Plone Conference is a global event with attendees connecting from a wide variety of countries. English is the official language, but Track 3 will be in French on Day 2.

What is the conference time zone?

The event’s official time zone is Central European, but LoudSwarm will show the schedule in your local time zone.

What technologies do I need to log in to for the conference?

LoudSwarm is the only requirement to be able to watch the talks, whether you are watching live or catching up later.

All conference attendees will get invitation link to LoudSwarm platform, and get access to all information and guides there.

But part of the fun of a conference is being able to talk to your fellow attendees and speakers. For that, we will use Jitsi and Slack.

Jitsi will be used for face-to-face interactions after each talk. Each speaker will join Jitsi after their talk to be available to answer questions or have discussions with the attendees.

Slack is our primary method of text-based communication. If you did not already receive an invitation, email chrissy@sixfeetup.com

How do I ask questions and network with other attendees?

A face-to-face Jitsi room is available for each talk. You can join this room during and after each session. The speaker will join once they are done with their talk to be available to answer questions.

The primary text-based discussion platform is Slack.

Do I have to turn my webcam or camera on?

Be ready to be on camera, it’s easier to connect with the other attendees when you can see each other face to face. Also be sure to check your microphone as you'll have the opportunity to ask questions to speakers, sponsors, organizers, and attendees.

Why is there extra time before the sessions start?

Come and join us an hour before the first talk each morning to get comfortable with the event and where everything is. It’s also a great opportunity to chat with others and network with other members of the community.

How do I ask questions?

In the LoudSwarm session, you will see a Sli.do widget on the right to ask questions to the speaker during the talk, and you can also ask questions in the Slack channel to continue the conversation with fellow attendees.

Will the talks be recorded?

Yes, all of the talks (including trainings) will be recorded and will be available to watch in LoudSwarm once they have been processed (usually about 10 minutes after the talk has finished). Recordings will be made available to the public one month after the conference.

I need help

Join the #conf2020_help_desk channel in Slack to ask the moderators your questions.

More information

All the needed information is delivered to attendees and can be found in LoudSwarm.

Get your tickets and get ready for Plone Conference 2020!

Foundation Board of Directors Nominations Open

Posted by PLONE.ORG on November 20, 2020 10:56 AM

The Plone Foundation is a not-for-profit, public-benefit corporation with the mission to promote and protect Plone. It protects the trademark, copyrights and other intellectual property, hires the release manager, works with our various communities and committees and develops policies where needed. It is a rewarding and much appreciated way to give back to the Plone community.

The official announcement describes the nomination process and gives more information about the work involved in being a board member.

Please Consider Serving!

Plone 5.2.3, Plone 5.1.7 and Plone 4.3.20 released!

Posted by PLONE.ORG on November 20, 2020 08:15 AM

General notes:

Plone 5.2.3

Plone 5.2.3 is a bug fix release of Plone 5.2.

Download Plone 5.2.3

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.2.3/versions.cfg.

  • Linux/BSD/Unix users: Use the Unified Installer. It is a configuration and setup kit with build scripts.
  • Windows 10 users: Use the Unified Installer. See Windows-specific installation instructions. Consider using the Unified Installer within the Windows Subsystem for Linux (WSL).
  • OS X users: use the Vagrant kit or install XCode command-line tools and use the Unified Installer.
  • Automated provisioning: See Plone's Ansible Playbook for a full-stack installation kit.
  • Cross-platform Docker: install Docker and use the Plone Docker image.

For the Plone 5.2 upgrade guide, see https://docs.plone.org/manage/upgrading/

Some highlights of this release are:

  • zope.interface: Fixed potential memory leak, see https://github.com/zopefoundation/zope.interface/issues/216. Fixed inconsistent resolution orders, see https://github.com/zopefoundation/zope.interface/issues/199.
  • Zope: fixes for a few template syntax errors. HTTP header encoding support.
  • A few possible information disclosure problems in handling of XML and of ical urls were reported by MisakiKata. They have been fixed by the Plone Security Team. Since they require an attacker to already have Manager or Site Administrator rights, we decided it was not necessary to create a hotfix for this. See https://github.com/plone/Products.CMFPlone/issues/3209
  • plone.recipe.zope2instance: added options clear-untrusted-proxy-headers and max-request-body-size.
  • Products.MailHost: support messages with explicit Content-Transfer-Encoding 8bit, see https://github.com/zopefoundation/Products.MailHost/issues/30. Note: in add-ons this may require changes to the tests.
  • mockup: fix plone toolbar action links being updated only on the first navigation action in the folder_contents structure pattern.
  • plone.staticresources: updated Bootstrap Icons to 1.0.0 final.
  • plone.app.contenttypes: allow passing a custom catalog-query to migrateCustomAT to constrain which objects to migrate.
  • plone.dexterity: make sure that Dynamic schema is updated on all ZEO clients on change.
  • z3c.form/plone.app.z3cform: fixed compatibility with changed repeat syntax in Zope 4.4, see https://github.com/zopefoundation/z3c.form/issues/94.
  • Products.ATContentTypes: drop use of test() in templates, unsupported since Zope 4.4.
  • Lots of deprecation warnings fixed, especially during startup.

For detailed changelog, go to https://plone.org/download/releases/5.2.3

Plone 5.1.7

Plone 5.1.7 is a bugfix release of 5.1. Note: this is the last release in the 5.1 series.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/5.1.7/versions.cfg.

Some highlights are:

  • Integrate Plone20200121 hotfix.
  • Security: depend on Products.isurlinportal. Version 1.1.0 has hardening against white space.
  • plone.recipe.zeoserver: Windows fixes
  • mockup and plonetheme.barceloneta: various frontend fixes, including translations
  • Lots of translation updates
  • Lots of bug fixes in many packages.
  • plone.namedfile: Range support
  • plone.scale:
    The mode argument replaces the old, now deprecated, direction argument.
    The new names are contain or scale-crop-to-fit instead of down,
    cover or scale-crop-to-fill instead of up and scale instead of thumbnail.
  • plone.supermodel: added support for choices of integers for improved registry.xml export.
  • Products.PluggableAuthService: Added new events to be able to notify when a principal is added to or removed from a group.

For detailed changelog, go to https://plone.org/download/releases/5.1.7

Plone 4.3.20

Plone 4.3.20 is a bugfix release of Plone 4.3. Note: this is the last release in the 4.3 series.

Note that support for Python 2.6 was dropped a while ago. It might still work, but you should use Python 2.7.

Experienced users can update their buildout config by pointing to https://dist.plone.org/release/4.3.20/versions.cfg.

Some highlights are:

  • Integrated PloneHotfix20200121 for increased security.
  • Moved the security check if a url is in the portal to a small separate package: Products.isurlinportal. You can immediately use this on Plone 4.3 and higher. Keep an eye on updates for this package: newer versions will increase the security. Often the impact of fixes is too small to warrant a real security hotfix package, but we want to do more regular fixes here.
  • Use Products.isurlinportal 1.1.0 with security hardening against whitespace: https://github.com/plone/Products.isurlinportal/issues/1
  • Removed broken X-XSS-Protection header from classic theme and unstyled theme.
  • Products.PluggableAuthService: Added new events to be able to notify when a principal is added to or removed from a group. Notify these events when principals are added or removed to a group in ZODBGroupManager. See https://github.com/zopefoundation/Products.PluggableAuthService/issues/17
  • z3c.autoinclude: When environment variable Z3C_AUTOINCLUDE_DEBUG is set, log which packages are being automatically included.

For detailed changelog, go to https://plone.org/download/releases/4.3.20

Plone Connection Podcast: Episode 01 - Philip Bauer

Posted by Starzel.de on November 19, 2020 04:56 PM

The Plone Connection Podcast is a monthly podcast produced by Six Feet Up. Every month, Six Feet Up's Director of Engineering T. Kim Nguyen sits down with a different member of the Plone Community and asks them about their work with or on the Plone CMS.

Many thank to my good friend Kim for doing this!

Your first Plone 6 Project

Posted by Starzel.de on November 18, 2020 01:25 PM

I've had the opportunity to give this talk at the most excellent Python Web Conference.

Data-driven documents with Volto and Plone

Posted by PLONE.ORG on November 09, 2020 12:05 PM

Executive Summary

Innovations in the Plone environment continue to expand the platform's capacities to provide scalable solutions. The BISE project realized by Eau de Web demonstrates the potential of joining Plone's world-class CMS back-end with the flexibility of the Volto front-end user interface. In this case, the integration enabled a highly flexible display of data-driven dashboard information with factsheets that are editable through the web by content managers. Get the full story below.


An Innovative Approach to a Known Problem

The BISE website (biodiversity.europa.eu) is one of the thematic environmental information websites run by the European Commission (DG-ENV) in cooperation with the European Environmental Agency. It used to run on Plone 4, but as part of the obligatory upgrade to Plone 5, we (Eau de Web) have migrated to a system based on Plone 5 and Volto 8.

Product of almost one year of work, as it stands right now, BISE is yet another example of the benefits of bringing closer the frontend development world to the solid foundation provided by Plone. The highlights of the BISE portal are, probably, the Country Biodiversity Factsheets, available in the Countries section.

Screenshot_20201029_155123.png

A mix of data, charts, and specially created visualizations would surely imply, in classic Plone development fashion, a dedicated content type and Python code, but in our case, the content type has been created through the web and holds little logic or knowledge about the end result. Instead, the factsheets are fully editable and composable through the web, based on somewhat generic Volto blocks: “plain” rich text blocks, chart blocks, a couple of “data table” types, and a “columns block” to enable composing these in a more complex but mobile-responsive layout. For even more flexibility, we’ve created a generic block styling framework that can be used for any Volto block and enables basic styling, such as alignment, text colors or to choose from a palette of predefined styles.

Everything, starting with the data definition (which can be provided either as plain CSV documents uploaded as Files in Plone or as SQL queries for a custom content type that communicates with the SQL server via REST) is completely generic and editable through the web, as part of the Volto UI.

The newly arrived “Volto dexterity-based content layout” bridges the gap between the goodness of Volto blocks and the templating power of content types. Add on top of that a thin layer that automatically adjusts templated content data (text, charts, data tables) to the context metadata (in our case, country code) and we get a high productivity environment that empowers the content experts to provide significant enhancements to the website without relying on the developers

plotly.png

Volto-Slate and Other Contributions

One of the key components of this website is also invisible due to its ubiquity: a new rich text editor, the volto-slate. The Biodiversity website relies heavily on the text editor: the website has a lot of textual information and its editors need to rely on a feature-rich editor. Also, the country factsheets use “templated text”, predefined (but editable) text that embeds numeric facts that come from the data connectivity framework. These “data-connected text entities” need to be fully configurable and styled by the rich text editor.

Early in the development cycle and having the experience of a prototype built with Volto’s default Draft.js editor, the development team decided to abandon it and reimplement a drop-in replacement with another library: SlateJS. Rather than providing an editor with a default configuration, SlateJS is different: it comes with the building blocks and a rich React api to enable building any kind of richtext editor. The Volto blocks and its full-page editing philosophy are rather specific, so we needed a library that can be used for this kind of deeper integration.

In the end, the volto-slate editor reached a stage where it can provide a level of integration that exceeds that of Volto’s built-in editor: improved block splitting and joining, better pasting with block splitting, automatic image and table block extraction and better overall handling of lists and sublists, just to name a few. It is now a stable target and provides plugin primitives for other rich-text integration tasks, sprouting other Volto addons dedicated to it: volto-slate-zotero, volto-slate-metadata-mentions, etc.

Screenshot_20201028_150648.png

More additions to the overall Volto ecosystem

In fact, volto-slate constitutes a substantial part of the overall development effort, but it wasn’t the only contribution that the BISE project made to the overall Volto ecosystem. Among them we can list:

  • Major improvements to the Volto addons architecture
  • Various bug fixes and enhancements to Volto core
  • Contributions to various Volto-connected projects, such as plone.restapi, the yeoman-based Volto app template generator and the styleguidist documentation
  • Many addons available as open source from the EEA github repo:

dotted-table-chart.png

Overall, the BISE website delivers a unique product that could have been created only with Volto and Plone. Through the freedom that they provide, Volto blocks are a foundation for innovation that enables Plone to step in line with the latest state of the art for web development.

Overall, the BISE website delivers a unique product that could have been created only with Volto and Plone.

Eau de Web is a software company specialising in Web development. The drivers of our activities are the open-source culture and the use of open standards due to the benefits of code reuse, openness in participating in projects worldwide, peer review and contributions from a wide community of developers. Our group has used Plone for web development from the early days, Eau de Web is a Plone foundation member, and our team members continue to demonstrate a long-term commitment to share and contribute the knowledge, software and resources to benefit others through Plone’s open source community.

Mikel Larreategi becomes a member of the Plone Foundation

Posted by PLONE.ORG on October 29, 2020 03:02 PM

The Plone Foundation welcomes a new member, after unanimous confirmation by the Foundation's Board of Directors on October 22, 2020.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.


Mikel Larreategi

erral.jpgMikel started working with Plone with the 2.1 version after joining CodeSyntax. Mikel attended several Plone Conferences (Naples, Washington DC, Budapest, Bristol (twice), San Francisco, Arnhem, Bucharest, Barcelona and Ferrara, and, between other community activities, maintains the translations of Plone into Spanish and Basque.

The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.

The Plone Foundation encourages applications from long time contributors to the Plone project and community. Learn more about the Foundation membership and the application process.

Plone Conference 2020 - A Sneak Peek

Posted by PLONE.ORG on October 28, 2020 08:04 PM

Over 65 Talks Submitted

The topics range from Plone, Zope, Volto and Guillotina to Python and Pyramid, and from fancy JavaScript to cool case studies. There are 30 and 45 minute talk slots, as well as lightning talks. 

The talks are targeted at a variety of different audiences such as designers, developers, integrators, users, and beginners alike. There will be something for everyone and the schedule will be planned so, that it is easy to find suitable talks for every need!

Volto Talks Galore

Volto, Plone's React front end, continues to gain interest, and there will be plenty of talks to catch you up on the latest developments:

  • Volto FTW
  • Building Volto Addons
  • Volto: A Journey Towards Personalisation
  • Plone and Volto in a Jamstack Project

What's in Store for Plone 6

Many people are helping move Plone 6 closer to reality - here are some of the topics they will be presenting:

  • Plone 6 Theming: Barceloneta
  • Plone 6 Theming with Diazo
  • Modernizing Plone's Classic UI

Agile, Patterns, Projects

We will dive into what goes into a successful project - from project management to technical architecture to building community:

  • Second Guessing the Single-Page-App Pattern
  • Asking Questions for the Benefit of your Future Self - Growing with the Plone Community
  • Agile Race To Zero
  • The Effectiveness of Open Source to Achieve a Common Heritage for Cities

And More!

Here are a few more intriguing-sounding talks:

  • Guillotina: Real Use Cases & Roadmap
  • State of Pillow
  • Collaborating With Orchid Data
  • Rapidly Building An Extensible Corruption Tool
  • Oh, the Places We've Been! Plone 2001 - 2020

There will be something for everyone!

Get you Plone Conference tickets now at https://2020.ploneconf.org/

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'xxx@xxx.de': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'xx@xxx.de': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone xxx@xxx.de'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

moniviestin workflow menu

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

moniviestin workflow

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

moniviestin example

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

moniviestin content rule

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

moniviestin rpa poll

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

  1. Video author selects “Publish with subtitles”.
  2. Plone content rule calls Jenkins webhook to schedule a new robot.
  3. Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
  4. Jenkins schedules a new job to poll pending ASR jobs.
  5. Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
  6. Plone content rule notifies the video author with email that automatically generated subtitles are now available.
  7. ???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

study guide tabs

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

hasura view

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

hasura relationships

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

volto contents

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

  1. Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

plone dexterity editor ,

  1. Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

volto sisu connector

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

study guide edit

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

study guide image

Something old, something new and something blue.

The perfect match.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.
  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.
  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.
  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.


All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.


If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.

Plone Beethoven Sprint 2019

Posted by kitconcept GmbH on September 25, 2019 09:00 AM

beethoven sprint group image

21 developers from nine different countries gathered in Bonn, Germany between June 20th and 24th to work on implementing the upcoming Plone 6. The sprint at the office of the kitconcept GmbH has been declared a “strategic sprint” by the Plone framework team. Sprint topics included working on Volto, the Plone REST API, and Guillotina.

Volto

Volto is a new ReactJS-based frontend for Plone. It was started by Rob Gietema in 2017 and it is actively developed since then. Volto will become the default frontend for Plone 6. It implements a complete new user interface called Pastanaga UI, which was developed by Albert Casado.

Victor Fernandez de Alba and Rob Gietema led the efforts to further enhance Volto together with Paul Roeland, Nicola Zambello, Piero Nicolli, Janina Hard, Jakob Kahl, Thomas Kindermann, Maurizio Delmonte, Stefania Trabucchi, Rodrigo Ferreira de Souza, Fred van Dijk, and Steffen Lindner.

Pastanaga UI Toolbar

The new editing toolbar has been part of Volto right from the start. Though, so far we did not fully implement every detail that Albert has imagined for the Pastanaga UI toolbar. Victor has been working on this for quite a while already. During the sprint, he integrated his work and polished it. Victor and Rob also worked on a new sidebar that shows up on the right side of the Volto user interface that holds additional information like page metadata and controls the settings of the individual tiles.

volto toolbar The new Volto edit toolbar

New Tiles: Collection, Slider and Table

At the Plone conference in Tokyo, we came up with a list of tiles for the new Volto composite page editor that we would like to implement. On top of that list were the listing/collection tile, the slider tile and the table tile. Piero, Rodrigo, and Rob started to work on the listing/collection tile that is supposed to become a replacement for the Collection Type in Plone.

Jakob and Janina worked on copying over the code of a slider tile that was developed by kitconcept. Rob implemented the table tile. Before you could ask Rob about the progress, he already finished his work, like always…

volto table The new Volto table tile

Accessibility

Accessibility has always been a first class citizen in the Plone community. Plone is used by many government and public websites where accessibility is a basic requirement. Paul, Thomas, Timo, and Victor worked on improving the accessibility of Volto. Our goal is to fully support the WCAG 2.1 standard with the “AA” level of conformance.

Timo set up ESlint-based static code analysis checks and Cypress-based axe accessibility tests. Paul, Thomas and Victor then started to fix the reported accessibility issues. At the end of the sprint, we were able to fix all the reported violations. Our CI build now fails if new accessibility violations are committed.

Documentation

Sven Strack, the Plone documentation team leader, joined our sprint remotely from Amsterdam to discuss how to maintain and structure the Volto documentation in the future. We all agreed that we will use Markdown as format for our docs, since this is the de-facto standard in the JavaScript community as well as in most other Open Source communities. As much as we like Restructured Text, it seems that Markdown won that battle.

We currently use MKdocs for the Volto documentation and Styleguidist to document our React components. There are lots of different tools to choose from, Docz, MKDocs, MDX, Gatsby to just name a few. We agreed that we need to do more research to make an educated decision regarding our doc toolchain.

There will be a Google Season of Docs project this year where a technical writer will help us to enhance the Volto docs further. More Volto Features Nicola added a feature to display the currently used Volto version in the Plone control panel, he added animations for page transitions and added an Italian translation for Volto. Together with Victor, he worked on a toast component for user notifications.

More Volto Features

Piero worked on improving and fixing the event type view and added styling for the listing tile.

Victor and Steffen worked on the image sidebar.

Rodrigo added a feature that allows the user to use SHIFT+RETURN to create a new line in a Volto text. By default Volto creates a new text tile on RETURN. Rodrigo also looked into the Add-ons control panel work that Eric Steele started some time ago.

Nilesh joined us remotely from India and continued his work on the users and groups control panel for Volto.

Steffen fixed a bug in the subjects search. Rodrigo and Thomas looked into our Cypress-based acceptance tests.

Fred looked into what consequences it would have for us to get rid of the description tile and how we could customize the domain in Plone to allow to send portal emails via the Volto frontend.

Stefania and Maurizio worked on a product definition of Volto and an elevator pitch for clients and users. Together with Paul, Fred and Timo, they had a longer discussion about how to position Volto and Plone 6 in the market.

Plone REST API

The Plone REST API builds the foundation for the new Volto frontend and Plone 6. It has been under active development in the last four years and seen more than 80 releases in that time period. The REST API is stable, battle-tested and production-ready and part of the Plone core since the release of Plone 5.2.

Thomas Buchberger and Lukas Graf led the REST API efforts during the sprint and worked with Carsten Senger, Roel Bruggink, Janina Hard and Timo Stollenwerk to further enhance the REST API, fix bugs, improve the documentation, and the error handling.

API Team in the graden Plone REST API discussion in the garden

Querystring Endpoint for the Collection Type

One of the most important new features, that has been developed during the sprint, is the new querystring endpoint. This endpoint will allow us to implement the Collection type for the Volto frontend. Lukas implemented this essential feature with help from Rob, who took care of the frontend implementation in Volto.

New Features, Bugfixes, and first time contributions

Thomas added support for the retrieval of additional metadata fields in summaries in the same way as in search results. Lukas made the @types endpoint expandable and the @users endpoint easier to customize. Janina fixed setting the effective date on workflow transitions. This was her first contribution to the Plone core. On the same day she signed the contributor agreement, got her fix merged and released, which is quite an accomplishment for a single day. Way to go Janina!

In addition to that, the team fixed lots of smaller bugs (#780, #729, #764, #755, #777, #732, #738) during the sprint. REST API Error handling Carsten, Nathan, Thomas, Lukas and Timo had a longer discussion about error handling in the Plone REST API and about how we can improve things. We considered and discussed implementing RFC7807 and decided against adopting it.

Developer in office Sprint discussions…

The RFC did not see much adoption and it only slightly differs from what we are already doing. A breaking change does not seem to be worth the effort. We decided to create a separate error reporting component unifying the error reporting and ensuring consistency. We also started to fix a few inconsistencies right at the sprint.

API Documentation

Guillotina is using Swagger to document its API. Nathan explained the approach he took for that and Carsten started to integrate Swagger to generate dynamic api docs in plone.restapi.

REST API Releases

Most of the work we did on Plone REST API during the sprint has been released in Plone REST API 4.1.4, 4.2.0, 4.3.0, and 4.4.0.

Dexterity Site Root

Roel Bruggink joined us on Saturday and Sunday to continue his work on turning the Plone site root into a Dexterity content object. He fixed some complex recursion and acquisition problem when migrating the site root to Dexterity. His Plone Improvement Proposal (PLIP #2454) has been accepted by the Plone framework team and it will build the foundation for a new Plone 6 branch that we plan to cut this year.

Guillotina

Guillotina is a new AsyncIO REST data API. It has been written from the scratch by Ramon Navarro Bosch and Nathan van Gheem. It can easily scale billions of objects via its REST API. With the CMS addon, it is compatible with the Plone REST API. Therefore you can run Volto on a Guillotina backend and Guillotina might be able to replace our existing Plone backend in the future.

Guillotina Developer The Guillotina master thinking hard. :)

Nathan and Ramon worked on Guillotina 5. The Guillotina 5 release brought Python 3.7 support and the use of the context vars APIs. They refactored quite a bit, moved parts of addons into core(swagger, caching, pg catalog), implemented OOTB PostgreSQL catalog, added a Helm/Kubernetes/Docker configurations for the Guillotina CMS addon that can be configured to act as a Volto backend, worked on improving caching and filestorage.

Websockets for Plone

At the Plone Open Garden 2019, Asko Soukka started to work on bringing websocket support to Plone. Websockets allow bi-directional communication between the server and the browser. With websockets, the server can send messages and notifications to the browser. This can be used to implement notification systems or collaborative editing to just name a few possible applications.

Asko continued his work during the Beethoven sprint and created an example where comments that are added to Plone automatically pop up as toast notifications for other users. He published a Docker image with a Twisted-based ZServer and implemented the foundation to use that feature for gatsby-source-plone. This will allow us to create an extension that will allow editors to live-preview changes they do in Plone in a GatsbyJS site.

Developer in Office

Karaoke, Barbecue and Garden Parties

A Plone sprint is much more than just programming. Collaborating, meeting old friends, talk, discuss and having a good time together are the ingredients that make a Plone sprint. We started our sprint on Thursday with a free React and Volto training for newbies. At the evening when everybody arrived, we went to a local Karaoke bar for a revival of the epic Karaoke sessions we had in Tokyo last year.

Karaoke Karaoke at the “Nyx”

On Friday evening we ordered Pizza and hacked the night away at the office and our garden. On Saturday we went to the “Arithmeum”, a mathematics museum owned by the Research Institute for Discrete Mathematics at the University of Bonn.

Arithmeum Our tour at the “Arithmeum”

Afterwards we had a garden party with a barbecue and we spend the afternoon sitting in the garden and the office and continued to drink and hack on Plone.

Developer in garden Barbecue in the garden of the kitconcept office

Summary

The Beethoven Sprint 2019 was a major success. We were able to start working on the main missing pieces for Volto. The new edit toolbar is in place and the foundation for the new tiles/metadata toolbar on the right side of the UI has been finished. We also implemented a new table tile.

We implemented a proof-of-concept for the new listing / collection tile and implemented the required querystring endpoint in Plone REST API. Both Volto and the Plone REST API got lots of enhancements and bugfixes.

Both the Plone REST API and Volto are stable and used in production. We will use the next month to further enhance Volto and implement the missing bits and pieces. Our plan is to present Volto 4 at the Plone conference in Ferrara at the end of this year and cut a Plone 6 branch afterwards that contains the Dexterity Site root, folderish content types and other enhancements that are needed make Volto the default frontend for Plone 6.

Guillotina and the Websockets support for ZServer together with Gatsby as another frontend for Plone are amazing projects by the smartest folks in our community. Our diversity is one of our main strength and we are looking forward what the future will bring!

We had a great time at the sprint. Thank you to everyone who attended! We thank the Plone and Python community for their ongoing support. The Plone Foundation, the Python Software Foundation, and the German Python Software Verband made this event possible with their sponsorship as well as our company sponsors Abstract, Starzel, Werkbank and Zest.

We are looking forward to see you all in Bonn next year!

Developer in garden The “Hofgarten” near the kitconcept office

Digital Signage management with Plone and GatsbyJS

Posted by Asko Soukka on September 01, 2019 09:00 AM

Sometimes complex problems have simple solutions, and this was one of those times. At work, we are piloting a new digital signage solution for our university. Unfortunately, the new system lacks in permission management, and for a while, it looked like we were unable to safely delegate digital signage content management to departments.

Lucky us that we knew a system that is very good in permission management: Plone.

Plone – An enterprise open source content management system

Plone is an open source content management system based on hierarchical object database. That’s why Plone excels in managing many kind of content and related permissions in hierarchies. That’s also why Plone has been a good fit for such hierarchical organizations where content management follows the hierarchy of the organization itself.

Plone excels in managing content and related permissions in hierarchies

So, how we are going to solve the shortcomings of our digital signage system with Plone and GatsbyJS? In brief, we

  • let our users to manage their digital signage content on Plone in the simples possible way: every display group has their own folder on Plone and the responsible users have required rights to manage content in their folders
  • we use GatsbyJS project to fetch that content from Plone and build simple digital signage content players hosted as static web pages
  • we use our on-premises GitLab CI and GitLab pages to build and host those web pages; a Plone feature called content rules is used to trigger a new updating build after each content update.

With this setup, instead of letting the content managers to directly manage the digital signage system (where we were unable to restrict their permissions to their own display groups), we configure the system to display those GatsbyJS built sites and let the users manage their digital signage content on Plone – on the content management system they are already familiar with.

Simple digital signage content loop can be populated with just dropping images into a Plone folder

And that’s just the beginning. Thanks to Plone’s customizable structured content types, we already added new content field for setting the display time of each content page. And in the future we can use this to add more configuration options to ensure that the content is displayed as intended by the content managers.

Plone supports customizing structured content fields directly form the browser

Also built-in content types like first class Image type can be enhanced with custom fields

gatsby-theme-ds-player

Show the code or it didn’t happen!

To be honest, our current solution is so simple that there is not that much to show.

Anyway, I put together a simple open source version where similar GatbyJS built web site is created from given Plone folder so that each subfolder has its own digital signage player to loop through the images dropped into that folder.

I packaged all this into a GatsbyJS “theme” plugin gatsby-theme-ds-player preconfigured to be best compatible with a Plone source using gatsby-source-plone. There is also a live demo of the result, automatically built from the theme package repository at Travis-CI:

Using Glitch for Pyramid Learning

Posted by Jazkarta Blog on May 09, 2019 08:02 PM

Screenshot: Pyramid Quick Tutorials showing pyramid-view-decorators, pyramid-view-predicates, pyramid-view-renderers and pyramid-asset-specifications

At this year’s Jazkarta sprint we decided to invest some time in creating Pyramid learning resources. We are fans of the Pyramid web framework, and we felt that it would be nice to work on something that would make it easier to get started with, especially for novice developers.

Enter Glitch. This is a platform that is designed for letting users explore web apps created by other Glitch users. When you see an application that does something that you want to do, you can “remix” it, which means that an exact copy, with all its code and resources, is created in your own workspace (which is some sort of Docker container). You can then change it in any way you like. This is very powerful, because you get a working application that is hosted for you, and every code change that you make is immediately reflected in the web app page.

This turns out to be really useful for teaching web application development concepts, because students can see the code for the example, read an explanation, and then right away remix the code to try out their changes, which they can see working instantly. No need for setting up a workspace, checking out code, or running a web server. Just try out an idea and see the result right there, or share it with other people. Like “view HTML source” but with code, which is one of Glitch’s goals.

We decided to create a Glitch collection that would showcase a few of Pyramid’s most distinctive features, along with a simple Hello World style tutorial. The first thing was to verify that Pyramid could be installed. Glitch’s main development target is Node.js, but its containers install Python and they allow the user to run pip, so it’s possible to install most PyPI packages including Pyramid. (For those who might be wondering: sorry, Plone is not installable, because it needs system dependencies that are not installed by Glitch.)

Glitch supports a setup file named glitch.json, which is what allowed us to install Pyramid:

{
 "install": "pip3 install --user pyramid",
 "start": "python3 pyramid_app.py"
}

In the json above, you can see that we install Pyramid using pip, and we can add any other dependencies for our project there. We can then use the “start” key to tell Glitch how to start our application.

Once we did that, the only other thing needed was to create our Pyramid app, which in this case uses code directly lifted from the trypyramid.com site:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response

def hello_world(request):
    return Response('Hello Glitch')

if __name__ == '__main__':
    with Configurator() as config:
       config.add_route('hello', '/')
       config.add_view(hello_world, route_name='hello')
       app = config.make_wsgi_app()
    server = make_server('0.0.0.0', 3000, app)
    server.serve_forever()

That’s it. Glitch automatically runs the application (the only caveat is that it needs to use port 3000). You can configure it so that any code change automatically restarts the app. After this, if a potential new user sees your project, she can just use the remix button and have a copy made in her workspace. Any change that she makes, like changing the hello text, will be immediately visible on the page.

Overall, we found Glitch to be a very intuitive and easy to use service. You can see the collection of simple tutorials that we created here:

Pyramid Quick Tutorials

 

Jazkarta Goes To LA

Posted by Jazkarta Blog on May 09, 2019 07:47 PM

The Jazkarta team at LAX-C, a city block sized Asian food distributor and restaurant supply

This year Jazkarta’s annual sprint was held in the city of Los Angeles, graciously hosted by Alec Mitchell who lives a block away from LA’s City Hall.

View of the Los Angeles City Hall from Los Angeles Street

We had fun exploring the city – Union Station, Chinatown, LAX-C (pictured above, a gigantic Asian food distributor and restaurant supply), Grand Central Market, Griffith Observatory, and the Hollywood Farmers’ Market where we bought lots of delicious fruit.

Citrus and strawberries from the Hollywood Farmers' Market

We attended an improv comedy performance at the Upright Citizens Brigade Theater, ate lots of tacos, had original Phillipe’s French Dip sandwiches, visited brew pubs and drank lots of local beer.

The Jazkarta team in front of the Highland Park Brewery.

We even had a private tasting of cheese, beer, and wine pairings.

Plate with five different cheeses and a variety of fruits and nuts

Oh, and we also sprinted on a variety of topics!

Alec and Jesse Snyder ported several of our Plone add-ons to Python 3 and Plone 5.2:

Carlos de la Guardia created a set of Pyramid Quick Tutorials on Glitch – more about that coming in another post.

Matthew Wilkes and Jesse researched the best way to set up Pyramid functional tests using pytest and webtest in an efficient manner. They were unhappy with their previous model, which relied on in-memory databases. They settled on a pattern that switched out private variables of the sessionmaker to join a longer-running transaction for each test. This is more self-contained than adapting the session setup machinery in the app itself, and works transparently on different database backends. Matthew made a gist of the relevant code.

Alec made a prototype of an app to provide faceted search of bibliography references kept in Zotero. The app has a Elasticsearch backend and a React front end. It uses the Zotero API to pull the references into Elasticsearch for querying.

Carlos investigated what it would take to move the CastleCMS quality checking features into a Plone add-on. It’s a work in progress.

Left to right: Carlos, Sally, Matthew, Alec, Jesse and Nate in front of a painting of the Griffith Observatory

It’s wonderful to be able to work together in person, instead of remotely from our home offices spread across 3 countries and 2 continents. We try to do it once a year and always have a fantastic time. This year was no exception.

Sprinting in Sorrento

Posted by Jazkarta Blog on May 08, 2019 10:44 PM

Sunset over the island of Capri, as viewed from the sprint venue

This year I helped organize the annual sprint in Sorrento called the Plone Open Garden. Plone is a highly secure and feature rich open source content management system written in Python. Plone has no single company behind it, its IP is held by a non-profit foundation. Progress on the software happens through the efforts of a self-directed community, whose members sponsor sprints to work on particular problems.

PLOG took place April 12-19 and it was a great success, attended by 18 people from 7 countries. There were two training classes:

  • Porting Plone sites and add-ons to Python 3
  • Creating a web application using the restapi and Volto, the new React front end for Plone

We also accomplished a lot of work:

  • Python 3 ports of over a dozen Plone add-ons
  • Accessibility improvements for classic Plone and Volto front ends
  • New Volto features and documentation
  • Marketing discussions
  • Documentation discussions
  • …and various other Plone improvements

I posted a news item on plone.org that describes the details of what we accomplished. But Plone sprints have a social side as well as a technical side. Friendships are renewed, meals and news are shared, and a good time is had by all. This year’s PLOG was true to form. Here are some of the things we enjoyed.

1) Breakfasts at the Hotel Mediterraneo, the sprint venue since 2007. Pictured below is the pastry table; there are also tables for cheeses, meats, fruits, breads, cereals, eggs, and more. You can squeeze your own orange juice and optionally add some Prosecco.

Breakfast buffet at the Hotel Mediterraneo - the pastry table

Dinners, also included, are a wonderful way to end a day of sprinting. First and second course, plus dessert.

A typical dessert at the Hotel Mediterraneo

2) Happy hour at the Mediterraneo. We book the rooftop for happy hour at least once during the sprint, where we enjoy the sunset over Capri (pictured at the top of this post) and the extensive cocktail menu. That’s a Black Mohito at the top and a Negroni at the bottom.

 Drinks from the Hotel Mediterraneo cocktail menu

3) Excursions. One day we walked to Sorrento’s old harbor and had lunch with a view of Mount Vesuvius.

A view looking down at Sorrento's old harbor

4) Being with old friends and meeting new ones. Always the best part.

Sprinters on the rooftop of the Hotel Mediterraneo

 

Four options to try Plone 5.2 on Python 3

Posted by Starzel.de on January 24, 2019 08:30 AM

Demo pages

There are nightly build of the current Plone 5.2 coredev (the development version) with Python 2 and 3:

Minimal Buildout

Here is a minimal buildout to run Plone 5.2rc1 on Python 3:

[buildout]

parts = instance

extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg



[instance]

recipe = plone.recipe.zope2instance

eggs =

Plone

Pillow

You set it up like this:

$ python3.7 -m venv .

$ ./bin/pip install -r https://dist.plone.org/release/5.2rc1-pending/requirements.txt

$ ./bin/buildout

And start it as usual with ./bin/instance fg

Standalone Development buildout

[buildout]

extends = https://dist.plone.org/release/5.2rc1-pending/versions.cfg



parts =

    instance

    zopepy

    packages

    test

    robot



eggs =

    Plone

    Pillow

    collective.easyform



test-eggs =

    collective.easyform [test]



auto-checkout =

    collective.easyform



extensions =

    mr.developer



show-picked-versions = true



[instance]

recipe = plone.recipe.zope2instance

user = admin:admin

eggs = ${buildout:eggs}

debug-mode = on

verbose-security = on



[zopepy]

recipe = zc.recipe.egg

eggs =

    ${buildout:eggs}

interpreter = zopepy

scripts =

    zopepy

    plone-compile-resources



[packages]

recipe = collective.recipe.omelette

ignore-develop = False

eggs = ${buildout:eggs}

ignores = roman



[test]

recipe = collective.xmltestreport

eggs = ${buildout:test-eggs}

defaults = ['--auto-color', '--auto-progress']



[robot]

recipe = zc.recipe.egg

eggs =

    ${buildout:test-eggs}

    Pillow

    plone.app.robotframework[reload,debug]



[sources]

collective.easyform = git git@github.com:collective/collective.easyform.git branch=python3



[versions]

Starzel buildout

The buildout that we at Starzel.de use supports Plone 5.2rc1 with Python 2 and 3.

https://github.com/starzel/buildout

It has some nice features:

  • It extends to config- and version-files on github shared by all projects that use the same version of Plone.
  • It allows to update a project simply by changing the version it extends.
  • It allows to update all projects of one version by changing remote files (very useful for HotFixes).
  • It is minimal work to setup a new project.
  • It has presets for development, testing, staging and production.
  • It has all the nice development-helpers we use.

Quickstart:

$ git clone -b 5.2rc1.x https://github.com/starzel/buildout <SOME_PROJECT>

$ cd <MY_PROJECT>

Remove all files that are not needed for a project but are only used for the buildout itself.

$ rm -rf linkto README.rst README.txt .travis.yml secret.cfg_tmpl VERSION.txt local_coredev.cfg CHANGES.rst

If you're not developing the buildout itself you want a create a new git repo.

$ rm -rf .git && git init

Add a file that contains a passwort. Do not use admin as a password in production!

$ echo -e "[buildout]\nlogin = admin\npassword = admin" > secret.cfg

Symlink to the file that best fits you local environment. At first that is usually development. Later you can use production or test. This buildout only uses local.cfg and ignores all local_*.cfg.

$ ln -s local_develop.cfg local.cfg

Create a virtualenv in Python 2.7 or Python 3.7 (Plone 5.2 only).

$ virtualenv .  # for Python 2.7

$ python3.7 -m venv .  # for Python 3 (Plone 5.2 only)

Install and configure Plone

$ ./bin/pip install -r requirements.txt

$ ./bin/buildout

Wrapup

I hope these options help you get started with Python 3. For serious projects you will likely create you own buildout.

Update (4.2.2019)

You should use the Plone coredev for developing Plone itself:

$ git clone git@github.com:plone/buildout.coredev.git coredev
$ cd coredev
$ git checkout 5.2
$ python3.7 -m venv .
$ ./bin/pip install -r requirements.txt
$ ./bin/buildout
$ ./bin/instance fg

Update 2 (24.2.2019)

Since Plone 5.2b2 wsgi is the default, even when running Python 2. See https://github.com/plone/Products.CMFPlone/issues/2763

This means you do not need to enable wsgi = on since it will be enabled automatically. You can still choose to switch wsgi = off in Python 2 if you have a good reason to do so (e.g. you need FTP or WebDAV). I updated the examples accordingly.

Update 3 (8.3.2019)

Release candidate 1 of Plone 5.2 is pending. See https://community.plone.org/t/plone-5-2rc1-soft-released/8163. I updated the examples accordingly.