Planet Plone - Where Developers And Integrators Write

Matthew Wilkes Publishes Book: Advanced Python Development

Posted by PLONE.ORG on September 15, 2020 04:43 PM

As some Plone community members know, one of our own, Matthew Wilkes, has been hard at work writing a book for the last year. We're very excited to announce that it has been published! It is Advanced Python Development: Using Powerful Language Features in Real-World Applications and it can be purchased at Matthew's website, Apress or Amazon.

The book follows the design and build of a real-world application example from prototype to production quality. In this context, it explains Python language features that aren’t routinely covered and introduces techniques that demonstrate large scale system design and development processes. Plus there are useful asides, best practices and library recommendations galore.

Matthew specializes in Python development, especially in the areas of web security and performance. He has been developing Python web applications for 15 years and has been very active in the open-source community in general and the Plone community in particular.

Matthew began using Plone in 2004 when it was at version 2.0. He has worked on various aspects of the core Plone software and was on the Plone framework and security teams through the Plone 4 era. He led the security team for over four years, modernizing security team practices by instituting unit testing of fixes across all versions, cooperation with Red Hat and other downstream vendors, and a pre-announcement policy. He has also contributed to the security of Zope, Django CMS and Pyramid, including recently rewriting Pyramid’s CSRF protection. This all gives him a wider view of security as a process rather than just deep knowledge of one system.

Matthew served as the administrator for Plone's participation in Google Summer of Code for many years. He has also served on the Plone Foundation's Membership Committee and Board of Directors, and he was instrumental in helping to overcome the many legal and logistical hurdles to bringing Zope under the Plone Foundation's umbrella.

Matthew is a welcome and frequent presence at Plone conferences and sprints and has been known to be involved in adventures in late night drinking. For example after the party at the first Bristol conference, Matthew was instrumental in arranging for the pub where many Plonistas were hanging out to continue serving them after official closing time. The party continued until 5 in the morning and we learned later that the place took in £16,000 that night. Matthew also collaborated with David Glick on the classic zodb.ws April Fool’s Day launch in 2011 when they managed to get a hacked-up version of the ZODB running in the browser - a prank which required both technical chops and a particular sense of humor.

Congratulations to Matthew!

Plone Conference 2020, November 16 - 20 Will Be an Online Event!

Posted by PLONE.ORG on September 11, 2020 08:23 PM

Welcome to Plone Conference 2020 from where ever you are!

This year the conference will be held 100% online. Due to the corona situation, we have decided to cancel the conference as a physical event, but instead, have the Plone conference as an online event.

Some preliminary information (might change still):

  • 4 days of talks + 1 of open spaces.
  • Training will happen over the weekend.
  • Powered by Loudswarm system developed by SixFeetUp.

The conference will be held from 16th to 20th November 2020. No matter where you are, you can participate.

Stay tuned for more information soon! Call for papers and registrations will be open in the next weeks.

https://2020.ploneconf.org

And one more thing...

Plone Conference 2021 will be held in Namur, Belgium!

Stay tuned for more information!

Plone Conference 2021 teaser

ZODB Database debugging

Posted by Starzel.de on August 24, 2020 11:00 AM

The problem

The ZODB contains python objects serializes as pickles. When a object is loaded/used a pickle is deserialized ("unpickled") into a python object.

A ZODB can contain objects that cannot be loaded. Reasons for that may be:

  • Code could not be loaded that is required to unpickle the object (e.g. removed packages, modules or classes)
  • Objects are referenced but missing from the database (e.g. a blob is missing)
  • The objects contains invalid entries (e.g. a reference to a oid that uses a no longer supported format)

The most frequent issues are caused by:

  • Improperly uninstalled or removed packages (e.g. Archetypes, ATContentTypes, CMFDefault, PloneFormGen etc.)
  • Code has changed but not all objects that rely on that code as updated
  • Code was refactored and old imports are no longer working

You should not blame the migration to Python 3 for these issues! Many issues may already exists in their database before the migration but people usually do not run checks to find issues. After a migration to Python 3 most people check their database for the first time. This may be because the documentation on python3-migration recommends running the tool zodbverify.

Real problems may be revealed at that point, e.g when:

  • Packing the Database fails
  • Features fail

You can check you ZODB for problems using the package zodbverify. To solve each of the issues you need to be able to answer three questions:

  1. Which object is broken and what is the error?
  2. Where is the object and what uses it?
  3. How do I fix it?

In short these approaches to fixing exist:

  1. Ignore the errors
  2. Add zodbupgrade mappings
  3. Patch your python-path to work around the errors
  4. Replace broken objects with dummies
  5. Remove broken objects the hard way
  6. Find our what and where broken objects are and then fix or remove them safely

I will mostly focus on the last approach.

But before you spend a lot of time to investigate individual errors it would be a good idea to deal with the most frequent problems, especially IntIds and Relations (see the chapter "Frequent Culprits") below. In my experience these usually solved most issues.

Find out what is broken

Check your entire database

Use zodbverify to verify a ZODB by iterating and loading all records. zodbverify is available as a standalone script and as addon for plone.recipe.zope2instance. Use the newest version!

In the simplest form run it like this:

$ bin/zodbverify -f var/filestorage/Data.fs

It will return:

  • a list of types of errors
  • the number of occurences
  • all oids that raise that error on loading

Note

zodbverify is only available for Plone 5.2 and later. For older Plone-Versions use the scripts fstest.py and fsrefs.py from the ZODB package:

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fstest.py var/filestorage/Data.fs

$ ./bin/zopepy ./parts/packages/ZODB/scripts/fsrefs.py var/filestorage/Data.fs

The output of zodbverify might look like this abbreviated example from a medium-sized intranet (1GB Data.fs, 5GB blobstorage) that started with Plone 4 on Archetypes and was migrated to Plone 5.2 on Python 3 and Dexterity:

$ ./bin/zodbverify -f var/filestorage/Data.fs



[...]



INFO:zodbverify:Done! Scanned 163955 records.

Found 1886 records that could not be loaded.

Exceptions, how often they happened and which oids are affected:



ModuleNotFoundError: No module named 'Products.Archetypes': 1487

0x0e00eb 0x0e00ee 0x0e00ef 0x0e00f0 0x0e00f1 0x2b194b 0x2b194e 0x2b194f 0x2b1950 [...]



ModuleNotFoundError: No module named 'Products.PloneFormGen': 289

0x2b1940 0x2b1941 0x2b1942 0x2b1943 0x2b1944 0x2b1974 0x2b1975 0x2b1976 0x2b1977 [...]



AttributeError: module 'App.interfaces' has no attribute 'IPersistentExtra': 34

0x2c0a69 0x2c0a6b 0x2c0ab7 0x2c0ab9 0x2c555d [...] 0x35907f



ModuleNotFoundError: No module named 'Products.CMFDefault': 20

0x011e 0x011f 0x0120 0x0121 0x0122 0x0123 0x0124 0x0125 0x0126 0x0127 0x0128 0x0129 0x012a 0x012b 0x012c 0x012d 0x012e 0x012f 0x0130 0x0131



ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 20

0x3b1cde 0x3b1ce0 0x3b1ce4 0x3b1ce6 0x3b1ce9 0x3b1ceb 0x3b1cee 0x3b1cf0 0x3b1cf4 0x3b1cf6 0x3b1cf9 0x3b1cfb 0x3b1cfe 0x3b1d00 0x3b1d04 0x3b1d06 0x3b1d09 0x3b1d0b 0x3b1d0e 0x3b1d10



ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

0x2c0a68 0x2c0a6a 0x2c555c 0x2c555e 0x2c560b 0x2c560d 0x2c5663 0x2c5665 0x2c571b 0x2c571d 0x2c5774 0x2c5776 0x2c5833 0x2c5835 0x33272d 0x33272f



ModuleNotFoundError: No module named 'fourdigits': 8

0x28030f 0x280310 0x280311 0x280312 0x280313 0x280314 0x280315 0x280316



ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

0x0e00e9 0x0e011a 0x0e01b3 0x0e0cb3



AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

0x2a712b 0x2a712c 0x2a712d



ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

0x11



ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

0x25



ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

0x37



ModuleNotFoundError: No module named 'plone.app.controlpanel': 1

0x0f4c2b



ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

0x3b1311

You can see all different types of errors that appear and which objects are causing them. Objects are referenced by their oid in the ZODB. See the Appendix on how to deal with oids.

You can see that among other issues there are still a lot of references to Archetypes and PloneFormGen (I omitted the complete lists) even though both are no longer used in the site.

Before the summary the log dumps a huge list of errors that contain the pickle and the error:

INFO:zodbverify:

Could not process unknown record 0x376b77 (b'\x00\x00\x00\x00\x007kw'):

INFO:zodbverify:b'\x80\x03cProducts.PloneFormGen.content.thanksPage\nFormThanksPage\nq\x00.\x80\x03}q\x01(X\x0c\x00\x00\x00showinsearchq\x02\x88X\n\x00\x00\x00_signatureq\x03C\x10\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7q\x04X\r\x00\x00\x00creation_dateq\x05cDateTime.DateTime\nDateTime\nq\x06)\x81q\x07GA\xd6\xdf_\xba\xd56"\x89X\x05\x00\x00\x00GMT+2q\x08\x87q\tbX\r\x00\x00\x00marshall_hookq\nNX\n\x00\x00\x00showFieldsq\x0b]q\x0cX\x02\x00\x00\x00idq\rX\t\x00\x00\x00thank-youq\x0eX\x11\x00\x00\x00_at_creation_flagq\x0f\x88X\x11\x00\x00\x00modification_dateq\x10h\x06)\x81q\x11GA\xd6\xdf_\xba\xd7\x15r\x89h\x08\x87q\x12bX\x05\x00\x00\x00titleq\x13X\x05\x00\x00\x00Dankeq\x14X\x0f\x00\x00\x00demarshall_hookq\x15NX\x0e\x00\x00\x00includeEmptiesq\x16\x88X\x0e\x00\x00\x00thanksEpilogueq\x17C\x08\x00\x00\x00\x00\x007k\xaaq\x18cProducts.Archetypes.BaseUnit\nBaseUnit\nq\x19\x86q\x1aQX\x07\x00\x00\x00showAllq\x1b\x88X\x12\x00\x00\x00_EtagSupport__etagq\x1cX\r\x00\x00\x00ts34951147.36q\x1dX\x0b\x00\x00\x00portal_typeq\x1eX\x0e\x00\x00\x00FormThanksPageq\x1fX\x0b\x00\x00\x00searchwordsq C\x08\x00\x00\x00\x00\x007k\xabq!h\x19\x86q"QX\x07\x00\x00\x00_at_uidq#X \x00\x00\x00a2d15a36a521471daf2b7005ff9dbc62q$X\r\x00\x00\x00at_referencesq%C\x08\x00\x00\x00\x00\x007k\xacq&cOFS.Folder\nFolder\nq\'\x86q(QX\x0e\x00\x00\x00thanksPrologueq)C\x08\x00\x00\x00\x00\x007k\xadq*h\x19\x86q+QX\x0f\x00\x00\x00noSubmitMessageq,C\x08\x00\x00\x00\x00\x007k\xaeq-h\x19\x86q.QX\x03\x00\x00\x00_mdq/C\x08\x00\x00\x00\x00\x007k\xafq0cPersistence.mapping\nPersistentMapping\nq1\x86q2QX\x12\x00\x00\x00__ac_local_roles__q3}q4X\x16\x00\x00\x00xxx@xxx.deq5]q6X\x05\x00\x00\x00Ownerq7asu.'

INFO:zodbverify:Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 62, in verify_record

    class_info = unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'Products.PloneFormGen'

Inspecting a single object

In this case the object with the oid 0x376b77 seems to be a FormThanksPage from Products.PloneFormGen. But wait! You deleted all of these, so where in the site is it?

If the offending object is normal content the solution is mostly simple. You can call obj.getPhysicalPath() to find out where it is. But ore often than not editing and saving will fix the problem. In other cases you might need to copy the content to a new item and delete the broken object.

But usually it is not simply content but something else. Here are some examples:

  • A annotation on a object or the portal
  • A relationvalue in the relatopn-catalog
  • A item in the IntId catalog
  • A old revision of content in CMFEditions
  • A configuration-entry in portal_properties or in portal_registry

The hardest part is to find out what and where the broken object actually is before removing or fixing it.

The reason for that is that a entry in the ZODB does not know about it's parent. Acquisition finds parents with obj.aq_parent() but many items are not-Acquisition-aware. Only the parents that reference objects know about them.

A object x could be the attribute some_object on object y but you will not see that by inspecting x. Only y knows that x is y.some_object.

A way to work around this is used by the script fsoids.py on ZODB. It allows you to list all incoming and outgoing references to a certain object.

With this you will see that x is referenced by y. With this information you can then inspect the object y and hopefully see how x is set on y.

More often than not y is again not a object in the content-hierarchy but maybe a BTree of sorts, a pattern that is frequently used for effective storage of many items. Then you need to find out the parent of y to be able to fix x.

And so forth. It can a couple of steps until you end up in a item that can be identified, e.g. portal_properties or RelationCatalog and usually only exists once in a database.

To make the process of finding this path less tedious I extended zodbverify in https://github.com/plone/zodbverify/pull/8 with a feature that will show you all parents and their parents in a way that allows you to see where in the tree is it.

Before we look at the path of 0x376b77 we'll inspect the object.

Pass the oid and the debug-flag -D to zodbverify with ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D:

$ ./bin/zodbverify -f var/filestorage/Data.fs -o 0x376b77 -D



INFO:zodbverify:Inspecting 0x376b77:

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

INFO:zodbverify:

Object as dict:

{'__Broken_newargs__': (), '__Broken_state__': {'showinsearch': True, '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7', 'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'), 'marshall_hook': None, 'showFields': [], 'id': 'thank-you', '_at_creation_flag': True, 'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'), 'title': 'Danke', 'demarshall_hook': None, 'includeEmpties': True, 'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>, 'showAll': True, '_EtagSupport__etag': 'ts34951147.36', 'portal_type': 'FormThanksPage', 'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>, '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62', 'at_references': <Folder at at_references>, 'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>, 'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>, '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>, '__ac_local_roles__': {'xxx@xxx.de': ['Owner']}}}

INFO:zodbverify:

The object is 'obj'

[2] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(118)verify_oid()

-> pickle, state = storage.load(oid)

(Pdb++)

Even before you use the provided pdb to inspect it you can see that it is of the class persistent broken, a way of the ZODB to give you access to objects even though their class can no longer be imported.

You can now inspect it:

(Pdb++) obj

<persistent broken Products.PloneFormGen.content.thanksPage.FormThanksPage instance b'\x00\x00\x00\x00\x007kw'>

(Pdb++) pp obj.__dict__

{'__Broken_newargs__': (),

 '__Broken_state__': {'_EtagSupport__etag': 'ts34951147.36',

                      '__ac_local_roles__': {'xx@xxx.de': ['Owner']},

                      '_at_creation_flag': True,

                      '_at_uid': 'a2d15a36a521471daf2b7005ff9dbc62',

                      '_md': <Persistence.mapping.PersistentMapping object at 0x111617c80 oid 0x376baf in <Connection at 11094a550>>,

                      '_signature': b'\xd9uH\xc0\x81\x14$\xf5W:C\x80x\x183\xc7',

                      'at_references': <Folder at at_references>,

                      'creation_date': DateTime('2018/08/22 17:19:7.331429 GMT+2'),

                      'demarshall_hook': None,

                      'id': 'thank-you',

                      'includeEmpties': True,

                      'marshall_hook': None,

                      'modification_date': DateTime('2018/08/22 17:19:7.360684 GMT+2'),

                      'noSubmitMessage': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xae'>,

                      'portal_type': 'FormThanksPage',

                      'searchwords': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xab'>,

                      'showAll': True,

                      'showFields': [],

                      'showinsearch': True,

                      'thanksEpilogue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xaa'>,

                      'thanksPrologue': <persistent broken Products.Archetypes.BaseUnit.BaseUnit instance b'\x00\x00\x00\x00\x007k\xad'>,

                      'title': 'Danke'}}

If you now choose to continue (by pressing c) zodbverify it will try to disassemble the pickle. That is very useful for in-depth debugging but out of the scope of this documentation.

Inspect the path of references

Now you know it is broken but you still don't know where this ominous FormThanksPage actually is.

Continue to let zodbverify find the path to the object:

INFO:zodbverify:Building a reference-tree of ZODB...

[...]

INFO:zodbverify:Created a reference-dict for 163955 objects.



INFO:zodbverify:

This oid is referenced by:



INFO:zodbverify:0x376ada BTrees.IOBTree.IOBucket at level 1

INFO:zodbverify:0x28018c BTrees.IOBTree.IOBTree at level 2

INFO:zodbverify:0x280184 five.intid.intid.IntIds at level 3

INFO:zodbverify:0x1e five.localsitemanager.registry.PersistentComponents at level 4

INFO:zodbverify:0x11 Products.CMFPlone.Portal.PloneSite at level 5

INFO:zodbverify:0x01 OFS.Application.Application at level 6

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f6 persistent.mapping.PersistentMapping at level 7

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at level 8

INFO:zodbverify: 8< --------------- >8 Stop at root objects



INFO:zodbverify:0x02f5 plone.app.redirector.storage.RedirectionStorage at level 6

INFO:zodbverify:0x02fa zope.ramcache.ram.RAMCache at level 7

INFO:zodbverify:0x02fd plone.contentrules.engine.storage.RuleStorage at level 8

INFO:zodbverify:0x338f13 plone.app.contentrules.rule.Rule at level 9

INFO:zodbverify:0x0303 BTrees.OOBTree.OOBTree at level 10

INFO:zodbverify:0x346961 plone.app.contentrules.rule.Rule at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x02fe plone.app.viewletmanager.storage.ViewletSettingsStorage at level 9

INFO:zodbverify:0x034d plone.keyring.keyring.Keyring at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x376864 BTrees.IOBTree.IOBucket at level 3

INFO:zodbverify:0x31049f BTrees.IOBTree.IOBucket at level 4

INFO:zodbverify:0x325823 BTrees.IOBTree.IOBucket at level 5

INFO:zodbverify:0x3984c8 BTrees.IOBTree.IOBucket at level 6

INFO:zodbverify:0x2cce9a BTrees.IOBTree.IOBucket at level 7

INFO:zodbverify:0x2c6669 BTrees.IOBTree.IOBucket at level 8

INFO:zodbverify:0x2c62b4 BTrees.IOBTree.IOBucket at level 9

INFO:zodbverify:0x2c44c1 BTrees.IOBTree.IOBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x377536 BTrees.OIBTree.OIBucket at level 2

INFO:zodbverify:0x376b14 BTrees.OIBTree.OIBucket at level 3

INFO:zodbverify:0x376916 BTrees.OIBTree.OIBucket at level 4

INFO:zodbverify:0x376202 BTrees.OIBTree.OIBucket at level 5

INFO:zodbverify:0x373fa7 BTrees.OIBTree.OIBucket at level 6

INFO:zodbverify:0x37363a BTrees.OIBTree.OIBucket at level 7

INFO:zodbverify:0x372f26 BTrees.OIBTree.OIBucket at level 8

INFO:zodbverify:0x372cc8 BTrees.OIBTree.OIBucket at level 9

INFO:zodbverify:0x36eb86 BTrees.OIBTree.OIBucket at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify: 8< --------------- >8 Stop after level 10!



INFO:zodbverify:0x407185 BTrees.OIBTree.OIBTree at level 10

INFO:zodbverify: 8< --------------- >8 Stop after level 10!

You can see from the logged messages that the FormThanksPage is in a IOBucket which again is in a IOBTree which is in a object of the class five.intid.intid.IntIds which is part if the component-registry in the Plone site.

This means there is a reference to a broken object in the IntId tool. How to solve all these is covered below in the chapter "Frequent Culprits".

Decide how and if to fix it

In this case the solution is clear (remove refs to broken objects from the intid tool). But that is only one approach.

Often the solution is not presented like this (the solution to intid was not obvious to me until I spent considerable time to investigate).

The following six options to deal with these problems exists. Spoiler: Option 6 is the best approach in most cases but the other also have valid use-cases.

Option 1: Ignoring the errors

I do that a lot. Especially old databases that were migrated all the may from Plone 2 or 3 up to the current version have issues. If these issues never appear during operation and if clients have no budget or interest in fixing them you can leave them be. If they do not hurt you (e.g. you cannot pack your database or features actually fail) you can choose to ignore them.

At some point later they might appear and it may be a better time to fix them. I spent many hours fixing issues that will never show during operation.

Option 2: Migrating/Fixing a DB with zodbupdate

Use that when a module or class has moved or was renamed.

Docs: https://github.com/zopefoundation/zodbupdate

You can change objects in DB according to rules:

  • When a import has moved use a rename mapping
  • To specify if a obj needs to be decoded decode mapping

Examples from Zope/src/OFS/__init__.py:

zodbupdate_decode_dict = {

    'OFS.Image File data': 'binary',

    'OFS.Image Image data': 'binary',



    'OFS.Application Application title': 'utf-8',

    'OFS.DTMLDocument DTMLDocument title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod title': 'utf-8',

    'OFS.DTMLMethod DTMLMethod raw': 'utf-8',

    'OFS.Folder Folder title': 'utf-8',

    'OFS.Image File title': 'utf-8',

    'OFS.Image Image title': 'utf-8',

    'OFS.Image Pdata title': 'utf-8',

    'OFS.Image Pdata data': 'binary',

    'OFS.OrderedFolder OrderedFolder title': 'utf-8',

    'OFS.userfolder UserFolder title': 'utf-8',

}



zodbupdate_rename_dict = {

    'webdav.LockItem LockItem': 'OFS.LockItem LockItem',

}

You can specify your own mappings in your own packages. These mappings need to be registered in setup.py so zodbupdate will pick them up.

Rename mapping example: https://github.com/zopefoundation/Zope/commit/f677ed7

Decode mapping example: https://github.com/zopefoundation/Products.ZopeVersionControl/commit/138cf39

Option 3: Work around with a patch

You can inject a module to work around missing or moved classes or modules.

The reason to want do this is usually because then you can safely delete items after that. They don't hurt your performance.

Examples in __init__.py:

# -*- coding: utf-8 -*-

from OFS.SimpleItem import SimpleItem

from plone.app.upgrade.utils import alias_module

from plone.app.upgrade import bbb

from zope.interface import Interface





class IBBB(Interface):

    pass





class BBB(object):

    pass





SlideshowDescriptor = SimpleItem





# Interfaces

try:

    from collective.z3cform.widgets.interfaces import ILayer

except ImportError:

    alias_module('collective.z3cform.widgets.interfaces.ILayer', IDummy)





try:

    from App.interfaces import IPersistentExtra

except ImportError:

    alias_module('App.interfaces.IPersistentExtra', IDummy)





try:

    from webdav.interfaces import IDAVResource

except ImportError:

    alias_module('webdav.interfaces.IDAVResource', IDummy)





# SimpleItem

try:

    from collective.easyslideshow.descriptors import SlideshowDescriptor

except ImportError:

    alias_module('collective.easyslideshow.descriptors.SlideshowDescriptor', SlideshowDescriptor)





# object

try:

    from collective.solr import interfaces

except ImportError:

    alias_module('collective.solr.indexer.SolrIndexProcessor', BBB)





try:

    from Products.CMFPlone import UndoTool

except ImportError:

    sys.modules['Products.CMFPlone.UndoTool'] = bbb

More: https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/patches.py

Plone has plenty of these (see https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/__init__.py)

Option 4: Replace broken objects with a dummy

If a objects is missing (i.e. you get a POSKeyError) or broken beyond repair you can choose to replace it with a dummy.

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    dummy = Persistent()

    dummy._p_oid = p64(oid)

    dummy._p_jar = app._p_jar

    app._p_jar._register(dummy)

    app._p_jar._added[dummy._p_oid] = dummy

transaction.commit()

You shoud be aware that the missing or broken object will be gone forever after you didi this. So before you choose to go down this path you should try to find out what the object in question actually was.

Option 5: Remove broken objects from db

from persistent import Persistent

from ZODB.utils import p64

import transaction



app = self.context.__parent__

broken_oids = [0x2c0ab6, 0x2c0ab8]



for oid in broken_oids:

    root = connection.root()

    del app._p_jar[p64(oid)]

transaction.commit()

I'm not sure if that is a acceptable approach under any circumstance since this will remove the pickle but not all references to the object. It will probably lead to PosKeyErrors.

Option 6: Manual fixing

This is how you should deal with most problems.

The way to go

  1. Use zodbverify to get all broken objects
  2. Pick one error-type at a time
  3. Use zodbverify with -o <OID> -D to inspect one object and find out where that object is referenced
  4. If you use fsoids.py follow referenced by until you find where in the tree the object lives. zodbverify will try to do it for you.
  5. Remove or fix the object (using a upgrade-step, pdb or a rename mapping)

Find out which items are broken

The newest version of zodbverify has a feature to that does the same task we discussed in Example 1 for you. Until it is merged and released you need to use the branch show_references from the pull-request https://github.com/plone/zodbverify/pull/8

When inspecting a individual oid zodbverify builds a dict of all references for reverse-lookup. Then it recursively follow the trail of references to referencing items up to the root. To prevent irrelevant and recursive entries it aborts after level 600 and at some root-objects because these usually references a lot and would clutter the result with irrelevant information.

The output should give you a pretty good idea where in the object-tree a item is actually located, how to access and fix it.

If 0x3b1d06 is the broken oid inspect it with zodbverify:

$ ./bin/instance zodbverify -o 0x3b1d06 -D



2020-08-24 12:19:32,441 INFO    [Zope:45][MainThread] Ready to handle requests

2020-08-24 12:19:32,442 INFO    [zodbverify:222][MainThread]

The object is 'obj'

The Zope instance is 'app'

[4] > /Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify_oid.py(230)verify_oid()

-> pickle, state = storage.load(oid)



(Pdb++) obj

<BTrees.OIBTree.OITreeSet object at 0x110b97ac0 oid 0x3b1d06 in <Connection at 10c524040>>



(Pdb++) pp [i for i in obj]

[<InterfaceClass OFS.EtagSupport.EtagBaseInterface>,

 [...]

 <class 'webdav.interfaces.IDAVResource'>,

 <InterfaceClass plone.dexterity.interfaces.IDexterityContent>,

 <InterfaceClass plone.app.relationfield.interfaces.IDexterityHasRelations>,

 [...]

 <SchemaClass plone.supermodel.model.Schema>]

The problem now is that obj has no __parent__ so you have no way of knowing what you're actually dealing with.

When you press c for continue zodbverify will proceed and load the pickle:

(Pdb++) c

2020-08-24 12:20:50,784 INFO    [zodbverify:68][MainThread]

Could not process <class 'BTrees.OIBTree.OITreeSet'> record 0x3b1d06 (b'\x00\x00\x00\x00\x00;\x1d\x06'):

2020-08-24 12:20:50,784 INFO    [zodbverify:69][MainThread] b'\x80\x03cBTrees.OIBTree\nOITreeSet\nq\x00.\x80\x03(cOFS.EtagSupport\nEtagBaseInterface\nq\x01cAcquisition.interfaces\nIAcquirer\nq\x02cplone.app.dexterity.behaviors.discussion\nIAllowDiscussion\nq\x03czope.annotation.interfaces\nIAnnotatable\nq\x04czope.annotation.interfaces\nIAttributeAnnotatable\nq\x05cplone.uuid.interfaces\nIAttributeUUID\nq\x06cProducts.CMFDynamicViewFTI.interfaces\nIBrowserDefault\nq\x07cProducts.CMFCore.interfaces\nICatalogAware\nq\x08cProducts.CMFCore.interfaces\nICatalogableDublinCore\nq\tczope.location.interfaces\nIContained\nq\ncProducts.CMFCore.interfaces\nIContentish\nq\x0bcOFS.interfaces\nICopySource\nq\x0ccwebdav.interfaces\nIDAVResource\nq\rcplone.dexterity.interfaces\nIDexterityContent\nq\x0ecplone.app.relationfield.interfaces\nIDexterityHasRelations\nq\x0fcplone.dexterity.interfaces\nIDexterityItem\nq\x10cplone.app.iterate.dexterity.interfaces\nIDexterityIterateAware\nq\x11cplone.dexterity.interfaces\nIDexteritySchema\nq\x12cplone.app.contenttypes.interfaces\nIDocument\nq\x13cProducts.CMFCore.interfaces\nIDublinCore\nq\x14cProducts.CMFCore.interfaces\nIDynamicType\nq\x15cplone.app.dexterity.behaviors.exclfromnav\nIExcludeFromNavigation\nq\x16cz3c.relationfield.interfaces\nIHasIncomingRelations\nq\x17cz3c.relationfield.interfaces\nIHasOutgoingRelations\nq\x18cz3c.relationfield.interfaces\nIHasRelations\nq\x19cplone.namedfile.interfaces\nIImageScaleTraversable\nq\x1acOFS.interfaces\nIItem\nq\x1bcplone.app.iterate.interfaces\nIIterateAware\nq\x1ccplone.portlets.interfaces\nILocalPortletAssignable\nq\x1dczope.location.interfaces\nILocation\nq\x1ecOFS.interfaces\nIManageable\nq\x1fcProducts.CMFCore.interfaces\nIMinimalDublinCore\nq cProducts.CMFCore.interfaces\nIMutableDublinCore\nq!cProducts.CMFCore.interfaces\nIMutableMinimalDublinCore\nq"cplone.app.content.interfaces\nINameFromTitle\nq#cApp.interfaces\nINavigation\nq$cProducts.CMFCore.interfaces\nIOpaqueItemManager\nq%cAccessControl.interfaces\nIOwned\nq&cAccessControl.interfaces\nIPermissionMappingSupport\nq\'cpersistent.interfaces\nIPersistent\nq(cOFS.interfaces\nIPropertyManager\nq)cplone.app.relationfield.behavior\nIRelatedItems\nq*cAccessControl.interfaces\nIRoleManager\nq+cplone.contentrules.engine.interfaces\nIRuleAssignable\nq,cProducts.CMFDynamicViewFTI.interfaces\nISelectableBrowserDefault\nq-cOFS.interfaces\nISimpleItem\nq.cplone.app.contenttypes.behaviors.tableofcontents\nITableOfContents\nq/cOFS.interfaces\nITraversable\nq0cplone.uuid.interfaces\nIUUIDAware\nq1cProducts.CMFEditions.interfaces\nIVersioned\nq2cplone.app.versioningbehavior.behaviors\nIVersioningSupport\nq3cProducts.CMFCore.interfaces\nIWorkflowAware\nq4cOFS.interfaces\nIWriteLock\nq5cOFS.interfaces\nIZopeObject\nq6czope.interface\nInterface\nq7cplone.dexterity.schema.generated\nPlone_0_Document\nq8cplone.supermodel.model\nSchema\nq9tq:\x85q;\x85q<\x85q=.'

2020-08-24 12:20:50,786 INFO    [zodbverify:70][MainThread] Traceback (most recent call last):

  File "/Users/pbauer/workspace/dipf-intranet/src-mrd/zodbverify/src/zodbverify/verify.py", line 64, in verify_record

    unpickler.load()

  File "/Users/pbauer/.cache/buildout/eggs/ZODB-5.5.1-py3.8.egg/ZODB/_compat.py", line 62, in find_class

    return super(Unpickler, self).find_class(modulename, name)

ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package



    0: \x80 PROTO      3

    2: (    MARK

    3: c        GLOBAL     'OFS.EtagSupport EtagBaseInterface'

   38: q        BINPUT     1

   40: c        GLOBAL     'Acquisition.interfaces IAcquirer'

   74: q        BINPUT     2

   76: c        GLOBAL     'plone.app.dexterity.behaviors.discussion IAllowDiscussion'

  135: q        BINPUT     3

  137: c        GLOBAL     'zope.annotation.interfaces IAnnotatable'

  178: q        BINPUT     4

  180: c        GLOBAL     'zope.annotation.interfaces IAttributeAnnotatable'

  230: q        BINPUT     5

  232: c        GLOBAL     'plone.uuid.interfaces IAttributeUUID'

  270: q        BINPUT     6

  272: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces IBrowserDefault'

  327: q        BINPUT     7

  329: c        GLOBAL     'Products.CMFCore.interfaces ICatalogAware'

  372: q        BINPUT     8

  374: c        GLOBAL     'Products.CMFCore.interfaces ICatalogableDublinCore'

  426: q        BINPUT     9

  428: c        GLOBAL     'zope.location.interfaces IContained'

  465: q        BINPUT     10

  467: c        GLOBAL     'Products.CMFCore.interfaces IContentish'

  508: q        BINPUT     11

  510: c        GLOBAL     'OFS.interfaces ICopySource'

  538: q        BINPUT     12

  540: c        GLOBAL     'webdav.interfaces IDAVResource'

  572: q        BINPUT     13

  574: c        GLOBAL     'plone.dexterity.interfaces IDexterityContent'

  620: q        BINPUT     14

  622: c        GLOBAL     'plone.app.relationfield.interfaces IDexterityHasRelations'

  681: q        BINPUT     15

  683: c        GLOBAL     'plone.dexterity.interfaces IDexterityItem'

  726: q        BINPUT     16

  728: c        GLOBAL     'plone.app.iterate.dexterity.interfaces IDexterityIterateAware'

  791: q        BINPUT     17

  793: c        GLOBAL     'plone.dexterity.interfaces IDexteritySchema'

  838: q        BINPUT     18

  840: c        GLOBAL     'plone.app.contenttypes.interfaces IDocument'

  885: q        BINPUT     19

  887: c        GLOBAL     'Products.CMFCore.interfaces IDublinCore'

  928: q        BINPUT     20

  930: c        GLOBAL     'Products.CMFCore.interfaces IDynamicType'

  972: q        BINPUT     21

  974: c        GLOBAL     'plone.app.dexterity.behaviors.exclfromnav IExcludeFromNavigation'

 1040: q        BINPUT     22

 1042: c        GLOBAL     'z3c.relationfield.interfaces IHasIncomingRelations'

 1094: q        BINPUT     23

 1096: c        GLOBAL     'z3c.relationfield.interfaces IHasOutgoingRelations'

 1148: q        BINPUT     24

 1150: c        GLOBAL     'z3c.relationfield.interfaces IHasRelations'

 1194: q        BINPUT     25

 1196: c        GLOBAL     'plone.namedfile.interfaces IImageScaleTraversable'

 1247: q        BINPUT     26

 1249: c        GLOBAL     'OFS.interfaces IItem'

 1271: q        BINPUT     27

 1273: c        GLOBAL     'plone.app.iterate.interfaces IIterateAware'

 1317: q        BINPUT     28

 1319: c        GLOBAL     'plone.portlets.interfaces ILocalPortletAssignable'

 1370: q        BINPUT     29

 1372: c        GLOBAL     'zope.location.interfaces ILocation'

 1408: q        BINPUT     30

 1410: c        GLOBAL     'OFS.interfaces IManageable'

 1438: q        BINPUT     31

 1440: c        GLOBAL     'Products.CMFCore.interfaces IMinimalDublinCore'

 1488: q        BINPUT     32

 1490: c        GLOBAL     'Products.CMFCore.interfaces IMutableDublinCore'

 1538: q        BINPUT     33

 1540: c        GLOBAL     'Products.CMFCore.interfaces IMutableMinimalDublinCore'

 1595: q        BINPUT     34

 1597: c        GLOBAL     'plone.app.content.interfaces INameFromTitle'

 1642: q        BINPUT     35

 1644: c        GLOBAL     'App.interfaces INavigation'

 1672: q        BINPUT     36

 1674: c        GLOBAL     'Products.CMFCore.interfaces IOpaqueItemManager'

 1722: q        BINPUT     37

 1724: c        GLOBAL     'AccessControl.interfaces IOwned'

 1757: q        BINPUT     38

 1759: c        GLOBAL     'AccessControl.interfaces IPermissionMappingSupport'

 1811: q        BINPUT     39

 1813: c        GLOBAL     'persistent.interfaces IPersistent'

 1848: q        BINPUT     40

 1850: c        GLOBAL     'OFS.interfaces IPropertyManager'

 1883: q        BINPUT     41

 1885: c        GLOBAL     'plone.app.relationfield.behavior IRelatedItems'

 1933: q        BINPUT     42

 1935: c        GLOBAL     'AccessControl.interfaces IRoleManager'

 1974: q        BINPUT     43

 1976: c        GLOBAL     'plone.contentrules.engine.interfaces IRuleAssignable'

 2030: q        BINPUT     44

 2032: c        GLOBAL     'Products.CMFDynamicViewFTI.interfaces ISelectableBrowserDefault'

 2097: q        BINPUT     45

 2099: c        GLOBAL     'OFS.interfaces ISimpleItem'

 2127: q        BINPUT     46

 2129: c        GLOBAL     'plone.app.contenttypes.behaviors.tableofcontents ITableOfContents'

 2196: q        BINPUT     47

 2198: c        GLOBAL     'OFS.interfaces ITraversable'

 2227: q        BINPUT     48

 2229: c        GLOBAL     'plone.uuid.interfaces IUUIDAware'

 2263: q        BINPUT     49

 2265: c        GLOBAL     'Products.CMFEditions.interfaces IVersioned'

 2309: q        BINPUT     50

 2311: c        GLOBAL     'plone.app.versioningbehavior.behaviors IVersioningSupport'

 2370: q        BINPUT     51

 2372: c        GLOBAL     'Products.CMFCore.interfaces IWorkflowAware'

 2416: q        BINPUT     52

 2418: c        GLOBAL     'OFS.interfaces IWriteLock'

 2445: q        BINPUT     53

 2447: c        GLOBAL     'OFS.interfaces IZopeObject'

 2475: q        BINPUT     54

 2477: c        GLOBAL     'zope.interface Interface'

 2503: q        BINPUT     55

 2505: c        GLOBAL     'plone.dexterity.schema.generated Plone_0_Document'

 2556: q        BINPUT     56

 2558: c        GLOBAL     'plone.supermodel.model Schema'

 2589: q        BINPUT     57

 2591: t        TUPLE      (MARK at 2)

 2592: q    BINPUT     58

 2594: \x85 TUPLE1

 2595: q    BINPUT     59

 2597: \x85 TUPLE1

 2598: q    BINPUT     60

 2600: \x85 TUPLE1

 2601: q    BINPUT     61

 2603: .    STOP

highest protocol among opcodes = 2

If you are into this you can read the pickle now :)

If you press c again zodbverify will build the refernce-tree for this object and ispect if for you:

(Pdb++) c

2020-08-24 12:22:42,596 INFO    [zodbverify:234][MainThread] ModuleNotFoundError: No module named 'webdav.interfaces'; 'webdav' is not a package: 0x3b1d06

2020-08-24 12:22:42,597 INFO    [zodbverify:43][MainThread] Building a reference-tree of ZODB...

2020-08-24 12:22:42,964 INFO    [zodbverify:60][MainThread] Objects: 10000

2020-08-24 12:22:44,167 INFO    [zodbverify:60][MainThread] Objects: 20000

2020-08-24 12:22:44,521 INFO    [zodbverify:60][MainThread] Objects: 30000

2020-08-24 12:22:44,891 INFO    [zodbverify:60][MainThread] Objects: 40000

2020-08-24 12:22:45,184 INFO    [zodbverify:60][MainThread] Objects: 50000

2020-08-24 12:22:45,507 INFO    [zodbverify:60][MainThread] Objects: 60000

2020-08-24 12:22:45,876 INFO    [zodbverify:60][MainThread] Objects: 70000

2020-08-24 12:22:46,403 INFO    [zodbverify:60][MainThread] Objects: 80000

2020-08-24 12:22:46,800 INFO    [zodbverify:60][MainThread] Objects: 90000

2020-08-24 12:22:47,107 INFO    [zodbverify:60][MainThread] Objects: 100000

2020-08-24 12:22:47,440 INFO    [zodbverify:60][MainThread] Objects: 110000

2020-08-24 12:22:47,747 INFO    [zodbverify:60][MainThread] Objects: 120000

2020-08-24 12:22:48,052 INFO    [zodbverify:60][MainThread] Objects: 130000

2020-08-24 12:22:48,375 INFO    [zodbverify:60][MainThread] Objects: 140000

2020-08-24 12:22:48,665 INFO    [zodbverify:60][MainThread] Objects: 150000

2020-08-24 12:22:48,923 INFO    [zodbverify:60][MainThread] Objects: 160000

2020-08-24 12:22:49,037 INFO    [zodbverify:61][MainThread] Created a reference-dict for 163955 objects.



2020-08-24 12:22:49,386 INFO    [zodbverify:182][MainThread] Save reference-cache as /Users/pbauer/.cache/zodbverify/zodb_references_0x03d7f331f3692266.json

2020-08-24 12:22:49,424 INFO    [zodbverify:40][MainThread] The oid 0x3b1d06 is referenced by:



0x3b1d06 (BTrees.OIBTree.OITreeSet) is referenced by 0x3b1d01 (BTrees.OOBTree.OOBucket) at level 1

0x3b1d01 (BTrees.OOBTree.OOBucket) is referenced by 0x11c284 (BTrees.OOBTree.OOBTree) at level 2

0x11c284 (BTrees.OOBTree.OOBTree) is _reltoken_name_TO_objtokenset for 0x11c278 (z3c.relationfield.index.RelationCatalog) at level 3

0x11c278 (z3c.relationfield.index.RelationCatalog) is relations for 0x1e (five.localsitemanager.registry.PersistentComponents) at level 4

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x11 (Products.CMFPlone.Portal.PloneSite) at level 5

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x01 (OFS.Application.Application) at level 6

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f6 (persistent.mapping.PersistentMapping) at level 7

8< --------------- >8 Stop at root objects

0x11 (Products.CMFPlone.Portal.PloneSite) is Plone for 0x02f7 (zope.component.persistentregistry.PersistentAdapterRegistry) at level 8

8< --------------- >8 Stop at root objects

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02f5 (plone.app.redirector.storage.RedirectionStorage) at level 6

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fa (zope.ramcache.ram.RAMCache) at level 7

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fd (plone.contentrules.engine.storage.RuleStorage) at level 8

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x338f13 (plone.app.contentrules.rule.Rule) at level 9

0x338f13 (plone.app.contentrules.rule.Rule) is ++rule++rule-2 for 0x0303 (BTrees.OOBTree.OOBTree) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346961 (plone.app.contentrules.rule.Rule) at level 10

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b59 (plone.app.contentrules.rule.Rule) at level 11

0x02fd (plone.contentrules.engine.storage.RuleStorage) is __parent__ for 0x346b61 (plone.app.contentrules.rule.Rule) at level 12

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x02fe (plone.app.viewletmanager.storage.ViewletSettingsStorage) at level 9

0x1e (five.localsitemanager.registry.PersistentComponents) is referenced by 0x034d (plone.keyring.keyring.Keyring) at level 10

0x034d (plone.keyring.keyring.Keyring) is referenced by 0x02fb (persistent.mapping.PersistentMapping) at level 11

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a32 (plone.keyring.keyring.Keyring) at level 12

0x02fb (persistent.mapping.PersistentMapping) is referenced by 0x3b1a33 (plone.keyring.keyring.Keyring) at level 13

0x1e (five.localsitemanager.registry.PersistentComponents) is __parent__ for 0x3b3dc4 (pas.plugins.ldap.plonecontrolpanel.cache.CacheSettingsRecordProvider) at level 11

0x3b1d01 (BTrees.OOBTree.OOBucket) is _next for 0x3b1cf1 (BTrees.OOBTree.OOBucket) at level 3

[...]

From this output you can find out that the broken object is (surprise) a item in the RelationCatalog of zc.relation. See the chapter "Frequent Culprits" for information how to deal with these.

Example 1 of using fsoids.py

In this and the next example I will use the script fsoids.py to find out where a broken objects actually sits so I can remove or fix it. The easier approach is to use zodbverify but I discuss this approach here since it was your best option until I extended zodbverify and since it might help you to understand the way references work in the ZODB.

$ ./bin/zodbverify -f var/filestorage/Data.fs



INFO:zodbverify:Done! Scanned 120797 records.

Found 116 records that could not be loaded.

Exceptions and how often they happened:

AttributeError: Cannot find dynamic object factory for module plone.dexterity.schema.generated: 20

AttributeError: module 'plone.app.event.interfaces' has no attribute 'IEventSettings': 3

ModuleNotFoundError: No module named 'Products.ATContentTypes': 4

ModuleNotFoundError: No module named 'Products.Archetypes': 5

ModuleNotFoundError: No module named 'Products.CMFDefault': 20

ModuleNotFoundError: No module named 'Products.CMFPlone.DiscussionTool': 1

ModuleNotFoundError: No module named 'Products.CMFPlone.MetadataTool': 1

ModuleNotFoundError: No module named 'Products.PloneLanguageTool': 1

ModuleNotFoundError: No module named 'Products.ResourceRegistries': 1

ModuleNotFoundError: No module named 'fourdigits': 8

ModuleNotFoundError: No module named 'plone.app.controlpanel': 2

ModuleNotFoundError: No module named 'plone.app.stagingbehavior.interfaces'; 'plone.app.stagingbehavior' is not a package: 34

ModuleNotFoundError: No module named 'webdav.EtagSupport'; 'webdav' is not a package: 16

Follow the white rabbit...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907d



oid 0x35907d BTrees.OIBTree.OISet 1 revision

    tid 0x03c425bfb4d8dcaa offset=282340 2017-12-15 10:07:42.386043

        tid user=b'Plone xxx@xxx.de'

        tid description=b'/Plone/it-service/hilfestellungen-anleitungen-faq/outlook/content-checkout'

        new revision BTrees.OIBTree.OISet at 282469

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        referenced by 0x35907b BTrees.OIBTree.OITreeSet at 911946038



[...]

Follow referenced by ...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x35907b



[...]

referenced by 0x3c5790 BTrees.OOBTree.OOBucket
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3c5790



[...]

referenced by 0x11c284 BTrees.OOBTree.OOBTree

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x11c284



[...]

referenced by 0x3d0bd6 BTrees.OOBTree.OOBucket

[...]
./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x3d0bd6



[...]

referenced by 0x11c278 z3c.relationfield.index.RelationCatalog

[...]

Found it!!!!!

Example 2 of using fsoids.py

In this example zodbverify found a trace of Products.PloneFormGen even though you think you safely uninstalled the addon (e.g. using https://github.com/collective/collective.migrationhelpers/blob/master/src/collective/migrationhelpers/addons.py#L11)

Then find out where exists in the tree by following the trail of items that reference it:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x372d00

oid 0x372d00 Products.PloneFormGen.content.thanksPage.FormThanksPage 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision Products.PloneFormGen.content.thanksPage.FormThanksPage at 912841984

        referenced by 0x372f26 BTrees.OIBTree.OIBucket at 912930339

        references 0x372e59 Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5a Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5b OFS.Folder.Folder at 912841984

        references 0x372e5c Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5d Products.Archetypes.BaseUnit.BaseUnit at 912841984

        references 0x372e5e Persistence.mapping.PersistentMapping at 912841984

    tid 0x03d40a3e52a41633 offset=921078960 2019-11-25 17:02:19.368976

        tid user=b'Plone pbauer'

        tid description=b'/Plone/rename_file_ids'

        referenced by 0x2c1b51 BTrees.IOBTree.IOBucket at 921653012

Follow referenced by until you find something...

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x2c1b51

oid 0x2c1b51 BTrees.IOBTree.IOBucket 1 revision

    [...]

Here I skip the trail of referenced by until I find 0x280184 five.intid.intid.IntIds:

./bin/zopepy ./parts/packages/ZODB/scripts/fsoids.py var/filestorage/Data.fs 0x280184

oid 0x280184 five.intid.intid.IntIds 1 revision

    tid 0x03d3e83a045dd700 offset=421126 2019-11-19 15:54:01.023413

        tid user=b''

        tid description=b''

        new revision five.intid.intid.IntIds at 8579054

        references 0x28018c <unknown> at 8579054

        references 0x28018d <unknown> at 8579054

    tid 0x03d3e90c4d3aed55 offset=915868610 2019-11-19 19:24:18.100824

        tid user=b' adminstarzel'

        tid description=b'/Plone/portal_quickinstaller/installProducts'

        referenced by 0x02f6 persistent.mapping.PersistentMapping at 915868690

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x02f7 zope.component.persistentregistry.PersistentAdapterRegistry at 915879394

        referenced by 0x1e five.localsitemanager.registry.PersistentComponents at 915898834

That is the IntId-Catalog from zope.intid. The problem seems to be that similar to the zc.relation catalog rerefences to broken objects stay in the catalog and need to be removed manually.

Here is a example of how to remove all broken objects from the catalog in a pdb-session:

(Pdb++) from zope.intid.interfaces import IIntIds

(Pdb++) from zope.component import getUtility

(Pdb++) intid = getUtility(IIntIds)

(Pdb++) broken_keys = [i for i in intid.ids if 'broken' in repr(i.object)]

(Pdb++) for broken_key in broken_keys: intid.unregister(broken_key)

(Pdb++)

(Pdb++) import transaction

(Pdb++) transaction.commit()

After packing the DB the problem is gone. o/

Other Options

Use zodbbrowser to inspect the ZODB.
It is Zope3 app to navigate a ZODB in a browser. At least I had problems getting it to run with a Plone-ZODB.
Use zc.zodbdgc
This tool can validate distributed databases by starting at their root and traversing to make sure all referenced objects are reachable. Optionally, a database of reference information can be generated.
Use collective.zodbdebug
A great tool to build and inspect reference-maps and backreference-maps of a ZODB. So for it does not work with Python 3 yet. Some if its features are also part of zodbverify.

Frequent Culprits

IntIds and Relations

The IntId-Tool and the relation-catalog are by far the most requent issues, especially if you migrated from Archetypes to Dexterity.

There may be a lot of RelationValues in these Tools that still reference objects that cannot be loadedif these removed objects were not properly removed.

The following code from collective.relationhelpers cleans up the IntId- and Relation-catalog but keeps relations intact. For large sites it may take a while to run because it also needs to recreate linkintegrity-relations.

from collective.relationhelpers.api import cleanup_intids

from collective.relationhelpers.api import purge_relations

from collective.relationhelpers.api import restore_relations

from collective.relationhelpers.api import store_relations



def remove_relations(context=None):

    # store all relations in a annotation on the portal

    store_relations()

    # empty the relation-catalog

    purge_relations()

    # remove all relationvalues and refs to broken objects from intid

    cleanup_intids()

    # recreate all relations from a annotation on the portal

    restore_relations()

For details see https://github.com/collective/collective.relationhelpers/blob/master/src/collective/relationhelpers/api.py

Annotations

Many addons and features in Plone store data in Annotations on the portal or on content.

It's a good idea to check IAnnotations(portal).keys() after a migration for Annotation that you can safely remove.

Here is a example where wicked (the now-removed wiki-style-editing feature of Plone) stored it's settings in a Annotation:

def cleanup_wicked_annotation(context=None):

    ann = IAnnotations(portal)

    if 'plone.app.controlpanel.wicked' in ann:

        del ann['plone.app.controlpanel.wicked']

Another example is files from failed uploads stored by plone.formwidget.namedfile in a annotation:

def cleanup_upload_annotation(context=None):

    # remove traces of aborted uploads

    ann = IAnnotations(portal)

    if ann.get('file_upload_map', None) is not None:

        for uuid in ann['file_upload_map']:

            del ann['file_upload_map'][uuid]

Appendix

Migrating a ZODB from py2 to py3

Since people often encounter issues with their ZODB after migrating here is a quick dive into migrating a ZODB from Python 2 to Python 3.

The migration is basically calling the script zodbupdate in py3 with the parameter --convert-py3.

$ ./bin/zodbupdate --convert-py3

You need to pass it the location of the database, the defaul-encoding (utf8) and a fallback-encoding (latin1) for items where decoding to utf8 fails.

Example:

$ ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8 --encoding-fallback latin1



Updating magic marker for var/filestorage/Data.fs

Ignoring index for /Users/pbauer/workspace/projectx/var/filestorage/Data.fs

Loaded 2 decode rules from AccessControl:decodes

Loaded 12 decode rules from OFS:decodes

Loaded 2 decode rules from Products.PythonScripts:decodes

Loaded 1 decode rules from Products.ZopeVersionControl:decodes

Committing changes (#1).

After that you should be able to use your ZODB in Python 3.

The process in a nutshell:

  1. First, run bin/zodbupdate -f var/filestorage/Data.fs So no python3 convert stuff yet! This will detect and apply several explicit and implicit rename rules.
  2. Then run bin/instance zodbverify. If this still gives warnings or exceptions, you may need to define more rules and apply them with zodbupdate. But you can still choose to migrate to py3 if this shows errors.
  3. Using Python 3 run bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8
  4. For good measure, on Python 3 run bin/instance zodbverify.

Read the docs: https://docs.plone.org/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.html

See also: https://community.plone.org/t/zodbverify-porting-plone-with-zopedb-to-python3/8806/17

Dealing with oids

Transforming oids from int to hex and text and vice versa:

>>> from ZODB.utils import p64

>>> oid = 0x2c0ab6

>>> p64(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'



>>> from ZODB.utils import oid_repr

>>> oid = b'\x00\x00\x00\x00\x00,\n\xb6'

>>> oid_repr(oid)

'0x2c0ab6'



>>> from ZODB.utils import repr_to_oid

>>> oid = '0x2c0ab6'

>>> repr_to_oid(oid)

b'\x00\x00\x00\x00\x00,\n\xb6'

Get a path for blobs:

from ZODB.blob import BushyLayout

if isinstance(oid, int):

    # e.g. oid = 0x2c0ab6

    from ZODB.utils import p64

    oid = p64(oid)

return BushyLayout.oid_to_path(None, oid)

Load a obj by oid from ZODB in a pdb-prompt:

oid = 0x2c0ab6

from ZODB.utils import p64

app = self.context.__parent__

obj = app._p_jar.get(p64(oid))

TODO

Finish, merge and document https://github.com/plone/zodbverify/pull/8

Links

Quite a lot of people from the Plone/Zope communities have wriotten about this issue. I learned a lot from these posts:

Plone Foundation Launches Governance Initiative

Posted by PLONE.ORG on August 20, 2020 06:07 PM

Last month the Plone Foundation Board of Directors announced a new initiative to review and refresh the Plone community ‘s governance. Two concrete actions are being taken, with more to follow based on the resulting discussions.

  1. Every two months beginning in September, a series of Steering Circle meetings will be held to discuss the issues and plot a way forward. Representatives from all Plone teams are being invited to the Steering Circle meetings. People who are not affiliated with a team may request an invitation from the Board.
  2. Beginning at the November Plone Conference, the Board will set up a series of open discussions about the organizational structure of all our projects (Plone, Zope, Guillotina, Volto, REST API, etc.) All community members will be welcome to attend and virtual participation will be possible.

This initiative is described in more detail in the Board‘s statement on the Plone governance process. Thanks to the Board for proposing this and we all look forward to a series of fruitful discussions!

Great Progress on Volto at the 2020 Beethoven Sprint

Posted by PLONE.ORG on July 24, 2020 01:40 PM

Everybody in the Plone community loves sprints. Not only because of the software we enhance or the code we write. The most crucial ingredient of a great sprint is meeting the people you like and that you haven’t seen in a while. This is harder to do in these coronavirus times, but the Beethoven sprint, held remotely from May 27th through the 30th, succeeded.

There were 34 participants, including substantial representation from kitconcept, RedTurtle, and Eau de Web. They used Discord voice channels throughout the day for a real sprint experience. The channels functioned like tables in a sprint room, where sprinters can go from one table to another to hear what people are discussing. The organizers also used Zoom calls for daily standups and Remo.co video calls in the evenings, which are structured like a bar. Sprinters brought their favorite beverage, good food, relaxed on a couch, and had good conversations in the evenings!

In addition to socializing, lots of work got done. The sprint was all about Volto, which is the new React front end for Plone 6. Highlights included:

  • Bug fixes, date widget improvements, a recurrence widget, a sort function, and work on widget reusability
  • Discussions and progress on a new Volto add-ons architecture based on slots
  • Improvements to Volto's multilingual story, including a test fixture and a translation for Romanian
  • Improvements to the Cypress accessibility test coverage
  • Discussions about Volto's theming story, including how to separate the CMS and public-facing themes
  • A form builder prototype, as well as lots of form builder discussions
  • A Volto listing block that includes sorting, view templates, and a table view
  • Progress on a Dexterity content types control panel for Volto, including a schema editor and RestAPI endpoint
  • Additional Volto training materials, as well as training test drives
  • Volto marketing and onboarding work, including a Volto website and screencasts
  • Discussions and improvements to the Plone backend, including a plone.restapi release
    and work on the thorny problem of a Dexterity site root object
  • Plone 6 roadmap discussions

Thank you to everyone who attended - Plone 6 is getting closer to reality!

Product Lifecycle Management with Plone

Posted by PLONE.ORG on July 14, 2020 07:02 PM

Plone at PNZ Produkte GmbH

PNZ Produkte GmbH is a niche manufactory of oil-based paints and coatings made from renewable resources. Next to its company brand product line, a substantial amount of business is generated from private label (PL) sales. These PL products are offered as a turn-key service: including promotional material, product information sheets and packaging. For some private label customers, PNZ also takes care of fulfillment to end consumers.

A typical finished product is derived from:

  • A Base Recipe which determines the ingredients, production methods, usage safety indications and applicable mandated regulations (e.g. GHS)  
  • A Master Product which defines the application specifications, usage instructions, packaging sizes, pigments and other general use attributes
  • A Finished Product which includes branding characteristics, marketing descriptions, article EANs and other customizations

Since 2015, Plone has been used as the engine for creating and maintaining product specifications. About 20 employees in multiple departments interact regularly with the software. The project received a “Digitalbonus” innovation award from Germany's Landesamt Oberbayern.

PNZ Product Settings Page

A Typical Product Lifecycle

When a new product is formulated, has passed its application and safety testing phase, and is ready to go in production, its specifications will be split in three separate information layers. Custom content types in Plone connect these layers and manage their contents. Plone's ability to quickly prototype content types through-the-web makes it easy to design an initial structure and refine it based on user experiences. Each information layer consists of 30 to 50 individual fields and can be extended as needs change over time. Connecting different types of data from dynamic and external sources is relatively simple as Plone offers a large variety of edit/display widgets out of the box. 

At each hierarchical level, new content undergoes a quality and regulations compliance control check before going through the Plone publishing workflow. This makes the document available for use in lower layers. Any base recipe can potentially power several master products, which in turn results in a large number of finished products. The current system powers some 25,000 individual articles or SKU's. 

At the lowest level, a "Finished Product" content type includes product-specific overrides and any information not inherited from the recipe and master product layers.

PNZ Product Information Flow

Since products may be offered to multiple geographical markets, all information must be translated. Plone's built-in multilingual capabilities make it possible to have each individual document available in several languages. Fifteen languages are currently in use within the PNZ system. Each field in a content type can be designated as translatable or multilingual, so that updates of multilingual fields can automatically propagate throughout the various translations.

Automating Content Creation

The information contained in its product lifecycle management system allows PNZ to automate the generation of a variety of ready to use content. Plone provides the BrowserView mechanism, used in conjunction with Zope Page Templates (ZPT), to determine what should be included in the content and how it should be rendered. PNZ Wrappers with inDesign and XML from Plone

PNZ creates colortone samplers and product information sheets, with a customized layout for each brand, using the tools provided by Plone. The documents then undergo a final conversion to PDF using “Produce & Publish” software, created by longtime Zope/Plone community contributor Andreas Jung at Zopyx.

In the packaging process, Plone powers the creative design of all product labels. A BrowserView is used to draw data from multiple content types and the company’s ERP system. This will generate a set of XML files for each product and its SKU variants, plus related dynamic images such as EAN codes, colortones, certifications and GHS warning signs.

PNZ Public Site With PloneThe XML files and images are then read into Adobe InDesign templates by the company's creative designers, resulting in a professionally designed wrapper for each SKU that always contains accurate information and complies with industry standards and specifications.

Other types of content include the generation of raw HTML and images for use in a PlentyMarkets shop, and the creation of Product Information Management (PIM) CSV export files for large customers such as DIY store chains. Finally, using plone.restapi, the system powers the product catalog of PNZ's soon to be relaunched public facing website which of course also uses Plone!

A Very Successful 2020 Python Web Conference

Posted by PLONE.ORG on June 26, 2020 03:09 PM

June 17-19, Six Feet Up hosted the 2nd Annual Python Web Conference using their newly developed all-in-one virtual event platform, LoudSwarm. Focusing on Python for the Web, the conference featured talks and trainings from several members of the Plone community.

The conference had a total of 266 attendees participating from 37 countries and 14 timezones. A unique aspect of the registration was that for each ticket purchased, a ticket was donated to a developer residing in a developing country. Those that received a free ticket would not have been able to otherwise attend, and they were able to gain valuable knowledge on the state of Plone and Python on the web.

Being a completely virtual event, several accommodations were set up to make the event as interactive as possible. Attendees could join the Gallery after each talk to ask the presenter questions and have discussions about the talk topic. Slack was also heavily used to foster discussions and interaction among the attendees. Evening Socials were created to network, play games, and keep the fun going after the day’s talks were done.

Having a Plone presence was great for the attendees who had never heard of Plone, and provided many with the opportunity to see Plone again in a new way. Many great discussions were had around new Plone features, customization tricks, and fond memories of past Plone Conferences.

Recordings of the talks are currently only available to registered attendees. Post-video access tickets are still available if you weren’t able to attend the live event. All talk and tutorial videos will be made publicly available on YouTube in September.

Plans for Python Web Conf 2021 are already underway. Visit https://2020.pythonwebconf.com to learn more about this year’s event.

Zope and the Plone Foundation

Posted by PLONE.ORG on June 25, 2020 02:59 PM

Plone was very early to open source governance - the Plone Foundation was one of the first 501(c)(3)s to be established to protect open-source software. In the years that have followed it's become more common for software to be protected under an umbrella organization, but that wasn't possible in the early days.

Zope started off as the property of Digital Creations, which became Zope Corporation. But in the early 2000's there was a desire to move the protection of Zope to a community effort following the same model, so a new non-profit was formed.

This worked well for ten years, but by the mid 2010's there were significantly fewer active developers of Zope and most were not interested in the administrative tasks required to maintain a non-profit corporation. Over the years, the Zope Foundation fell behind in its regulatory compliance until it became inactive. It still owned the copyright to Zope, as well as some funds, but it had very limited rights over them.

The Plone Foundation's mission to protect Plone implies a responsibility to protect Zope, as Plone depends on Zope for much of its core infrastructure. Informal discussions began around this time about Plone taking over the role of the Zope Foundation, but no substantial progress was made.

As the years passed, the need to solve the regulatory problem became more pronounced. It was clear that the Zope Foundation would not restart operations in any meaningful sense, but Zope development was undergoing a resurgence thanks to the need to port to Python 3.

At the 2018 Plone conference in Tokyo, after quite a few whiskies, people resumed talking about this problem, and we decided to try to solve it again. Because one of the biggest blockers was a lack of enthusiasm in the Zope community with regards to dealing with the bureaucracy, some people who were members of both Foundations reached out to the IRS, the Delaware Department of State and some experts in US Corporation law to plan a way forward.

A few months later in early 2019, the Zope Foundation held a special general meeting to pass a motion on the future of the Foundation. The membership mandated the board of directors to transfer all assets to the Plone Foundation as a donation, and to formally close the Zope Foundation. Later that same day the board of directors voted to donate all property of the Zope Foundation, including copyright in the Zope software, to Plone.

Since then, Plone community members have been working on the bureaucratic aspects of this move. We've harmonized contributor agreements to improve international enforceability, and work is underway to ensure that Zope contributors are properly represented in Plone's democratic structure before the next internal election in November. Zope contributors can already apply to be members of the Plone Foundation, and many are, but we're looking to streamline the process for former Zope Foundation members.

The Zope and Plone Security Teams have already been integrated, and now we need Zope representatives on the Membership Team. Any Zope people who are interested in helping out should please contact Érico Andrei.

Plone’s Migration to Python 3

Posted by PLONE.ORG on June 24, 2020 01:32 AM

July 19, 2019, marked the day Plone officially entered the world of Python 3 when version 5.2 was released.  Support for Python 3.6, 3.7, and 3.8 is now included with Plone 5.2, opening up the future for Plone’s continuing 19-year reputation for unrivaled security and stability.

Plone is the venerable open-source Python web content management system that was created in 2001. It has continued to be maintained and enhanced by hundreds of contributors around the world and is protected by the nonprofit Plone Foundation.

Plone 5.2 is the culmination of over 3.5 years of development that was driven by the looming deadline of January 1, 2020: the impending end of support for Python 2. Work on Plone 5.2 ran in parallel with continuing feature-centric work on versions 5.0 and 5.1.  The move to Python 3 was to be a major undertaking, given its huge codebase, its large ecosystem of add-ons, and its dependency on the Zope web application framework (https://www.zope.org), which itself would have to be upgraded to Python 3.  Many sprints were held over these past few years in which Plone and Zope developers collaborated in this joint effort.  One of the earliest fruits of this collaboration was to see all 9000 tests passing as a result of work done at the Alpine City Sprint in February 2017, but more work was still needed.  At the core of the security model for which Zope and Plone are renowned is a feature known as Restricted Python.  Although core developers had feared that Restricted Python could not be ported to Python 3, this was accomplished in Innsbruck, Austria: Restricted Python now runs in Python 3.6 and up.

ZODB, the ACID compliant and extremely robust database, was upgraded to the latest version for Plone 5.2, along with a Zope instance running via a WSGI server instead of ZServer.

Zope developers came together and released the Python 3 compatible Zope 4 in May 2019. Plone developers were able to release its Python 3 compatible version 5.2 soon after, complete with its traditional in-place upgrade, with an additional database upgrade step required to convert to Python 3 strings.  Plone 5.2 is the first release to fully support Python 3 while maintaining complete compatibility with Python 2.7, which is expected to be dropped with the upcoming release of Plone 6.

Once available as a separate add-on, a REST API is now also included in the core of Plone 5.2, making it even easier to use Plone with modern JavaScript front ends such as Angular and React and to integrate Plone with other applications.

This release provides the Plone community the certainty of having a stable backend without an expiration date and the excitement of developing a new frontend.  Plone is more than 19 years old, and Zope is more than 22 years old. Yet both projects have reinvented themselves several times and are maintained by a worldwide, dynamic community. These two platforms provide a very secure, flexible, and mature solution in the often uncertain, unstable landscape of web applications.

Automated subtitles – destructuring a successful Plone CMS integration

Posted by Asko Soukka on June 22, 2020 12:00 PM

Never underestimate the importance of being able to make changes to your software – especially when they are critical to your business processes.

Our university has its own audio and video publishing platform, Moniviestin. The first version was released in 2003 – well before Youtube. The latest major iteration was done in 2010, and is built on top of Plone CMS platform, with microservice architecture based video encoding pipeline. After 10 years and counting, we had yet another critical feature request: most of the new published recordings must have subtitles, as automatically as possible.

Once the team had benchmarked the available automatic speech recognition (ASR) services for Finnish speech, they selected Sanelius ASR HTTP API from Ääni Company.

Now they had the specification, but how did relying on Plone help with the integration?

From “Publish” to “Publish with subtitles”

To make the end user experience as convenient as possible, the team decided to connect the automated subtitle generation into the current video publication workflow.

In Plone, every content object may be supported with one or more state managing workflows. For example, our video content pages, called “Media pages”, have one workflow for managing the publication process and another one for managing the video encoding process. Because Plone is designed to work as a web publishing platform out-of-the-box, only the publication workflow is exposed for end-users by default.

moniviestin workflow menu

The obvious starting point for the integration was to branch the publication workflow with a new path: “Publish with subtitles”. That was enough to provide the required user interface for the feature, just next to the familiar “Publish” action. The new branch in the worklow made also possible the other required states and transitions to support the actual integration.

moniviestin workflow

What can be seen, can be automated

Our platform had support for manually configurable subtitles already. Plone is based on hierarchical object database, not unlike filesystem with folders and files. Therefore, our platform was built to represent its content with folder-like “Media page” containers, which could contain any amount of related attachment items, like slides, lecture notes, and… subtitles!

moniviestin example

So, “Subtitles” was already a feature on our CMS based platform, and configurable within “Media page” manually through the user interface. But not only was “Subtitles” available to the end-users, it was also available through Plone REST API. And Plone REST API provided out-of-the-box most of the necessary actions to fetch “Media pages” waiting for subtitles, post new subtitles, update existing subtitles, and confirm the updates according to the workflow.

Having workflow changes, addable “Subtitles”-content and scriptable REST API in place, one more Plone automation feature still needs to be mentioned: configurable event based actions, also known as “content rules”:

moniviestin content rule

For automated subtitles, Plone content rules made it simple to trigger automation service when “Publish with subtitles” was selected. Similarly it made simple to configure email notifications when automated subtitles were received.

Finally, let robots do the hard work

RPA, or Robotic Process Automation, is usually associated with expensive automation platforms with visual programming features. Yet, deep inside, this fancy term could simply mean any kind of script automation, being triggered by external events or timer, to perform some business value providing action. Actually, a very useful approach for casual automation…

For automated subtitles, our team did not need to build tight integration between our video publishing platform and the selected ASR service. Neither did the team need to build any new continuosly running services to handle the integration. Our team simply needed to ensure that both ends had consumable HTTP APIs, and then write required scripts (“robots”) to handle the required communication between services.

The most obvious place for the required integration scripts was our existing Jenkins based RPA platform: Jenkins provides us job configuration, secrets management, webhooks endpoints, scheduled executions and archival of execution logs. Simply everything our team needed to manage the execution of these subtitle automation tasks.

moniviestin rpa poll

In addition, our RPA Jenkins workers were already powered by Nix package manager to provide all run-time dependencies for the automation scripts. I was told that just using Nix saved up to day in development time, because it provided complete and up-to-date ffmpeg installation for the integration scripts without any additional effort.

Let there be subtitles!

Welcome our full subtitle automation story:

  1. Video author selects “Publish with subtitles”.
  2. Plone content rule calls Jenkins webhook to schedule a new robot.
  3. Robot reads pending task from Plone, streams the video, extracts and converts the audio track, and submits the track to ASR service.
  4. Jenkins schedules a new job to poll pending ASR jobs.
  5. Robot reads completed task from ASR service, downloads the text, converts it into subtitles format and uploads the file to Plone.
  6. Plone content rule notifies the video author with email that automatically generated subtitles are now available.
  7. ???

Profit.

Static is fast, but CMS still required – a JAMstack story

Posted by Asko Soukka on June 12, 2020 12:00 PM

An another iteration of our university’s new study guide web site has been completed. The project that started more than a year ago as a JAMstack experiment with GatsbyJS and Hasura, has finally matured enough to get its long expected expansion: a content management system!

study guide tabs

In the beginning there was data. A lot of it. A great amount of granular JSON chunks, to be turned into fast and well connected study guide web site. Regularly updated, of course. For years we had solved similar use cases by building and synchronizing CMS content out of more or less structured data. This time we had not enough resources for a such “sophisticated” CMS integration, but we had to look for more agile alternatives – think out of the box. We chose to go JAMstack, with GatsbyJS.

Hasura – the magical GraphQL gateway

At first, of course, we had to make sense of our data. GatsbyJS requires to use GraphQL queries to select the published content. By chance, we found Hasura, which is designed to turn any PostgreSQL database into well-connected GraphQL API. So, we built a pipeline to dump our JSON data into JSONB columns within simple PostgreSQL tables. Once the data was in the database, and we learned which parts we really needed, we could build dynamic database views to expose the data exactly as we wanted.

hasura view

As designed, Hasura was able to publish those views as GraphQL types and connect them with the relations we needed. Suddenly we had complete GraphQL API for our data. Almost, but not quite entirely unlike, magic. Awesome. Really.

hasura relationships

But publishing all that structured data with fast and accessible user experience was not enough. More information was required to be included on the site. This time with images, video embeds and attachments. And the master system was not designed to handle that.

After all, a real CMS was required.

Volto – a breeze of fresh air

Lucky us, we had just right CMS product and experience available. Plone CMS with its latest user interface, Volto, provided us both hierarchical object database and modern user interface required for managing the additional content. Plone shines in managing content in folder tree like hierarchies, and sharing access rights for them accordingly. Volto, on the other hand, makes Plone snappier and easier to use than ever.

volto contents

And when it comes to Plone as a data source for GatsbyJS: I personally mentored Google Summer of Code students both in 2018 and 2019, and then continued the work, to make sure that Plone integrates perfectly with any GatsbyJS project.

The last piece in our puzzle was to connect Volto authored Plone CMS content with our structured data from the Hasura powered GraphQL API. The flexibility of Plone CMS, with fresh customization possibilities provided by Volto, enabled our solution:

  1. Plone ships with out-of-the-box customizable structured content types. Without any custom code, we were able to enhance our Volto-editable pages with metadata fields to store the connecting information. This also made the data available in Plone REST API for GatsbyJS data source integration.

plone dexterity editor ,

  1. Thanks to Volto user interface being customizable with our own ReactJS code, it was possible to customize the select widget of our primary connecting field, to search our Hasura GraphQL API for all the possible value options, to be saved with the content page.

volto sisu connector

All this required successful teamwork, not only with a few developers doing the technical implemenation, but also with the dozens of our content editors creating and connecting the actual content. That said, we successfully reached our goal:

What you see, edit and connect in Volto…

study guide edit

…is what you get in our GatsbyJS built study guide web site, knitted together with the original study guide data.

study guide image

Something old, something new and something blue.

The perfect match.

Plone 6 Remote Sprint - April 2020

Posted by PLONE.ORG on May 01, 2020 07:21 PM

A Successful Remote Sprint

Last week we held the first Plone remote sprint in the time of the coronavirus pandemic. It was a positive experience and a lot got done! Sprint plans and results were captured on the plone-6-sprint Etherpad and some chatting occurred on the plone/sprint Gitter channel. But the main method of communication was by video. Participants stayed connected by video all day, either on Jitsi or Discord. Each team had their own "room", and it was nice to be able to see everyone while working, even when microphones were muted. This mimicked the sort of frictionless in-person conversations that are the hallmark of sprints.

Sprint Topics

The sprinters tackled six different topics, here are the final reports on each. For more detailed information, see the plone-6-sprint Etherpad.

Modernize Plone's Default Theme

Peter Holzer and Stefan Antonelli led the charge with help from Robert Kuzma, plus Peter Mathis taking the lead on z3c.form. The team made a long list of contributions toward the following PLIPs:

Their work changes the default theme plonetheme.barceloneta to be based on Bootstrap 4 and SCSS. Plone specific styles have only been added where Bootstrap is lacking definitions for Plone elements. There is an overrides directory where all template overrides will be collected until they are merged back into the original packages.

Markup in general was improved. main_template markup was updated and the structure cleaned up, empty and unneeded elements have been removed. Header, navigation, portlets, status messages, batching - all markup and nav styles were updated. The generic view for content types was updated (this includes title, description, byline, related items and keywords). control_panel_overview, control_panel default, and some control panels with custom views were all updated.

For forms, Bootstrap markup was added to plone.app.z3cform to give widgets and fields (buttons, input fields, etc.) a basic styling. Plone's classic backend forms are also fully functional as long as Bootstrap's default CSS is independently available. In fact a theme (e.g. Barceloneta) is not necessary as long as Bootstrap CSS is available.

Icons also got some love. icon_expr for actions, content types, etc. was reused to define SVG icons. bootstrap-icons were added as an available icon resource to plone.staticresources. The team selected and added icons to the control panel settings in the controlpanel.xml in plonetheme.barceloneta.

Mosaic was changed to use Flex for columns based on the updated Barceloneta theme. All LESS files were replaced with SCSS and 'npm run watch' is now used to compile the SCSS files. Redundant CSS classes were cleaned up and the default column limit was set to 6 columns. A reset button was added to Mosaic cells to reset to the automatic column width, and an automatic reset was added for column widths if a new column gets inserted. A label was added to indicate the current column width, with "0" for auto width.

A demo of classic Plone 5.2 with the latest Barceloneta default theme is at demo.plone.org. The Coredev buildout including the current state of work on the PLIP for Bootstrap 4 markup is at classic.plone.de.

The team plans to meet regularly to continue working every Wednesday at 5 PM CEST starting on May 13. They will meet in the plone-6-sprint-barceloneta Jitsi room. Everyone is welcome to join!

Simplify Resource Registry

Maik Derstappen led this project, in coordination with the Barceloneta work. They added a custom.css view, which renders the custom CSS added in the Theming Control Panel, after all other CSS resources are loaded. This allows you to override CSS coming from a Diazo theme.

The team also discussed what really needs to change in the Resource Registry. They decided to remove the build-button from the theming editor because it will not work with SCSS (which is now used by Barceloneta).

Developers should register resources already compiled as Resource Bundles with TTW compiling disabled. They can use the merge flag to merge it with default or logged-in bundles, but this only makes sense if they are always needed. Otherwise the resource can be disabled and loaded on demand via Python in the view/viewlet or by adding the header lines directly into the templates.

Variables can be defined as CSS variables. Variables from the registry are no longer used in SCSS/CSS in plonetheme.barceloneta.

Push Plone to Zope 5

Jens Klein reports that lots of "boring but important work" was done by Maurits van Rees including cleanup of package metadata and releases of packages. Jens Vagelpohl released Zope 5.0a2 and Jens Klein created a zope-5 branch for Plone 6.0 on buildout.coredev. Version dependencies were updated on this branch, and version pins that are orphaned or handled upstream by Zope were removed. Version dependencies still outstanding:

  • z3c.form - this will be handled later
  • Update Pillow from 6 latest to 7.x - check is needed on separate branch, this applies to the Plone 5.2 branch as well
  • Parts of the Robotframework test environment - due to non backward compatible changes in Selenium, and possibly other issues
  • Unicode Encode/Decode  - this one has been open for a long time and Jens would like to see some fellow Plonista take the challenge to update or get rid of it!
  • Cleanup of any orphaned dependencies - for some packages it is difficult to know were they are used, especially for dependencies of our ecosystem packages or tools

Many failing tests were fixed. Four tests (partly related to zope.interface 5.1) are still open and in-progress. Jens would appreciate help to get them fixed and merged.

Volto

Timo Stollenwerk led the Volto efforts with Victor Fernandez de Alba, Rob Gietema, Steffen Ring, Katja Süss, Alok Kumar and Rodrigo Ferreira de Souza. They did a complete overhaul of the folder contents view according to Albert Casado's Pastanaga UI design, and made numerous other UI improvements based on a full UX review that Albert did. Alok (frontend) and Rodrigo (backend) implemented the Add-ons Control Panel and Alok implemented the Users and Groups Control Panel. Alok also implemented print CSS for the Volto front end. Victor fixed the listing block to remove the current item from query results for ZCatalog 5.1. Finally, Steffan and Katja worked on content for the upcoming VoltoCMS.com website.

Image Scaling

Asko Soukka worked on image scaling in coordination with the Volto team, with additional help from Jens Klein and Alexander Loechel. They fixed several bugs:

They implemented an option (disabled by default) to generate all configured scales immediately once an image has been updated. Enabling this would slow down the save of an image field, but would prevent most writes on read for image scaling. The code is in plone.formwidget.namedfile.

They implemented a proof of concept for PLIP #3090 for asynchronous multiprocessing of image scaling, which compliments the above option to generate all configured scales on save. It could provide image scaling with good performance for Plone without any additional services.

They implemented a proof of concept for the use of picture & srcset in Volto for perfectly responsive images. The code is in Asko's Github account.

Mastering Plone 6

Philip Bauer led a team consisting of Steve Piercy, Katja Süss and Janina Hard to create and update the developer training class for Plone 6. Much was accomplished. The first alpha version of chapters 1 through 36 are now usable and will be deployed soon for testing.

Many of the chapters now exist in two flavors: one for Volto and one for Classic, with links between them. Logos at the top clearly identify which flavor you are reading. The default navigation follows the Volto story.

The team plans to do bi-weekly half day sprints and to finish the training at the Beethoven Sprint which is scheduled for May 26th. The outstanding chapters are: 

  • Customizing Volto
  • Membrane
  • Speaker content type and view component
  • Volto Frontpage

Plone Marketing

Sally Kleinfeldt and Érico Andrei investigated tools and planned a series of "how to" screencasts for the Plone Youtube channel about managing content in Plone. Érico also worked on adding the videos from the 2003 to 2009 Plone conferences to the channel.

Sally discussed how to position headless Plone and Volto with Timo Stollenwerk, Victor Fernandez de Alba and Katja Süss. As noted above, the Volto team is working on a new voltocms.com website which will primarily be aimed at React developer on-boarding. We discussed how plone.org/plone.com can coordinate with and compliment their efforts.

After discussion with additional interested parties (Rikupekka Oksanen, Stefania Trabucchi, Fulvio Casali and William Fennie), we decided to create a "What is Plone?" section on plone.org which is aimed at decision makers. It will give a bit of Plone history and describe the current variety of frontend and backend options and when to use them. We began brainstorming ideas and will meet again in 3 weeks. Input from other community members would be welcome.

Refreshing CMS, in a theme, with Plone

Posted by Asko Soukka on April 13, 2020 12:00 PM

“How hard can it be? It is just a theme…”

Of course, it was. Unless it was a collection of configurable interactive components. With features like tabbed carousels, photo filters, hyphenation, and syndication of news or calendar feeds from various sources. All responsive. All accessible. All reusable around the site. All with multilingual user interface elements, when required.

layout

Some might confuse that for requirement specification of a new CMS/WCM project. For us it was just a theme refresh for the current installation. And, to be honest, thanks to Plone, the hardest part really was the CSS.

Real-time layouts with Plone Mosaic

Being able to see the content in its themed context while editing it, has always been the definitive part of Plone editor experience. WYSIWYG to the max, they say. There are still options to keep that Plone promise alive in the era of “modern web tech”. Our choice has been Plone Mosaic site layouts.

wysiwyg accordion

Plone Mosaic site layouts turn the principles of traditional CMS theming upside down (continuing the tradition of Plone Diazo). Instead of theming content in CMS, the CMS content gets merged into theme, themed Plone Mosaic site layouts.

We build our themed layouts with Webpack. Plone Webpack integration allows us to bring in all the bells and whistles we need from the huge open source JavaScript ecosystem without extra effort. And thanks to Patternslib and Webpack code-splitting, huge libraries like MathJax are only loaded when required.

Eventually, the CMS content gets pulled into Plone Mosaic site layouts with “tiles” and “panels”: Tiles are placeholders for any CMS content from page title to body text. Panels are customizable areas, where more tiles can be placed in customizable grid layouts. And when that is not enough, some things can still be tweaked with Plone Diazo XSLT rules…

Configurable blocks with Theme Fragments

The days when it was enough to theme the existing features of a CMS are long gone. On the contrary, nowadays it seems that themes redefine the required features. Lucky us, not only was Plone there from the very beginning, Plone itself started as a themed user interface for Zope Content Management Framework. While the details have changed, in my opinion, Plone could still market itself as a low-code platform for web content management.

configurable tile

Plone Theme Fragments provide flexible way to enrich theme with configurable functional blocks. Minimally, theme fragments are re-usable static HTML fragments usable around the theme. But they can also use all the power of Plone templating language to render the current content in custom manner. Even more, fragments can be bundled with Python functions to allow complex business logic calling most of the Plone backend API while keeping the templates itself simple.

With Plone Mosaic, theme fragments can be also used as tiles in any Mosaic layout. And when that itself is not flexible enough, theme fragment tiles can be made configurable with the full power of Plone Supermodel XML schemata.

All this. Simply as part of any Plone theme. No wonder we have been using these beasts a lot.

Everything bundled with Theme Site Setup

When implementing a theme refresh for an existing web site cluster with dozen of independent CMS installations with tens of thousands of individual pages, it is important to be able to iterate fast. And for theming a Plone cluster that means, to be able to update the theme without need to restart the backend services after each update.

Plone Theme Site Setup add-on for the rescue! Thanks to Theme Site Setup, our theme packages may include everything we need from the usual theming resources and Theme Fragments, to Plone Mosaic layouts, custom language localization catalogs and Plone site configuration changes (like customizing cached image scales).

In practice, we use Plone Webpack integration to produce complete theme packages with all the required resources supported by Plone Theme Site Setup. Then we use Plone Theme Upload to upload the resulting package to our sites on demand. No restarts needed.

Personally cached with Varnish ESI

At the end, no features matter if the resulting web site is slow or its content is not up-to-date. Unfortunately, these two requirements are often contradicting each other. Especially on web portal front pages that mostly aggregate the current content from all the other pages.

configurable intranet

Fortunately, Plone Mosaic was designed with ESI (Edge Side Includes) and tile specific caching configuration in mind. With simple customization of Plone Mosaic rendering pipeline and Plone caching rules, we have been able to achieve everything we wanted:

  • Different parts of our pages are cached for different periods of time. For example, news listings tiles are invalidated from cache in every few minutes, while the rest of the page is only updated when modified.
  • Cache is shared with anonymous and logged-in users when safe. For example, the same cached versions of header and footer tiles get shared between all users.
  • Also most of the tiles for logged-in users can be securely cached: users with the same set of user roles share the same cached version.
  • Thanks to Varnish’ recursive ESI support, we are able to provided cached personalized news listing tiles: the first response is a non-cached tile rendering just the ESI-reference for the cached version, and all users with matching configuration get the same cached version. Fast.

Finally, Plone Webpack integration allows us to build a theme with all its front-end resources server from a separate server, possibly from a CDN, in an optimized manner. Allowing all our sites with the same theme share the same resources, and let Plone to focus on managing and serving the content.


All that said, and as already been said, thanks to Plone, the hardest part really was (and remains to be) the CSS.

Plone and the Pandemic

Posted by PLONE.ORG on April 03, 2020 04:51 PM

The COVID-19 pandemic has upended life across the planet. Scientists, doctors and other experts are engaged in a giant global effort to combat the disease. Open source software is integral to this effort because many tools and libraries used by these experts are open source. That includes Plone, a great platform for building secure, scalable, highly customizable, content-rich websites for scientists and researchers.

One such Plone site is already being used in the COVID-19 fight. The Onkopedia portal publishes medical guidelines for oncology and hematology for Germany, Austria and Switzerland - treatment guidelines, medical studies, protocols, certifications, drug information, etc. for over 60 diseases. Patients with these diseases are at high risk for the new coronavirus, which created an urgent need for providing COVID-19 specific information. In just 3 days the Onkopedia development team was able to extend the site's data model and roll out necessary UI updates to create a new Onkopedia section with COVID-19 content for over 40 different diseases. Congratulations to the team for a great example of agile development, and to Plone for providing a flexible and robust platform.

Do you have a success story about Plone helping to fight the pandemic? Let us know at news@plone.org.

PLOG 2020 Replaced by Remote Sprint

Posted by PLONE.ORG on April 03, 2020 04:01 PM

PLOG Felled by the Pandemic

After months of planning, in early March the PLOG organizers had to face the reality that traveling to Italy in April was not going to be feasible. We tried to reschedule the event in September, but the hotel was already booked up. So we've had to make the sad decision to cancel PLOG for 2020.

PLOG will return in 2021!!!

Remote Sprinting to the Rescue

For the foreseeable future, Plone community sprints will need to happen remotely - individual sprinters in their own spaces, communicating via audio, video, and text. Happily, the big April hole that PLOG's cancellation left in the Plone calendar has now been filled by a remote sprint on Plone 6. Kudos to Peter Holzer, Maik Derstappen and Jens Klein for organizing it! The sprint will tackle a number of topics:

  • Modernize classic Plone's default theme - The goal is to make working with the classic UI fun again. We will make the templates compatible with bootstrap markup and reduce the amount of custom CSS so that it is easy to use any bootstrap template with Plone. 
  • Push Plone to the latest Zope - The Plone CMS runs on top of the Zope web framework which has recently seen some massive performance improvements. To take advantage of that we will work on making Plone 5.2 run with latest Zope 4 and Plone 6 with latest Zope 5.
  • Mastering Plone 6 - We will update the Mastering Plone training to use Plone 6 and Volto for all frontend-tasks.
  • Create marketing materials - Headless Plone and Volto are important to our future, but they aren't clearly presented on plone.org or plone.com. We will work on how-to screencasts for both Classic and Volto UIs for the Plone Youtube channel, and plone.org content about headless Plone and Volto.

Join Us!

Every experience level is welcome, and additional sprint topics are welcome too. Sign up and add your ideas!

The Plone Symposium – Germany, Dresden 2020

Posted by PLONE.ORG on March 25, 2020 03:30 PM

Each year, the Plone German-speaking Community organize a 3 days Symposium (Plone-Tagung 2020) and a two days sprint for German-speaking universities, NGO, companies as well as for users, service providers, and developers.

The symposium this year took place at the Technical University of Dresden (TUD) from 09-13. of March 2020. The Symposium has been supported by the Plone Foundation and sponsored by Werkbank (Gold Sponsor), Zopyx and FlyingCircus (Silber Sponsor) and Abstract-Technology (Bronze Sponsor).

Plone Tagung 2020

Symposium topics

The Plone Symposium in Dresden focused on the opening keynote from Anne-Marie Nebe on accessibility and why web accessibility is so important for user-centered software development. The Humanities and Social Sciences Department of the TU Dresden contributed with their expertise on the linguistic and visual implementation of a barrier-free web presence. And also the second-day keynote "Chaos und Diversity“ from Christian Theune illustrated the characteristics and dynamics of complex systems and talked about new approaches for a conscious use of diversity.

A complete overview of all topics of the Symposium is available, in German, at the Plone Tagung website.

How to make Plone cookies

Building an open-source community isn’t easy but now you can print your own Plone cookie cutter and make these goodies on your own! https://www.thingiverse.com/thing:4226416 thanks to @quirk_dispenser.

Plone Cookies

A big thank goes to the Technical University of Dresden (TUD) for hosting and the Plone-Team TUD
(@plone_tudresden) for the organization of the event and to the speakers and sponsors and to all participants, who made the event special.

And if you weren’t able to make it this year, we hope you’ll join the Plone Tagung next year! See you then.

Volto 4.0 Released

Posted by PLONE.ORG on March 01, 2020 03:30 PM

Volto 4.0 is now ready, following a 9 month alpha period that included many sprints.

A lot of effort has been put into keeping Volto's learning curve low and ensuring that developing with it is a delightful experience.

Here are some of its new features:

* Improved Pastanaga editor
* New Pastanaga editor sidebar
* New mobile first toolbar
* Developing blocks is now easier than ever
* New object browser
* Listing, table of contents, and lead image blocks
* New blocks chooser and future-proof blocks definitions
* Body classes like Plone's hinting at content types, section and current view
* New message system
* React hooks support
* Several internal libraries updated, including Redux, Router ones that support hooks
* Many bug fixes

For further details, please see the Volto 4.0 announcement forum post

Creating Plone content with Transmogrifier on Python 3

Posted by Asko Soukka on February 25, 2020 12:00 PM

TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.

Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.

The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.

Transmogrifying RSS into Plone

So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:

At first ./buildout.cfg for the Plone instance:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
user = admin:admin

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Then buildout must be run to create the instance with a Plone site:

$ buildout

Next the transmogrifier ./pipeline.cfg must be created to define the pipeline:

[transmogrifier]
pipeline =
    from_rss
    prepare
    create
    patch
    commit

[from_rss]
blueprint = transmogrifier.from
modules = feedparser
expression = python:modules['feedparser'].parse(options['url']).get('entries', [])
url = http://rss.slashdot.org/Slashdot/slashdot

[prepare]
blueprint = transmogrifier.set
portal_type = string:Document
id = python:None
text = path:item/summary
_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')

[create]
blueprint = transmogrifier.set
modules = plone.api
object = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)

[patch]
blueprint = transmogrifier.transform
modules = plone.app.textfield
patch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

This example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.

Transmogrifying JSON files into Plone

There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.

In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)

At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:

[buildout]
extends = http://dist.plone.org/release/5-latest/versions.cfg
parts = instance plonesite
versions = versions

extensions = mr.developer
sources = sources
auto-checkout = *

[sources]
transmogrifier = git https://github.com/collective/transmogrifier

[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    transmogrifier
    plone.restapi
user = admin:admin
extra-paths = local
zcml = custom

[plonesite]
recipe = collective.recipe.plonesite
site-id = Plone
instance = instance

Before running buildout we ensure a proper local Python module structure with:

$ mkdir -p local/custom
$ touch local/custom/__init__.py
$ echo '<configure xmlns="http://namespaces.zope.org/zope" />' > local/custom/__init__.py

Only then we run buildout as usually:

$ buildout

Now, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:

# -*- coding: utf-8 -*-
from transmogrifier.blueprints import Blueprint

import json
import pathlib


class Glob(Blueprint):
    """Produce JSON items from files matching globbing from option `glob`."""
    def __iter__(self):
        for item in self.previous:
            yield item
        for p in pathlib.Path(".").glob(self.options["glob"]):
            with open(p, encoding="utf-8") as fp:
                yield json.load(fp)


class Folders(Blueprint):
    """Minimal Folder item producer to ensure that items have containers."""
    def __iter__(self):
        context = self.transmogrifier.context
        for item in self.previous:
            parts = (item.get('_path') or '').strip('/').split('/')[:-1]
            path = ''
            for part in parts:
                path += '/' + part
                try:
                    context.restrictedTraverse(path)
                except KeyError:
                    yield {
                        "_path": path,
                        "_type": "Folder",
                        "id": part
                    }
            yield item

And complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

  <include package="transmogrifier" file="meta.zcml" />

  <transmogrifier:blueprint
      component=".blueprints.Glob"
      name="custom.glob"
      />

  <transmogrifier:blueprint
      component=".blueprints.Folders"
      name="custom.folders"
      />

</configure>

Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:

  • generate new pipeline items from exported JSON files
  • inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
  • create minimal Folder and Document objects with plone.api.
[transmogrifier]
pipeline =
    generate_from_json
    generate_containers
    set_container
    create_folder
    create_document
    commit

[generate_from_json]
blueprint = custom.glob
glob = data/**/*.json

[generate_containers]
blueprint = custom.folders

[set_container]
blueprint = transmogrifier.set
_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])

[create_folder]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Folder"
modules = plone.api
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])

[create_document]
blueprint = transmogrifier.set
condition = python:item.get("_type") == "Document"
modules =
  plone.api
  plone.app.textfield
_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))

[commit]
blueprint = transmogrifier.finally
modules = transaction
commit = modules['transaction'].commit()

Finally, the pipeline can be run and content imported with:

$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSite

Obviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.


If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.

Why Upgrade?

Posted by Jazkarta Blog on February 21, 2020 03:52 PM

Plone 5.2, The Future-Proofing Release: Python 3 and REST API

Technology never stands still.

It’s tempting to think of technology investments as discrete expenditures that permanently solve a problem, but that would be a mistake. A new website that costs $25K, $50K, $100K or more feels like it should last forever. But technology ages and an organization’s needs evolve. Everyone is happy for a short while after the website is completed, but then they become less and less happy as it works less and less well.

A better approach is to not think of technology needs as being solved by big, herculean efforts that happen occasionally, but as an ongoing program that requires ongoing resources. This is what the University of Minnesota Press has done. Since their current website’s initial launch in 2011, they have:

  • 2012: Added a searchable bibliography to the Test Division portion of the website
  • 2015: Done a responsive redesign so that the website works seamlessly on mobile devices
  • 2018: Upgraded the website’s e-commerce infrastructure with modern components providing improved PCI compliance
  • Plus they’ve had a yearly support contract to fix bugs, add features, and keep up with minor version upgrades

This pattern of ongoing investment is typical of our clients. And because of technology changes that have occurred over the last few years, a new round of investments has become imperative: upgrades.

Since 2011 the Press website has been running on version 4 of the content management system Plone, and version 2 of Python, the programming language used to implement Plone. Those versions are nearing obsolescence.

  • Plone 5 has been out since 2016, and Plone 6 is expected soon. When Plone 6 is released, the Plone security team will end official support for Plone 4.
  • Python 3, a major, backwards-incompatible release with many new features, has been out since 2008. Official support for Python 2 ended January 1, 2020.

Because of this, it became essential for the Press – like other organizations that use Plone – to budget for major version upgrades of its website technology stack. This long-term investment will ensure that all technology components are stable, supported, and up to date.

The Plone 5 version upgrade will also provide benefits to users, editors, and website developers.

Users will see:

  • Pages that render 15-20% faster due to a new templating engine
  • Improved accessibility compliance

Editors and admins will see a number of new features under the hood:

  • An improved editing toolbar
  • The latest version of the WYSIWYG rich text editor (TinyMCE)
  • Facebook OpenGraph meta tags and Twitter card support
  • Bulk editing operations such as adding multiple files and images at once
  • The ability to quickly find, sort, reorder, and select content items on the contents view
  • Automatic CSRF (cross-site request forgery) protection integrated into the database layer

Website developers will be able to use:

  • All the features in the latest Python
  • Plone’s improved and easier to use content type framework, Dexterity, as well as other new features in the code

Visit Plone.com to read more about the advantages of Plone 5.

The good news is that this upgrade work can be done in phases – meaning the work can be budgeted over several years if necessary.

  1. A Plone 5.1 upgrade, which includes migrating Plone’s core content types to Dexterity.
  2. Migrating custom content types to Dexterity and replacing any add-ons that are not compatible with Plone 5.2.
  3. A Plone 5.2 and Python 3 upgrade.

Phases 1 and 2 must be done before phase 3 because the old content type framework (Archetypes) is not supported in Python 3.

The end result of this upgrade path is to open up a world of possibilities to organizations using Plone. Out of the box Plone 5.2 includes:

  • plone.restapi, which supports the full set of Plone features (users, groups, roles, workflow, navigation, search, even breadcrumbs)
  • Volto, a modern Javascript front end for Plone based on React

These new components are game changers. In particular, the REST API allows Plone to integrate easily with other systems and to operate as a headless CMS – with the content delivery front end decoupled from the back end.

Now that’s worth upgrading for.

Plone Foundation Ambassadors for 2020

Posted by PLONE.ORG on February 15, 2020 04:17 PM

The Plone Foundation Board has appointed these outstanding individuals from around the world to serve as ambassadors for Plone.

They are known for their community involvement in their regions and for their expertise in a particular market of Plone.

In their capacity as Plone ambassadors, they will continue to promote Plone regionally and in their specific markets for Plone.

Plone's ambassadors for 2020 are:

  • David Bain, Jamaica
  • Leonardo Caballero, Venezuela
  • Manabu Terada, Japan
  • Max Jakob, Germany, Education
  • William Fennie, United States, Education
  • Thomas Buchberger, Switzerland, PloneGov
  • Joël Lambillotte, Belgium, PloneGov
  • Ramiro Batista, Brazil, PloneGov


See https://plone.org/foundation/board/ambassadors for more information


"Flags of member nations flying at United Nations Headquarters". 30/Dec/2005. UN Photo/Joao Araujo Pinto.

New Waitress version, and updated 20200121 hotfix

Posted by PLONE.ORG on February 11, 2020 11:29 PM

Waitress

If you use Waitress, please upgrade from 1.4.2 to 1.4.3. 

The Pylons Project released a new version of Waitress to fix a bug in the regular expression that was used to parse HTTP headers. The bug could cause the waitress process to use excessive CPU.

As Plone 5.2.1 uses Waitress 1.4.2, we recommend changing the version pin to 1.4.3 in your buildout.

[versions]

waitress = 1.4.3

Updated 20200121 Hotfix

As announced previously, the 20200121 hotfix includes several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting.

Version 1.1, released on February 11, 2020, includes an update for the SQL Injection fix, which will not be needed for all installations.

If you are not using SQL in your website you do NOT need to upgrade, though you can if you want to. Default Plone does not need it. Upgrading to this version is especially recommended when you use PostgreSQL. Note that RelStorage is not affected. For details and discussion, see DocumentTemplate issue #48

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the update patch on 2020-02-11T22:47:25+0000.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

PLOG 2020 Training - 3 Weeks Left to Register!

Posted by PLONE.ORG on February 08, 2020 10:08 PM

Free Training

PLOG is a unique sprint where you can spend the mornings training, the afternoons sprinting, and all day enjoying the southern Italian weather and food. Thank you to the Plone Foundation, which has provided funding so that we can offer the following FREE training classes. It's a great way to brush up some old skills and pick up new ones at the halfway point between Plone conferences.

Thank you to the Plone Foundation, which is providing funding to support the training classes. 

Hands on Volto -- Taught by the RedTurtle Team -- 2 Mornings

This class will be based on the Volto training given at the last Plone Conference. It assumes some knowledge of React (but you don't have to be an expert.)

How to Get More Value From Your Plone Site Using GatsbyJS -- Taught by Asko Soukka -- 2 Mornings

This class will teach you how to deploy a static website from Plone content, and how to integrate it with the GatsbyJS ecosystem (plugins, themes, cloud services, etc.)

Plone Tips and Tricks -- Taught by Philip Bauer -- 1 Morning

This class will cover a number of advanced Plone developer topics, including:

  • Python Debugging Best Practices
  • Debugging ZODB Issues

Strategic Sprint

Because of its importance to the community, PLOG has been designated a strategic sprint by the Foundation Board and the Marketing Team. A wonderful cross section of people will attend, so there will be sprinting activities for all interests. Some will be working on Volto and RestAPI improvements, others will be creating marketing collateral in the form of "how to" screencasts of Plone features. Bring your own topic - the final plan will be made at the sprint.

Less Than 3 Weeks Left to Register!

The registration deadline is February 28th, after that we cannot guarantee the discounted rates or room availability.

Register now

Want to Learn More?

Read all about the Plone Open Garden, a tradition in our community since 2007. There's even a PLOG video. Dates, prices, and all of the details are given on the PLOG 2020 event page.

Security patch released 20200121

Posted by PLONE.ORG on January 21, 2020 03:10 PM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers: CVE-2020-7936, CVE-2020-7937, CVE-2020-7938, CVE-2020-7939, CVE-2020-7940, CVE-2020-7941.

Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.

The patch was released at 2020-01-21 15:00 UTC.

Installation

Full installation instructions are available on the HotFix release page.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

If you do not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team released the patch at 2020-01-21 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as tarball-style archives that may be unpacked into the products folder of a buildout installation (for Plone 5.1.x and earlier only) and as Python packages that may be installed by editing a buildout configuration file and running buildout. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers  There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

20200121

Posted by PLONE.ORG on January 21, 2020 03:00 PM
Several fixes for privilege escalation, open redirect, password strength, overwriting files, SQL injection, and cross site scripting. Version 1.1 released February 11, 2020, with an update for the SQL Injection fix, which will not be needed for all.

Security vulnerability pre-announcement: 20200121

Posted by PLONE.ORG on January 08, 2020 04:43 AM
This is a routine patch with our standard 14 day notice period. There is no evidence that the issues fixed here are being used against any sites.

CVE numbers not yet issued.

Versions Affected: All supported Plone versions (4.x, 5.x). Previous versions could be affected but have not been tested.

Versions Not Affected: None.

Nature of vulnerability: Low severity, no data exposure or privilege escalation for anonymous users.

The patch will be released at 2020-01-21 15:00 UTC.

Preparation

This is a pre-announcement of availability of this security fix. 

The security fix egg will be named Products.PloneHotfix20200121 and its version will be 1.0. Further installation instructions will be made available when the fix is released.

Standard security advice

  • Make sure that the Zope/Plone service is running with minimum privileges. Ideally, the Zope and ZEO services should be able to write only to log and data directories. Plone sites installed through our installers already do this.
  • Use an intrusion detection system that monitors key system resources for unauthorized changes.
  • Monitor your Zope, reverse-proxy request and system logs for unusual activity.
  • Make sure your administrator stays up to date, by following the special low-volume Plone Security Announcements list via email, RSS and/or Twitter

These are standard precautions that should be employed on any production system, and are not tied to this fix.

Extra Help

Should you not have in-house server administrators or a service agreement for supporting your website, you can find consulting companies at plone.com/providers

There is also free support available online via the Plone forum and the Plone chat channels.

Q: When will the patch be made available?
A: The Plone Security Team will release the patch at 2020-01-21 15:00 UTC.

Q. What will be involved in applying the patch?
A. Patches are made available as Python packages that may be installed by editing a buildout configuration file and running buildout. For Plone 5.1 and lower they are also available as tarball-style archives that may be unpacked into the products folder of a buildout installation. Patching is generally easy and quick to accomplish.

Q: How were these vulnerabilities found?
A: The vulnerabilities were found by users submitting them to the security mailing list.

Q: My site is highly visible and mission-critical. I hear the patch has already been developed. Can I get the fix before the release date?
A: No. The patch will be made available to all administrators at the same time. There are no exceptions.

Q: If the patch has been developed already, why isn't it made available to the public now?
A: The Security Team is still testing the patch against a wide variety of configurations and running various scenarios thoroughly. The team is also making sure everybody has appropriate time to plan to patch their Plone installation(s). Some consultancy organizations have hundreds of sites to patch and need the extra time to coordinate their efforts with their clients.

Q: How does one exploit the vulnerability?
A: This information will not be made public until after the patch is made available.

Q: Is my Plone site at risk for this vulnerability? How do I know if my site has been exploited? How can I confirm that the hotfix is installed correctly and my site is protected?

A: Details about the vulnerability will be revealed at the same time as the patch.

Q: How can I report other potential security vulnerabilities?

A: Please email the Plone Security Team at security@plone.org rather than publicly discussing potential security issues.

Q: How can I apply the patch without affecting my users?

A: Even though this patch does NOT require you to run buildout, you can run buildout without affecting your users. You can restart a multi-client Plone install without affecting your users; see http://docs.plone.org/manage/deploying/processes.html  

Q: How do I get help patching my site?

A: Plone service providers are listed at plone.com/providers There is also free support available online via the Plone forum and the Plone chat channels

Q: Who is on the Plone Security Team and how is it funded?

A: The Plone Security Team is made up of volunteers who are experienced developers familiar with the Plone code base and with security exploits. The Plone Security Team is not funded; members and/or their employers have volunteered their time in the interests of the greater Plone community.

Q: How can I help the Plone Security Team?

A: The Plone Security Team is looking for help from security-minded developers and testers. Volunteers must be known to the Security Team and have been part of the Plone community for some time. To help the Security Team financially, your donations are most welcome at http://plone.org/sponsors

General questions about this announcement, Plone patching procedures, and availability of support may be addressed to the Plone support forums If you have specific questions about this vulnerability or its handling, contact the Plone Security Team at security@plone.org

To report potentially security-related issues, email the Plone Security Team at security@plone.org We are always happy to credit individuals and companies who make responsible disclosures.

Information for Vulnerability Database Maintainers

We will apply for CVE numbers for these issues. Further information on individual vulnerabilities (including CVSS scores, CWE identifiers and summaries) will be available at the full vulnerability list.

A Volto gotcha when dealing with async calls

Posted by PloneExpanse on December 11, 2019 08:35 PM
Just some quick notes, in case this might help someone. After quite a bit of time and tests in trying to use asyncConnect to get data in a Volto component view (strictly focusing on the SSR side), I’ve realized that what I’m trying to do is not supported by the redux-connect library. In Volto, right now there are two components that use asyncConnect: App.jsx and Search.jsx. The purpose of asyncConnect is to have the server side rendered page “dynamic”, depending on the input from the originating request.

PLOG 2020 Registration Now Open

Posted by PLONE.ORG on November 27, 2019 04:47 PM

Sign up now!

Registration deadline is February 28, 2020

Training classes on several topics will be held in the mornings, sprinting will happen in the afternoons and discussions will go on all day. The Hotel Mediterraneo is in a beautiful location overlooking the Bay of Naples, with wonderful food and a lovely garden where the sprinting and training will be held.

We have reserved a limited number of rooms and they are available on a first come first served basis. Breakfast and dinner are included. The rooms for three and four persons are the most economical and are expected to sell out early.

We are looking for women to share a triple!

Questions? Email us at plog@plone.org

Event details ~~ Read About PLOG ~~ Watch the Video ~~ Register

Python, the most popular programming language of the year

Posted by CodeSyntax on November 19, 2019 06:56 AM
IEEE Spectrum has published its sixth annual list with the most popular programming languages of the year across multiple platforms, and, once again Python repeats in 2019 as the undisputed leader, as it will happen in 2017 and 2018.

Speedup volto razzle builds

Posted by PloneExpanse on November 17, 2019 12:58 PM
I’ve been looking for a way to speedup Volto razzle/webpack builds, both while developing and for “production” mode, when building the final bundle. Fortunately, this solution exists and it’s extremely easy to integrate. Let’s define the problem, to see how to approach it: what is Volto actually? What do you get when you open, in your browser, a Volto frontend Plone website? To greatly simplify (and I hope I didn’t get anything wrong as I am not a Volto core developer):

Four Members Join the Plone Foundation

Posted by PLONE.ORG on November 13, 2019 09:47 PM

The Plone Foundation welcomes four new members, after unanimous confirmation by the Foundation's Board of Directors on October 10, 2019.

Membership in the Foundation is conferred for significant and enduring contributions to the Plone project and community. The Plone Foundation Membership Committee overwhelmingly recommended each applicant for their ongoing contributions to Plone.

Fulvio Casali

Fulvio Casali

Fulvio started working with Plone in 2008 and since 2012 he has been operating a consultancy business, Soliton Consulting, specializing almost exclusively on Plone. Fulvio organized Emerald Sprints (2013, 2014), Plone Open Garden (2016, 2017, 2019, 2020), participated in every Plone Conference since 2010 (except one!), has given two conference talks, and attended the Plone Symposium East 2010 and the Plone Konferenz in 2012. Fulvio has provided Plone training for Mastering Plone, Diazo theming, Rapido, Angular, and TTW Dexterity.

Stefania Trabucchi

Stefania Trabucchi

Stefania is the co-founder of Abstract Technology in Germany. She was the organizer of Plone Meet Up Berlin 2004-2014, of World Plone Day Berlin 2009-2015, of Plone Social Sprint Berlin 2014, and of Plone's Europython 2014 presence and associated marketing. She has participated in the Plone Konferenz 2012 as a speaker, the Plone Beethoven Sprint 2019, the Plone Conference 2014 sprint, and the Plone Open Garden sprints in 2014 and 2015. Stefania represented Plone as a member of the CMS Garden 2014-2017 and is active in the Plone Intranet Consortium with Quaive.

Thomas Buchberger

Thomas Buchberger

Thomas is the CTO of 4teamwork AG, has been working with Plone since 2006 and became a code contributor in 2011. He has attended many Plone conferences, beginning with Naples in 2007. He attended many sprints, including the Barcelona Strategic Sprint (2016), the Bonn Beethoven Sprint (2017 and 2018), the Sorrento Sprint on Frontend Modernization and Python 3 Porting (2019), and the Beethoven Sprint (2019). Thomas' main code contributions have been in the plone.rest and plone.restapi modules.

Andrea Cecchi

Andrea Cecchi

Andrea was the lead organizer of the Plone Conference 2019 in Ferrara. He participated at earlier Plone conferences and sprints (Sorrento 2019, Tokyo 2018, Barcelona 2017, Bristol 2014). As part of the RedTurtle team, he has organized and participated in World Plone Day events since its inception. Andrea is the maintainer of over 100 packages on PyPI.org.

 

 

The Foundation is the trustee for Plone's intellectual property, works to protect and promote Plone, and has over 80 active members.

Essential Plone Add-ons

Posted by Jazkarta Blog on November 11, 2019 03:45 PM

If it’s fall, it must be time for the Plone Conference. This year the annual gathering took place in Ferrara, a beautiful small city in the north of Italy. The weather was perfect, the streets medieval, the party was in a real castle, and the food – well! The food was amazing. This is the traditional dish cappellacci di zucca al ragù, pasta stuffed with pumpkin in a meat sauce. Yes it tastes as good as it looks.

Cappellacci di zucca al ragù

Following the tradition begun at the Barcelona conference and continued in Tokyo, we held a popularity contest to identify the best add-ons for Plone, Python’s open source CMS. Plone comes with tons of features out-of-the-box – like workflows, search, a multilingual UI, conformance to accessibility standards, and granular user roles and permissions – but it also offers an extensible platform for building new add-ons. Attendees nominated their favorites and the results were posted in the conference venue where people voted their top 5 using sticky dots.

Add-on voting sheets

Thirty-three add-ons were nominated, and the voting revealed a few that are particularly popular – notably for form generation and faceted search. Others included add-ons for document generation (Word, PDF, etc.), image cropping, taxonomies, authentication, and lazy loading. The full results can be found at the 2019 essential Plone add-ons page.

Plone Foundation Board Officers Selected for 2019-2020

Posted by PLONE.ORG on November 08, 2019 09:22 PM

Following the election held in October 2019, the 2019-2020 Plone Foundation Board held its first meeting on November 7, 2019, and its first order of business was to select its officers:

  • President: Chrissy Wainwright
  • Vice President: Paul Roeland
  • Secretary: Andy Leeb
  • Treasurer (non-voting): Jen Myers

The Board also selected the following committee chairs:

  • Marketing: Érico Andrei (assisted by Kim Nguyen)
  • Membership Co-Chairs: Érico Andrei, Kim Nguyen

The following (unofficial) team liaisons were selected:

  • Framework: Chrissy Wainwright
  • Frontend: Victor Fernández de Alba
  • Education: to be determined
  • Security: Jens Klein
  • Guillotina: Andy Leeb

The new Board is determined to reflect the changing landscape of the Plone Foundation, where it is not just about the Plone CMS but a host of technologies and communities that reflect the same spirit, including Volto, Guillotina, Zope and others.


About the Plone Foundation

About the Plone Foundation Board

New Foundation Board for 2019-2020

Posted by PLONE.ORG on November 07, 2019 05:12 PM

The Plone Foundation Board is now in its 2019-2020 term. 

The results of the Foundation Membership vote were announced at the Annual General Meeting held in Ferrara on October 25, 2019.

The 2019-2020 Board members are:

William Fennie was not elected although he received over 30% of votes.

Our outgoing Board members chose not to run this year. We thank them for their many years of service!

  • Alexander Loechel
  • Carol Ganz
  • T. Kim Nguyen

Lightning talks Friday

Posted by Maurits van Rees on October 31, 2019 07:36 AM

Wolfgang Thomas: OIRA

Demo of tool for online risk assessment. Export to Word (with python-docx). Training module. Export to Powerpoint presentation with python-pptx.

Alex, Guido: Quaive

State of Plone Intranet in 2019. Dozens of packages. Social stream, search, auditing, real-time document collaboration. Classifieds app makes it nice to share info, legal app showing contracts. PAS graph optimizations, see talk by Alessandro. SVG rendering. Progressive web app, we did not want to create a special app, but it does really give a mobile experience.

Busy supporting Plone 5.2 and Python 3, we will manage that. We will use crossbar.io for push notifications. Encryption via CWS. Johannes is joining Syslab to work on Quaive fulltime. iMio is helping too.

Michele Finelli: vini delle sabie

The last of my three easy pieces on Ferrara.

vini delle sabie is a wine. Sand is influencing the grapes. The grapes are actually immune to some diseases. Most typical is de fortana grape. Give it a try. Perfect companion to the dishes I talked about.

Fred van Dijk: collective.ldapsetup

Example package with LDAP setup. Uses pas.plugins.ldap, which is a replacement for plone.app.ldap. The core package now supports wildcard search, with help from Asko Soukka. Robert Niederreiter did a lot of work to get it working on Python 3. LDAPS support.

Code: https://github.com/collective/collective.ldapsetup

Gauthier Bastien collective.documentgenerator

Desktop document generation (.odt, .pdf, .doc, ...) based on appy framework and OpenOffice/LibreOffice. Templating in LibreOffice.

Code: https://github.com/collective/collective.documentgenerator

Maik Derstappen: EasyNewsletter

Create news letters in Plone. Send mails to subscribers. Let email server do this, or use an external delivery service. On a collection you can define how items are rendered in the email. You can customize the output and aggregation template. Or fully write your own. Works now on Plone 5.1/5.2 Python 2.7/3.7.

Planned: integrate Mosaico editor.

Code: https://github.com/collective/Products.EasyNewsletter

Paul Grunewald: Plone Tagung

9 to 11 March 2020 in Dresden. Talks and sprints. Come!

Philip Bauer: Training Plone 6

We plan to create quickstart Plone trainings for Python devs, one for frontend devs, one for users. Mastering Plone 5.2 talks about a lot, you should follow it.

But what about mastering Plone 6. It would need to be about backend and frontend. I would love feedback about what would be required. We do not want to overwhelm people.

Maurits van Rees: 3 authentication add-ons

I present three PAS plugins:

  • collective.denyroles
  • pas.plugins.headers
  • collective.contentgroups

See the full presentation.

Treasure hunt solutions

There was a treasure hunt this week, where every day you had to find three items in the city and make pictures. We show the solutions.

Timo Stollenwerk: Cypress

We use this Acceptance testing framework to test Volto. It has gained traction in the JavaScript world.

I use robotframework and love it. But Cypress is written in JavaScript, so an intern could write extra plugin library for it.

See https://www.cypress.io

Timo Stollenwerk: Sprints

The next two days we will have Plone sprints. Everyone is welcome. It is a great chance to share code, get to know the Plone code, ask questions to coders, and generally work together. We will find someone to pair up with you.

See the list of sprint topics. You can work on other stuff, you can add ideas, also when you will not work on it. You can add your name to topics when you are interested, also multiple topics.

Sally Kleinfeldt: Essential Plone 5 add-ons

People could suggest add-ons, and then people voted for them. Clear winners:

  • collective.easyform
  • plone.restapi (in core now actually)
  • eea.facetednavigation

Others:

  • collective.documentgenerator
  • collective.z3cform.datagridfield
  • collective.taxonomy
  • and more

Plone Conference 2020

The Plone conference next year will be in... Namur, Belgium. November 16-22.

Thank you

Thank you Red Turtle for organising such a lovely conference!

Panel: Frameworks comparison

Posted by Maurits van Rees on October 25, 2019 04:04 PM

Plan:

  • Brief introduction to each framework
  • Followed by discussion focused on use cases
  • Maurits will live blog (thanks!)
  • Framework vs. Product - let’s not worry about that!

For each system:

  • What use cases is it well suited to
  • What use cases is it poorly suited to

Discussion:

  • Focus on client use cases and what frameworks would be a good or poor fit
  • You can also ask questions!

Live blogging.

Use cases:

  • Non-profit with custom forms and import/export
  • Concert venue with strict event listing and advanced settings.
  • NGO needs a database application for their highly confidential data.
  • Generate a map out of data, and store extra information, like images for the data.
  • Three-person company with basically static pages, non-technical users.
  • Real-time application with websockets.
  • A large company wants to monitor how well their customer service staff are doing. Strict form with statistics on it.

Pyramid:

  • Best for APIs, not really CMS or websites, but applications
  • Start small, grow
  • Complex non-profit use case: you would have to build everything yourself.
  • Event listing: we have a site that manages stuff for music bands as example, you just need some endpoints. Maybe websauna on top.
  • NGO privacy: security really good, and simple enough to understand the full system. You can fully build your own fine grained security system. UI is a lot of work.
  • 100 person company wants intranet with various content, installable by local IT team.
  • Intranet: try to create a good solution for multiple clients, not tailored to one in a consulting project. Pyramid is good for building such products.
  • Map: pyramid does not care which database you use. In a project we are mapping electical grids. In Guillotina they think they are nice with AsyncIO, but it makes their whole codebase harder to understand.
  • Real-time: we had websockets for a year, but it was too slow, so we rewrote it in Go-Lang.
  • Form statistics: you can create a form, store it in a database, export it, fine.

Django:

  • Aimed at 80% of use cases, simple auth system
  • Good for CRUD apps, smooth beginning
  • Non-profit: export/import is good, we can do forms. Sharing access gets tricky
  • Event listing: Django can do it, but just use Wordpress. But the technical user can do this in the admin interface.
  • NGO privacy: permissions are too simple OOTB, not per object. You may want end-to-end encryption, which no framework offers. Limited built-in audit log, nice start.
  • Intranet: not good fit for Plone for the people who do not often work with it. Django has easier CMSes for this. The expenses claim form could be easier to integrate into Django. Deployment just needs Postgres. DjangoCMS is Plone Light. Wagtail is WordPress Plus.
  • Map: any of the frameworks except Plone can do the geo stuff in postgres. Images just on the file system. Static and uploaded files are handled fine by Django, can also be in the cloud.
  • Real-time: not many Django users will care about websockets, will never be the focus.
  • Form statistics: you can but I would not do it. This feels like a custom web application. Do it in any framework.

Guillotina:

  • Small framework, scale from small to big
  • Non-profit with forms: decent fit, use JSON schema
  • Event listing: go to wix.com or something, it seems too simple
  • NGO privacy: Good fit, permissions are no problem for this. You need to build the full UI. Okay, you may be able to start with Volto as UI, but that needs integration in the backend.
  • Intranet: backend can handle it, need to build UI, deployment can be with docker and kubernetes.
  • Map: for the large files, Guillotina wins because of AsyncIO.
  • Real-time: we use websockets just fine.
  • Form statistics: you can build an app, depending on your UI skills.

Plone:

  • Good when you have different use groups with complex requirements for security.
  • Edit interface is doable, but less so on mobile.
  • Import/export complicated, but you can hire Jens.
  • Difficult to find skilled people.
  • Non-profit with forms: really good fit, end-users can create the forms. The import/export would need help from a provider.
  • Event listing: do not bother with Plone. It sounds more relational, Plone would be overkill.
  • NGO privacy: you need deep understanding of Plone security, hierarchy for storing data. Audit logging with add-on.
  • Intranet: Plone is fine. You may need to help with installing at first, but it will keep running once installed.
  • Small almost static site: none of us.
  • Real-time: once Asko is done with his ZServer improvements, we can do it.
  • Form statistics: do not use Plone. Use sed, awk, perl. Technically you could use some add-ons.

Maurits van Rees: 3 authentication add-ons

Posted by Maurits van Rees on October 25, 2019 03:28 PM

Three PAS plugins:

  • collective.denyroles
  • pas.plugins.headers
  • collective.contentgroups

collective.denyroles

Deny access to roles like Manager and Editor

  • Use case: Manager only logs in to edit-domain, not live site.
  • By default deny access to Manager, Editor, etc.
  • env DENY_ROLES=0 to disable
  • or Apache/nginx header X_DONT_CHECK_ROLES
  • Actually not a plugin, but a patch.

Code: https://github.com/collective/collective.denyroles/

pas.plugins.headers

PAS plugin for authentication based on request headers.

  • Use case: Apache/nginx adds SAML headers to requests.

  • Configuration in ZMI or profiles/default/pas.plugins.headers.json:

    {
        "userid_header": "uid",
        "required_headers": ["uid"],
        "roles_header": "roles",
        "allowed_roles": ["student", "teacher"],
        "deny_unauthorized": true,
        "redirect_url": "https://maurits.vanrees.org",
        "memberdata_to_header": [
            "fullname|HEADER_firstname HEADER_lastname"
        ]
    }
    

Code: https://github.com/collective/pas.plugins.headers/

collective.contentgroups

Plone PAS plugin for content as groups.

  • Use case: create content item that works as a group.
  • dexterity behavior
  • No Products.membrane, no Products.remember, no dexterity.membrane.
  • No separate membrane_catalog.
  • Only groups, not users.
  • No multiple inheritance, just AccessControl.users.BasicUser.

Code: https://github.com/collective/collective.contentgroups

Alessandro Pisa: PAS adventures

Posted by Maurits van Rees on October 25, 2019 01:36 PM

What is PAS? It is the Pluggable Authentication Service from Zope and Plone. It manages everything related to users and groups, like authentication, permissions, searching. It can get or set information from your site or the request or an external service like LDAP.

It uses plugins, so you can register your own. You can see a lot of those in the ZMI at the Zope root, at http://localhost:8080/acl_users/plugins/manage_active.

There are about twenty different plugin types, and they interact with each other.

In Plone we have Products.PlonePAS. It adds Plone-specific plugins and types in PAS, for example the local roles plugins. Local roles are regular parts of Zope, but not pluggable there. So this is at http://localhost:8080/Plone/acl_users/plugins/manage_active

So this is the basis to manage security, users and groups, using plugins that can be activated, deactivated, and ordered, and these plugins interact with each other.

Now let's go to the kitchen and make a plugin. We want our Plone Intranet to use an LDAP plugin. At the time, we started using Products.PloneLDAP, which builds on a few other Zope packages. (Currently pas.plugins.ldap is the more modern choice.)

We also wanted Products.membrane: this allows to create content that works as user or group. It uses a separate membrane_catalog, a sort-of copy of the portal_catalog, for better or worse.

We wanted collective.workspace to manage roles through groups instead of sharing.

And Plone Intranet itself had some extra sauce (plugins) on top. And important: caching, for performance.

It worked fine! But after a while it slowed down. We had many calls to LDAP, in the wrong way, even with the cache in place. We were using the membrane catalog too much, causing the catalog queue to flush, causing a slowdown in the standard portal_catalog search. We were also writing on SOLR, again external, which did not help for performance.

So what was the reason of the bottleneck? The big deal is when PAS does not find what it is looking for. It then tries all relevant plugins, which are intertwined. For example it finds a workspace group, and needlessly goes to LDAP to search for this group there, where it will not be found, etcetera.

So maybe we could store the LDAP information in Plone. We decided to sync LDAP in the Plone objects.

And then we also patched PAS. Finding a group results in trying to get the properties, parent groups, roles. Our patched PAS skipped plugins that looked unrelated.

There was another problem: a custom local role manager. There were users that belonged to lots of workspaces. So also to lots of groups. So role assignment was slow: you need to get all roles of all groups. We replaced Plone's local role manager with a custom one.

Another plugin is the recursive groups plugin. Standard in Plone. This can be really expensive. For each of those thousands of groups you would check if they have a parent group. We replaced this with a utility that could handle this fast, using graphs with networkx. With this, we could replace all our group plugins, especially the recursive groups plugin.

Possible problem: in the graph we had 20.000 nodes, 25.000 edges, 25 relations. But creating the graph took 50 milliseconds, so that was okay. The nodes are strings like principal_type:UID, for example collective.workspaces:Admins. We stored the information in OOBTrees: uid2path, path2name, name2uid, and an OOTreeSet called edges. And then nx.DiGraph(uid2path) to create the graph. This helped speed things up a lot.

To keep the data structure up to date, we use events.

To recap:

  • remove external connections when possible
  • know the limitations of your plugins
  • you can patch PAS to avoid some plugins
  • you can use a custom group plugin

Further ideas:

  • lazily create collective.workspace groups
  • lazily fetch group and user properties and roles

Takeaways:

  • PAS is great, but mixing and abusing plugins can be deadly
  • Sometimes replace plugins with a new one that does exactly what you need.

Rodrigo Ferreira de Souza: Data migration to Plone 5.2 and Volto

Posted by Maurits van Rees on October 25, 2019 12:26 PM

At KitConcept we have some websites that need to be deployed in Plone 5.2. We could have used collective.transmogrifier to migrate from 4.3 to 5+. And from 5.1 to 5.2 migrate the ZODB from Python 2.7 to Python 3. Transmogrifier is also an option for that part.

Why would we use transmogrifier? There are many generic pipelines available for common cases. Flexibility to deal with different use cases. And it is actually a brilliant wayto use the iterator design pattern. Every piece of the pipeline changes a small thing, and hands the information over to the next piece.

Clients:

  • Large university site
  • High-profile government client
  • Large research institution client website

Challenge: go to Python 3, Plone 5.2 and Volto. With Volto, they spare a migration from Plone 5 to 6, at least partially. Volto is a way to sell the client a Python 3 upgrade: it is the right time to use Volto.

We use Jenkins to test the migration.

General backend things:

  • Old ATTopics to core Collections. Use collective.jsonify to export it, in a way that already looks more like a Collection.
  • Rich text: migrate to Volto blocks.
  • Post migration: collective.cover needed special handling.

What we polish to enter in Volto land:

  • We use collective folderish types.
  • deal with default pages
  • Convert RichText to Volto DraftJS
  • easily point to old website when content is not imported
  • fix some urls (we want to use resolveuid)
  • simple folders we should turn info page with listing block
  • simple collections we should turn info page with collection block

Question: can you open source the module that handles the RichText to Volto blocks migration?

Answer: yes, it is actually just 38 lines with React in Node. I tried to do it in Python, but could not get it to work. And we can sprint on it.

Open Plone Board Meeting

Posted by Maurits van Rees on October 25, 2019 11:46 AM

See the annual report.

Jen Myers has graciously offered to stay on next year as treasurer.

The Zope Foundation has been added to the Plone Foundation. Still ongoing, complex process.

With the Pylons community, those talks have stalled at the moment. Might happen in the future.

Financially we lost more money than we usually do. There was a trademark conflict that cost a lot more than we thought.

Some money should come from the Zope Foundation by the way.

Results of the vote for the board.

48 valid votes, one invalid. 2 went to the spam folder, but we found them. 30 percent of votes were in paper.

We had a vote by the foundation membership. Voted into the Plone board have been: Victor, Erico, Chrissy, Andy, Paul, Jens, Fulvio.

Meeting closed by Paul.

Thanks to Alexander, Kim and Carol who are leaving the board.

Riccardo Lemmi: Deployment Automation

Posted by Maurits van Rees on October 25, 2019 10:34 AM

We wanted to find an easy way to reproduce the installation process. We use:

  • Vagrant
  • Fabric
  • AWS
  • Boto 3 / awscli

Vagrant manages virtual machines and containers, for example with VirtualBox. I use the init command to do some more configuration on a default box.

Then I use Fabric to make an ssh connection to the machine and do remote actions. It uses invoke and paramiko for this. You can let it run the same actions on different machines. You can also use it to transfer files to and from the server, or use sudo to restart Apache.

More libraries with Fabric:

  • fabtools to make sure that for example Python is installed, or a user is created.
  • Cuisine: update files. You can also use this tool to ensure packages and users, so parts are very similar to fabtools.

Next as AWS, Amazon Web Services. With this we deploy production and test machines in a simple and replicable way. You can choose to add more CPUs, bigger disks, more memory, etcetara. I use EC2 (elastic compute cloud), EBS (elastic block storage) and EIP (elastic IP) for the most. Snapshots as simple backup tool. Security group rules as a firewall.

I create a machine with Boto 3 or awscli, both available with pip install. Why would I script this? To have "infrastructure as code". When you manually try to replicate a server, you can easily forget things.

Philip Bauer: Migrations! Migrations! Migrations!

Posted by Maurits van Rees on October 25, 2019 10:02 AM

I have some upgrade code that you can use, or for some parts copy and adapt for your use case.

Code: https://github.com/collective/collective.migrationhelpers

Upgrade steps:

  • Do imports in the function.
  • Do them conditional, so it does not fail if some package does not exist.
  • Do not let a step fail when run a second time.

You may want to disable LDAP temporarily in an upgrade step.

Get the birds eye view, with some code that reports:

  • how many items are there
  • which portal types
  • how big
  • local roles
  • etcetera
  • How many items are there that need to be replaced or removed, like PloneFormGen or Collage.

Divide and conquer:

  • Deal with one problem at a time.
  • Ignore problems that don't block you. You may try to solve something in the old Plone 4.3 site which is already fixed just by completing the migration to Plone 5.

You can register upgrade steps conditionally in zcml if needed, for example with zcml:condition="installed plone-52".

Make big problems small:

  • Write something that removes 98 percent of your content, for testing. Keep the structure in place though: Folders may have portlets or local roles that give problems that you want to know of.
  • Do not migrate blobs. The PDF that lives on the filesystem will not change. Move the blobstorage out of the way, use experimental.gracefulblobmissing during migration, and move blobstorage in again.
  • Copy the Data.fs.index too, if it is a really big site.

Forget the past:

  • remove all revisions
  • maybe manually use collective.revisionmanager for this.
  • pack the database with zero days (bin/zeopack keeps the last day by default).
  • Remove no longer needed portlets.
  • You can use uninstall and upgrade profiles. I like having the upgrade code in Python, but sometimes a profile is much easier.
  • Remove utilities and adapters from add-ons that will be removed.
  • Sometimes you cannot easily remove a package. You can make an alias for it. plone.app.upgrade has code for this, for example to not crash on an old site that still expects the kupu editor package somewhere.

Migrating LinguaPlone:

  • Content editors may have done crazy things, combining folders and content from different languages.
  • We have code to migrate this to plone.app.multilingual in migration.plonehelpers.

Update to Plone 5.2:

  • Use Python 2.7 at first.
  • Include Archetypes if needed for migration.
  • Run the migration to migrate to dexterity.

Archetypes to dexterity:

  • Use the methods from pac_migration from plone.app.contenttypes.
  • Start with the containers/folders.
  • Do one type at a time.
  • Especially for blob-like items there are options to speed this up, like disabling updating SearcheableText.
  • You can write custom migrators, which might just need a few lines.

Alternative: inplace migrator from ftw.upgrade. Interesting for large folders. Might be a nice PR to get this part into core.

To Python 3:

  • We need to remove some archetypes tools.
  • Then remove the Archetypes eggs from buildout and use Python 3.
  • Run the zodbupdate script.
  • When using RelStorage, you may want to switch to filestorage temporarily.
  • Read the documentation please, especially: - upgrade to Python 3 - upgrade zodb to Python 3

Why do it this way? I don't want every Plone company to have their own migration code stashed away somewhere. Please contribute to the core.

Upgrade to Plone 6? Install plone.restapi, use Volto, done.

Asko Soukka: ZServer reloaded with HTTP/2 and WebSocket support

Posted by Maurits van Rees on October 25, 2019 08:46 AM

A year ago, ZServer was one of the last Zope parts that did not run on Python 3. The idea was to use WSGI instead. I wondered why Python 3 would not be possible. So I tried porting it. It worked. And I added websocket support. You can use this to automatically push pages from Plone to Gatsby. Or show a notification when a page needs review, via content rules.

I replaced the old Medusa server with Twisted. ZServer was optionally using Twisted years ago, but this was ripped out, and I don't know why. But I got it working again. So for HTTP and webdav we have a Twisted http server and thread pool with an async FileSender, an AsyncIO/uvloop when avilable, standard ZConfig configuration and logging, And HTTP/2 works as well.

Bonus: websockets with PubSub messaging. It uses the Autobahn Twisted WebSocket protocol. ZeroMQ PubSub cockets using IPC-socket. The connection autosubscribes to local events. The server publishes a guarded event, for example indicating that an event is restricted to Editors.

One problem at this moment with the wsgi setup using waitress, is that the requests are limited to the number of wsgi threads, so server large files can become problematic. There are probably ways around this, but ZServer does not have this problem.

Status:

  • My ZServer fork is at https://github.com/datakurre/ZServer. See mostly branch datakurre/master.
  • You will need a special branch of plone.recipe.zope2instance as well.
  • Plus collective.wsevents, plonectl, collective.taskqueue.

Upstream:

  • In Zope 4, WebDav was made dependent on ZServer, by mistake?
  • Zope 5 assumes ZServer and WebDav no longer exist.

Remaining steps:

  • Get it upstream or not?
  • Eliminate dead code.
  • QA: tests, documentation
  • Release it.
  • Restore wsgi=off on Python 3.

I would be interested to hear when others are interested in using this. It is better if more people use this.

I am an introvert from Finland, so I did not ask the Zope people, or visited the Zope sprints.

Jens: There was no real reason to drop ZServer, except that no one has tried it because the code was so very old. Now you got it working, let's get this into Zope.

See the slides.

Paolo Perrotta: A Deep Learning Adventure

Posted by Maurits van Rees on October 25, 2019 08:14 AM

It is hard to see what part of machine learning is over hyped, and what part is actually useful.

Basis of ML (Machine Learning) is to get input and transform it to output. The ML gets a picture of a duck, and it gives as answer "duck". That is image recognition. You train the ML with images that you already label. And then you give it an image, and hope it gives a good answer. Same: from English to Japanese text. From an fMRI scan to an image visualized.

Simpler example: solar panel. Input: time of day, output: how much generated power. You would train this with data. The ML would turn this into a function that gives an approximately good answer.

Simplest model: linear regression. Turn the data into a function like this:

a * X + b

No line through the training points will be perfect. You find the line that minimizes the average error. So the ML uses linear regression to find a and b.

In our example, we try to guess the amount of mojitos we sell, based on the number of passengers that a daily boat brings to our beach bar.

But this may also depend on the temperature, or the number of sharks. With two variables you would not have a line, but a plane. With n, you would get an n-dimensional shape. You have n inputs, and give a weight to each input, and add them.

This is about numbers. For image recognition, you don't get a number as output. But we can apply a function to the result, and get a number between 0 and 1 that gives a likelyhood. For an image we may have 0.02 certainty that it is a cat, and 0.9 that it is a duck, so our answer will be the highest: a duck. Translated to the mojitos: what is the likelyhood that my bar breaks even?

This system with weights and a reducer function is called a perceptron. With a short Python program I got 90 percent accuracy on the standard NDIST test. We can do better.

We look at neural networks. This is basically: mash two perceptrons together. Or more. We are finding a function that approximates the data, and reduce the error iteratively.

What would I say is deep learning? Why has it grown?

  1. Neural networks with many layers...
  2. ... trained with a lot of data...
  3. ... with specialized architectures.

We helped Facebook with neural networks by tagging our photos. We gave them training data!

A lot of engineering is going into deep learning.

Generative Adversarial Networks: GANs. Example: a horse discriminator. Train the system with images of horses and others, and it should answer with: yes or no horse.

Other part: horse generator. Randomly generate images for feeding to the horse discriminator. You train this by saying: you did or did not manage to trick the discriminator. And you try to get better. I trained this a night, and after about a million iterations, I got pictures that are not quite horses, there is something wrong, but they are amazingly close.

Lightning talks Thursday

Posted by Maurits van Rees on October 24, 2019 09:29 PM

Michele Finelli: cappellacci

My second of three easy pieces on Ferrara.

Now about cappellacci, or caplaz in the local dialect. No, it is not tortelloni or tortellini, please.

Pasta filled with pumpkin is a tradition of manu parts or Northern Italy, but easily butter and grated cheese. With vegetables? Not if you want to avoid getting arrested.

And remember: spaghetti a la bolognese does not exist.

Jens Klein: yafowil, declarative forms

Yet Another Form Widget Library. It is now a drop in replacement for z3c.form, by activating the yafowil behavior per portal type.

For example usage, we have an example package. And documentation.

Federico Campoli: Postgres carbonara

Spaggheti carbonara, the PostgreSQL way. Lots of SQL code and demo.

Code is in a gist.

Eric Brehault: PLIPs

PLIPs are PLone Improvement Proposals. There are no PLIPs at the moment. A lot is happening in Volto, but that is outside core.

If you submit, should you do the work yourself? No! You can, but it is not needed. Just give ideas to people who can develop it. Please do not hesitate.

There is a PLIP about the Dublin Core metadata behavior. This is a meta behavior for four others. This may change. If you have a concern about this, please add your thoughts to the PLIP.

Asko Soukka: robot tests

Robot framework is a way to automate tests. You can combine this with Jupyter notebooks. It can help you write better tests, including auto completion. See https://robots-from-jupyter.github.io/public/

Sven Strack and Erico Andrei: Jekyll and Hyde

How to rank Plone events. So many years, so many pictures. How do you compare pants and pools?

There can only be one answer. Food!

  • Bronze medal: Awesome Tokyo, Plone conference 2018.
  • Runner up: Somni Català, Barcelona Plone conference 2017.
  • Honorable mention: The Plone Cake, Boston Plone Conference 2016.
  • The winner is: Red Turtle Tiramisu, Ferrara Plone conference 2019.

Paul Grünewald: Digital Signage and Plone

[Note from Maurits: I expected this to go about digital signatures, but it is about showing signs on monitors, for example to inform your visitors.]

University Dresden. Content types for monitors, slide sets and slides. Contents: text, images, timetables, fullscreen video, ticker. Editing WYSIWYG inline using CKEditor, preview, scheduling

Code: tud.addons.monitor

Sven Strack: Docs analytics

Margot Bloomstein: "If we don't know if our documentation is successful, how will we know what we need to do to improve it?"

For the Plone docs we use an overwatch dashboard. Alerts when files on docs.plone.org have not been updated in a year. Open issues and PRs. Accessitility, performance, speed. Graphana, Prometeus, Matomo.

Christine Baumgartner and Ilvy: Alpine City Strategic Sprint 2020

Side note: German symposium "Plone Tagung" March next year in Dresden. See https://plonetagung.de/2020/

11 tot 14 februari in Innsbruck, Austria, come sprint with us (and enjoy beautiful and snowy Austria):

https://alpinecity.tirol

Rob Gietema: Volto form editor

I said there wasn't a form editor. But actually there is a schema editor. Saved to JSON schema. No backend implementation, but maybe we can sprint on this.

Panel: Ask Me Anything on Volto

Posted by Maurits van Rees on October 24, 2019 03:05 PM

Is Volto compatible with Guillotina? Not 100 percent. Question is how we keep the APIs in sync. If people figure out a shared generic API that works with both Plone and Guillotina as backend, this can work.

How do you migrate? You can migrate from Plone 4.3 to Volto. We did that with transmogrifier. Biggest problem is moving composite pages, like cover, to the Volto blocks. In a post migration step, you can fix things up.

Why Semantic UI? Rob researched all the popular ones, concentrating on how easy it is to theme or override stuff. Semantic UI makes this easy. You derive from a theme, and only override some parts.

For add-ons, how do you keep the backend part in sync with the frontent part? You don't. You use Python packages in the backend and node packages in the frontend. There will probably be a lot of packages that are only in the backend or only in the frontend. You may need to be careful in the frontend so that it can work with several versions of the backend of this add-on.

TypeScript has won the JavaScript wars. Will you support that? We don't want to support two ways of doing the same thing. So if we switch to TypeScript, we want to stop using ES6, also in the documentation. Same for class based versus function based.

Did you check accessibility? Yes. We have a static code checker ALM that helped us fix issues. Also the Cyprus tool. But we also do manual checks. Plone Foundation is trying to get funding for an audit.

Alok Kumar: Gatsby Preview with Plone

Posted by Maurits van Rees on October 24, 2019 01:54 PM

Gatsby Preview gives you hot reloading of content. So you change something in Plone CMS, and it immediately is visible on your Gatsby site.

Plone CMS -> Web hook -> Gatsby source plugin. When content is created, edited, deleted, we will fire a websocket event.

When you change the title of a folder, you should update the breadcrumbs of the children too, and also navigation in several places. It took me a long time, but I have implemented this.

I have created a Gatsby theme for Plone: https://github.com/collective/gatsby-theme-plone

Maik Derstappen: State of Plone back-end development today

Posted by Maurits van Rees on October 24, 2019 01:33 PM

Frontend is nice, but it needs a backend, so let's talk about that. I will talk about plonecli, plone.api, Plone snippets for VS Code and plone.restapi.

You should use plonecli. It saves a lot of time for boring stuff. It helps you to create a product and enhance it step by step. It can cleanup and create a fresh virtualenv with a specific Python version and requirements and run buildout.

It has sub templates, for example to add a content type to your package. Now a sub template for a restapi service. Also one for upgrade steps, adding an upgrade profile per upgrade step.

We create a structure and files, and if you don't use some parts you can ignore or remove them. We will check if you have a clean git status before, and ask to commit, and afterwards we automatically commit.

You have good test coverage right from the start. All features added by plonecli have at least basic test coverage. You only write the tests for your own code. We make a tox environment to test different Plone and Python versions.

You can configure plonecli/mr.bob to your taste, changing the default answers or ignoring questions in a .mrbob file.

It is extensible, you can write your own custom bobtemplate packages and register them for the plonecli.

Visual Studio Code: snippets for Plone

There are snippets for schemas, fields, registry xml.

Use plone.api. It makes add-on code much easier to understand, without arcane incantations.

Ideas for the future.

  • plonecli:
    • add option to set Interface for view, viewlet.
    • REST API sub templates for de/serializer.
    • Graphical UI (ncurses-like) to make selecting options easier.
  • VS Code: plone.api autocompletion. VS Code is not so smart with namespaces. And it does not know buildout. You can use a buildout recipe that helps here though. Or use the generated zopepy script as your Python.

I would love more contributions. Don't be shy. There is no grand jury who makes decisions. Publish your own bobtemplates. Improve the VS Code snippets, or for other editors. Meet me at the sprints.

Question: is there a bobtemplate to create a new bobtemplate?

Answer: No. Should be possible.

Andreas Jung: Migrating a large university site to Plone 5.2

Posted by Maurits van Rees on October 24, 2019 12:50 PM

https://ugent.be is a Plone 4.3 site of a large Belgian university. Started in 2002 as Zope/CMF site. 90.000 pages, sub sites, 40.000 students, hundreds of editors, 90 add-ons. They wanted to move to Plone 5.2 and Python 3.

  • Traditional in-place migration: too manu add-ons, no one-on-one mapping possible.
  • Transmogrifier: not yet Python 3 at the time, too much magic hidden in too many places with blueprints.
  • So: custom migration solution.

Content types: standard types, plus four custom content types, including PloneFormGen. So that is quite reasonable. There is extensive usage of archetypes schema extenders.

Start: analyze and investigate your dependencies. - Based on Archetypes? Obsolete, replace. - No longer needed? Remove it. In Confluence we compiled a big table with for each package the basic information of how we would handle it: upgrade it, replace it, unknown yet, status of Python support.

Start with a minimal Plone 5.2 setup. Add one verified Python 3 compatible add-on at a time. Test extensively. Focus on content types first. Things like portlets can be handled later.

You need an export. We used a customized version of collective.jsonify. Core numbers: 90.000 json files, 55 GB data, 90 minutes, binary files base64 encoded.

We exported portlet assignments, default pages, layout information, workflow state, local roles, and pre-computed values for further efficient processing.

So we had 90.000 json files on the file system. We imported this in ArangoDB. Why use such a migration database? This allowed us to import only some portal types, or do parallel imports, and test complex migration steps like for PloneFormGen.

We briefly tried MongoDB, but that could not handle data over 16 MB. The json could be dumped unchanged in ArangoDB. This took 45 minutes.

Now we need to import this into Plone. Clean Python 3.7, Plone 5.2, plus the minimal set of packages needed for the content types. Import via plone.restapi. On top we have a dedicated migration package with special views. This handled things like translating between UIDs and paths.

The "magic" migration script is based on configuration in YAML.

  • Phase 1: pre-check the migration, remove target site if it already exists from previous test, create new Plone Site, install add-ons.
  • Phase 2: create all Folders, query ArangoDB for this.
  • Phase 3: create all non folders.
  • Phase 4: global actions. Check and migrate paths to UIDs in rich text fields. Assign portlets. Other specific fixup operations, like reindexing.

We migrated PloneFormGen (Archetypes) to easyform (dexterity). Export: one JSON for the FormFolder, one JSON file per field and action adapter. In easyform this needed to be turned into an EasyForm instance and a schema.

Topics (AT) to Collections (dexterity). Code was largely taken over from the plone.app.contenttypes migration.

From AT schema extenders to dexterity behaviors. First we made a list of which there were, are they in use, what do they do? Check which dexterity replacements there are. Create new behaviors.

Migrate packages to Python 3. This is mostly covered by talks of Philipp Bauer and David Glick. Common problems: utf-8 versus unicode, import fixes, implements to implementer. I rarely used the 2to3 and modernizr tools.

Some reimplementations: - portal skins to browser views - some packages with AT replaced with new packages with dexterity

Other problems:

  • improper file and image metadata
  • migration of vocabulary values, like old to new departments
  • repetitive cycles: always a bug occurs after a day of migration right before the end.

Quality control: - you need to check that migrated content and configuration is complete - "works for me" is nice, but others need to check too

Most of the packages have been removed, the setup is much smaller.

Status: - content migration is complete - must be tested in detail - integrate with the new theme, test this - need a replacement of a specific membrane usage - need work on castle/cas plugin

Takeaways: - Export Plone to JSOJN: 2 hours. Fast. - Import JSON to ArangoDB: 45 minutes. Fast. - Import ArangoDB to plone.restapi: 36-48 hours. Painfully slow. - 1.5 - 2.0 seconds per content object on average - cannot parellellize this import, because you would get conflict errors - So Plone and ZODB and painfully slow for creating mass content.

Question: in a similar setup we did a live migration from a live site to a separate new site. Did you consider that?

Answer: I tried this for other sites, but here I wanted to be able to partial imports, independent of the live site.

Question: default migration patches away all kinds of expensive indexing. You might want to consider looking at that. Migrating ten items per second is possible, although that is inline migration. And can you share the code, especially for the easyform migration?

Answer: could be done, but is not the primary focus of the budget currently.

A slightly older version of the code is on community.plone.org.

Plone Beethoven Sprint 2019

Posted by kitconcept GmbH on September 25, 2019 09:00 AM

beethoven sprint group image

21 developers from nine different countries gathered in Bonn, Germany between June 20th and 24th to work on implementing the upcoming Plone 6. The sprint at the office of the kitconcept GmbH has been declared a “strategic sprint” by the Plone framework team. Sprint topics included working on Volto, the Plone REST API, and Guillotina.

Volto

Volto is a new ReactJS-based frontend for Plone. It was started by Rob Gietema in 2017 and it is actively developed since then. Volto will become the default frontend for Plone 6. It implements a complete new user interface called Pastanaga UI, which was developed by Albert Casado.

Victor Fernandez de Alba and Rob Gietema led the efforts to further enhance Volto together with Paul Roeland, Nicola Zambello, Piero Nicolli, Janina Hard, Jakob Kahl, Thomas Kindermann, Maurizio Delmonte, Stefania Trabucchi, Rodrigo Ferreira de Souza, Fred van Dijk, and Steffen Lindner.

Pastanaga UI Toolbar

The new editing toolbar has been part of Volto right from the start. Though, so far we did not fully implement every detail that Albert has imagined for the Pastanaga UI toolbar. Victor has been working on this for quite a while already. During the sprint, he integrated his work and polished it. Victor and Rob also worked on a new sidebar that shows up on the right side of the Volto user interface that holds additional information like page metadata and controls the settings of the individual tiles.

volto toolbar The new Volto edit toolbar

New Tiles: Collection, Slider and Table

At the Plone conference in Tokyo, we came up with a list of tiles for the new Volto composite page editor that we would like to implement. On top of that list were the listing/collection tile, the slider tile and the table tile. Piero, Rodrigo, and Rob started to work on the listing/collection tile that is supposed to become a replacement for the Collection Type in Plone.

Jakob and Janina worked on copying over the code of a slider tile that was developed by kitconcept. Rob implemented the table tile. Before you could ask Rob about the progress, he already finished his work, like always…

volto table The new Volto table tile

Accessibility

Accessibility has always been a first class citizen in the Plone community. Plone is used by many government and public websites where accessibility is a basic requirement. Paul, Thomas, Timo, and Victor worked on improving the accessibility of Volto. Our goal is to fully support the WCAG 2.1 standard with the “AA” level of conformance.

Timo set up ESlint-based static code analysis checks and Cypress-based axe accessibility tests. Paul, Thomas and Victor then started to fix the reported accessibility issues. At the end of the sprint, we were able to fix all the reported violations. Our CI build now fails if new accessibility violations are committed.

Documentation

Sven Strack, the Plone documentation team leader, joined our sprint remotely from Amsterdam to discuss how to maintain and structure the Volto documentation in the future. We all agreed that we will use Markdown as format for our docs, since this is the de-facto standard in the JavaScript community as well as in most other Open Source communities. As much as we like Restructured Text, it seems that Markdown won that battle.

We currently use MKdocs for the Volto documentation and Styleguidist to document our React components. There are lots of different tools to choose from, Docz, MKDocs, MDX, Gatsby to just name a few. We agreed that we need to do more research to make an educated decision regarding our doc toolchain.

There will be a Google Season of Docs project this year where a technical writer will help us to enhance the Volto docs further. More Volto Features Nicola added a feature to display the currently used Volto version in the Plone control panel, he added animations for page transitions and added an Italian translation for Volto. Together with Victor, he worked on a toast component for user notifications.

More Volto Features

Piero worked on improving and fixing the event type view and added styling for the listing tile.

Victor and Steffen worked on the image sidebar.

Rodrigo added a feature that allows the user to use SHIFT+RETURN to create a new line in a Volto text. By default Volto creates a new text tile on RETURN. Rodrigo also looked into the Add-ons control panel work that Eric Steele started some time ago.

Nilesh joined us remotely from India and continued his work on the users and groups control panel for Volto.

Steffen fixed a bug in the subjects search. Rodrigo and Thomas looked into our Cypress-based acceptance tests.

Fred looked into what consequences it would have for us to get rid of the description tile and how we could customize the domain in Plone to allow to send portal emails via the Volto frontend.

Stefania and Maurizio worked on a product definition of Volto and an elevator pitch for clients and users. Together with Paul, Fred and Timo, they had a longer discussion about how to position Volto and Plone 6 in the market.

Plone REST API

The Plone REST API builds the foundation for the new Volto frontend and Plone 6. It has been under active development in the last four years and seen more than 80 releases in that time period. The REST API is stable, battle-tested and production-ready and part of the Plone core since the release of Plone 5.2.

Thomas Buchberger and Lukas Graf led the REST API efforts during the sprint and worked with Carsten Senger, Roel Bruggink, Janina Hard and Timo Stollenwerk to further enhance the REST API, fix bugs, improve the documentation, and the error handling.

API Team in the graden Plone REST API discussion in the garden

Querystring Endpoint for the Collection Type

One of the most important new features, that has been developed during the sprint, is the new querystring endpoint. This endpoint will allow us to implement the Collection type for the Volto frontend. Lukas implemented this essential feature with help from Rob, who took care of the frontend implementation in Volto.

New Features, Bugfixes, and first time contributions

Thomas added support for the retrieval of additional metadata fields in summaries in the same way as in search results. Lukas made the @types endpoint expandable and the @users endpoint easier to customize. Janina fixed setting the effective date on workflow transitions. This was her first contribution to the Plone core. On the same day she signed the contributor agreement, got her fix merged and released, which is quite an accomplishment for a single day. Way to go Janina!

In addition to that, the team fixed lots of smaller bugs (#780, #729, #764, #755, #777, #732, #738) during the sprint. REST API Error handling Carsten, Nathan, Thomas, Lukas and Timo had a longer discussion about error handling in the Plone REST API and about how we can improve things. We considered and discussed implementing RFC7807 and decided against adopting it.

Developer in office Sprint discussions…

The RFC did not see much adoption and it only slightly differs from what we are already doing. A breaking change does not seem to be worth the effort. We decided to create a separate error reporting component unifying the error reporting and ensuring consistency. We also started to fix a few inconsistencies right at the sprint.

API Documentation

Guillotina is using Swagger to document its API. Nathan explained the approach he took for that and Carsten started to integrate Swagger to generate dynamic api docs in plone.restapi.

REST API Releases

Most of the work we did on Plone REST API during the sprint has been released in Plone REST API 4.1.4, 4.2.0, 4.3.0, and 4.4.0.

Dexterity Site Root

Roel Bruggink joined us on Saturday and Sunday to continue his work on turning the Plone site root into a Dexterity content object. He fixed some complex recursion and acquisition problem when migrating the site root to Dexterity. His Plone Improvement Proposal (PLIP #2454) has been accepted by the Plone framework team and it will build the foundation for a new Plone 6 branch that we plan to cut this year.

Guillotina

Guillotina is a new AsyncIO REST data API. It has been written from the scratch by Ramon Navarro Bosch and Nathan van Gheem. It can easily scale billions of objects via its REST API. With the CMS addon, it is compatible with the Plone REST API. Therefore you can run Volto on a Guillotina backend and Guillotina might be able to replace our existing Plone backend in the future.

Guillotina Developer The Guillotina master thinking hard. :)

Nathan and Ramon worked on Guillotina 5. The Guillotina 5 release brought Python 3.7 support and the use of the context vars APIs. They refactored quite a bit, moved parts of addons into core(swagger, caching, pg catalog), implemented OOTB PostgreSQL catalog, added a Helm/Kubernetes/Docker configurations for the Guillotina CMS addon that can be configured to act as a Volto backend, worked on improving caching and filestorage.

Websockets for Plone

At the Plone Open Garden 2019, Asko Soukka started to work on bringing websocket support to Plone. Websockets allow bi-directional communication between the server and the browser. With websockets, the server can send messages and notifications to the browser. This can be used to implement notification systems or collaborative editing to just name a few possible applications.

Asko continued his work during the Beethoven sprint and created an example where comments that are added to Plone automatically pop up as toast notifications for other users. He published a Docker image with a Twisted-based ZServer and implemented the foundation to use that feature for gatsby-source-plone. This will allow us to create an extension that will allow editors to live-preview changes they do in Plone in a GatsbyJS site.

Developer in Office

Karaoke, Barbecue and Garden Parties

A Plone sprint is much more than just programming. Collaborating, meeting old friends, talk, discuss and having a good time together are the ingredients that make a Plone sprint. We started our sprint on Thursday with a free React and Volto training for newbies. At the evening when everybody arrived, we went to a local Karaoke bar for a revival of the epic Karaoke sessions we had in Tokyo last year.

Karaoke Karaoke at the “Nyx”

On Friday evening we ordered Pizza and hacked the night away at the office and our garden. On Saturday we went to the “Arithmeum”, a mathematics museum owned by the Research Institute for Discrete Mathematics at the University of Bonn.

Arithmeum Our tour at the “Arithmeum”

Afterwards we had a garden party with a barbecue and we spend the afternoon sitting in the garden and the office and continued to drink and hack on Plone.

Developer in garden Barbecue in the garden of the kitconcept office

Summary

The Beethoven Sprint 2019 was a major success. We were able to start working on the main missing pieces for Volto. The new edit toolbar is in place and the foundation for the new tiles/metadata toolbar on the right side of the UI has been finished. We also implemented a new table tile.

We implemented a proof-of-concept for the new listing / collection tile and implemented the required querystring endpoint in Plone REST API. Both Volto and the Plone REST API got lots of enhancements and bugfixes.

Both the Plone REST API and Volto are stable and used in production. We will use the next month to further enhance Volto and implement the missing bits and pieces. Our plan is to present Volto 4 at the Plone conference in Ferrara at the end of this year and cut a Plone 6 branch afterwards that contains the Dexterity Site root, folderish content types and other enhancements that are needed make Volto the default frontend for Plone 6.

Guillotina and the Websockets support for ZServer together with Gatsby as another frontend for Plone are amazing projects by the smartest folks in our community. Our diversity is one of our main strength and we are looking forward what the future will bring!

We had a great time at the sprint. Thank you to everyone who attended! We thank the Plone and Python community for their ongoing support. The Plone Foundation, the Python Software Foundation, and the German Python Software Verband made this event possible with their sponsorship as well as our company sponsors Abstract, Starzel, Werkbank and Zest.

We are looking forward to see you all in Bonn next year!

Developer in garden The “Hofgarten” near the kitconcept office