Planet Plone

This is where developers and integrators write about Plone, and is your best source for news and developments from the community.

August 26, 2015

Andreas Jung: Converting DITA to PDF using CSS Paged Media

2015-08-26T16:31:48Z

Alternative approaches for the PDF generation from DITA maps.

August 25, 2015

Reinout van Rees: Easy maintainance: script that prints out repair steps

by Reinout van Rees at 2015-08-25T18:34:00Z

At my work we have quite a number of different sites/apps. Sometimes it is just a regular django website. Sometimes django + celery. Sometimes it also has extra django management commands, running from cronjobs. Sometimes Redis is used. Sometimes there are a couple of servers working together....

Anyway, life is interesting if you're the one that people go to when something is (inexplicably) broken :-) What are the moving parts? What do you need to check? Running top to see if there's a stuck process running at 100% CPU. Or if something eats up all the memory. df -h to check for a disk that's full. Or looking at performance graphs in Zabbix. Checking our "sentry" instance for error messages. And so on.

You can solve the common problems that way. Restart a stuck server, clean up some files. But what about a website that depends on background jobs, run periodically from celery? If there are 10 similar processes stuck? Can you kill them all? Will they restart?

I had just such a problem a while ago. So I sat down with the developer. Three things came out of it.

  • I was told I could just kill the smaller processes. They can be re-run later. This means it is a good, loosely-coupled design: fine :-)

  • The README now has a section called "troubleshooting" with a couple of command line examples. For instance the specific celery command to purge a specific queue that's often troublesome.

    This is essential! I'm not going to remember that. There are too many different sites/apps to keep all those troubleshooting commands in my head.

  • A handy script (bin/repair) that prints out the commands that need to be executed to get everything right again. Re-running previously-killed jobs, for instance.

The script grew out of the joint debugging session. My colleague was telling me about the various types of jobs and celery/redis queues. And showing me redis commands that told me which jobs still needed executing. "Ok, so how do I then run those jobs? What should I type in?"

And I could check serveral directories to see which files were missing. Plus commands to re-create them. "So how am I going to remember this?"

In the end, I asked him if he could write a small program that did all the work we just did manually. Looking at the directories, looking at the redis queue, printing out the relevant commands?

Yes, that was possible. So a week ago, when the site broke down and the colleague was away on holiday, I could kill a few stuck processes, restart celery and run bin/repair. And copy/paste the suggested commands and execute them. Hurray!

So... make your sysadmin/devops/whatever happy and...

  • Provide a good README with troubleshooting info. Stuff like "you can always run bin/supervisorctl restart all without everything breaking. Or warnings not to do that but to instead do xyz.
  • Provide a script that prints out what needs doing to get everything OK again.

August 24, 2015

Reinout van Rees: Runs on python 3: checkoutmanager

by Reinout van Rees at 2015-08-24T12:45:00Z

Checkoutmanager is a five-year old tool that I still use daily. The idea? A simple ~/.checkoutmanager.cfg ini file that lists your checkouts/clones. Like this (only much longer):

[local]
vcs = git
basedir = ~/local/
checkouts =
    git@github.com:nens/syseggrecipe.git
    git@github.com:buildout/buildout.git
    git@github.com:reinout/reinout-media.git
    git@github.com:rvanlaar/djangorecipe.git
[svn]
vcs = svn
basedir = ~/svn/
checkouts =
    svn+ssh://reinout@svn.zope.org/repos/main/z3c.recipe.usercrontab/trunk

In the morning, I'll normally do a checkoutmanager up and it'll go through the list and do svn up, git pull, hg pull -u, depending on the version control system. Much better than going though a number of them by hand!

Regularly, I'll do checkoutmanager st to see if I've got something I still need to commit. If you just work on one project, no problem. But if you need to do quick fixes on several projects and perhaps also store your laptop's configuration in git... it is easy to forget something:

$ checkoutmanager st
/Users/reinout/vm/veertien/efcis-site
 M README.rst

And did you ever commit something but forgot to push it to the server? checkoutmanager out tells you if you did :-)

Porting to python 3. The repo was originally on bitbucket, but nowadays I keep having to look all over my screen, looking for buttons, to get anything done there. I'm just too used to github, it seems. So after merging a pull request I finally got down to moving it to github.

I also copied over the issues and added one that told me to make sure it runs on python 3, too. Why? Well, it is the good thing to do. And... we had a work meeting last week where we said that ideally we'd want to run everything on python 3.

Two years ago I started a django site with python 3. No real problems there. I had to fix two buildout recipes myself. And the python LDAP package didn't work, but I could work around it. And supervisord didn't run so I had to use the apt-get-installed global one. For the rest: fine.

Recently I got zest.releaser to work on python 3 (that is: someone else did most of the hard work, I helped getting the pull request properly merged :-) ). For that, several test dependencies needed to be fixed for python 3 (which, again, someone else did). Checkoutmanager had the same test dependencies, so getting the test machinery to run was just a matter of updating dependencies.

What had to be done?

  • print 'something' is now a function: print('something'). Boring work, but easy.

  • Some __future__ imports, mostly for the print function and unicode characters.

  • Oh, and setting up travis-ci.org testing. Very easy to get both python 2.7 and 3.4 testing your software that way. Otherwise you keep on switching back/forth between versions yourself.

    (There's also 'tox' you can use for local multi-python-version testing in case you really really need that all the time, I don't use it myself though.)

  • Some from six.moves import xyz to work around changed imports between 2 and 3. Easy peasy, just look at the list in the documentation.

  • It is now try... except SomeError as e instead of try... except SomeError, e. The new syntax already works in 2.7, so there's no problem there.

  • The one tricky part was that checkoutmanager uses doctests instead of "regular" tests. And getting string comparison/printing right on both python 2 and 3 is a pain. You need an ugly change like this one to get it working. Bah.

    But: most people don't use doctests, so they won't have this problem :-)

  • The full list of changes is in this pull request: https://github.com/reinout/checkoutmanager/pull/9 .

  • A handy resource is http://python3porting.com/problems.html . Many common problems are mentioned there. Including solution.

    Django's porting tips at https://docs.djangoproject.com/en/1.8/topics/python3/ are what I recommended to my colleagues as a useful initial guide on what to do. Sane, short advice.

Anyway... Another python 3 package! (And if I've written something that's still used but that hasn't been ported yet: feel free to bug me or to send a pull request!)

T. Kim Nguyen: The open horizon

2015-08-24T11:20:02Z

11 years of Plone at UW Oshkosh, and now a new start

2015-08-24T11:19:26Z

Plone description fields and PloneFormGen field help text are plain text, not rich text (HTML). Here's how to make URLs they contain clickable.

August 23, 2015

Davide Moro: Kotti CMS - frontend decoupled from backend. How we did it (part 2)

by davide moro at 2015-08-23T22:05:39Z



In the previous article http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html we have seen that:
  • decoupled public website from the private content management is cool
  • Python and Pyramid is cool
  • don't use PHP please :)
No we will see:
  • how to build a pure Python Kotti based setup with a private content management area decoupled from the public website (with tips, links, technical details and screenshots)
Here you can see some screenshots, implementation details and links.

Project setup

The installation folder is a package that contains all the application-specific settings, database configuration, which packages your project will need and where they lives.

From https://pypi.python.org/pypi/mr.developer.
The installation folder is a "one command install" meta package:
  • replicable
  • under version control + git flow
  • documented + change list for each plugin and the project itself
  • with a tag based deploy
  • based on Python pip/requirements.txt
  • batteries included
  • exceptions logging on file (with logrotate) enabled by default
  • automated deploy
so let the computer works for us and have fun.
See:

Populators

Populators are functions with no arguments that get called on system startup, they may then make automatic changes to the database like content initialization.

Populators are very important because when you install the project folder during development or on the very first production instance you'll find all the most important contents and sections by default. Things will be created automatically if the database is empty, so you don't obtain a blank site on the very first install.

Populators are also good for improving the first impact of the end users (I mean editors) with the platform because they see all the main sections already there.

See:

Private backend area

Turning Kotti CMS into a private content administration is quite easy:
Later I've created a generic package that does all the things for you (kotti_backend):
so things are even easier now (install kotti_backend, done).

Multilingual

kotti_multilingual is your friend.

Goto frontend link, translation management and link to the technical documentation online based on Sphinx
See:

Elastic search

kotti_es provides ElasticSearch integration for fulltext search. This plugin needs more love and and a complete refactor (it was built in a hurry and I'm not yet satisfied) but it proved there are no known issue after months of intensive usage.
Probably things will change, hope other guys with the same needs will contribute.

See:

Main navigation and header/footer links

You can use the kotti_actions plugin if you want to implement footer, header links or even nested main navigation menus. Obviously kotti_actions is ment to be used with a decoupled frontend.

As you can see a custom colour can be assigned to courses, navigation links, sections and every kind of object thanks to the json annotations column provided by default by Kotti. So you can add arbitrary fields.

How the multilevel menu looks like on the public website
See:

Portlets

The main layout based on box managers for portlets
The kotti_boxes is your friend. This plugin is ment to be used with a decoupled frontend. And it was quite quick implementing portlets because we didn't need to customize the private backend area.

You can define different page layouts for each resource type (home page, news, courses, etc) and show boxes in well defined areas (box managers), for example LeftBoxManager, AboveFooterBoxManager and so on.

So box and box managers are just non publishable contents and you can:
Banner portlets with links
  • copy/paste them
  • assign workflow with different security policies to box and box managers
  • assign different views
  • share edit permission to certain box or box managers to particular users or groups
  • prevent certain type of boxes to be included in some areas (for example: banner box image only addable to the left box manager).
See:

Editor toolbar

As you can see if you are logged in the frontend will show an editor toolbar with:
  • link to the backend version of the page
  • object information (state, type of object, etc)
Info and links to the backend, edit and folder contents

or see exactly the website as an anonymous user (very common customer request):

Anonymous view
You can also add more features, for example direct edit links for images or portlets or live edit features.

Talking about a pure Python solution, you might implement this feature with a Pyramid Tween (I hope I'll have enough spare time to do that. Anyone would want to contribute? We are very welcome, contact me!):

Course types (custom content types)

The course view with portlets and collapsable paragraphs
They are a sort of rich documents with an image attachment column and integrated with an external ecommerce site. When you add a course type there is an event that initializes automatically subobjects and the main image attachement by default, so less work for editors.

In addition all image content types and image attachments are addable or editable just by allowed users thank to custom roles and global or local permission sharing.

Collapsable paragraphs are implemented with custom content types not directly reachable on the frontend.

There are a lot of fields on this content type, so they are grouped together using fieldsets.
Editors can also create a base private course model and then copy and paste it when new courses should be added.

Sometimes you want to prevent the addability on the site root for particular object types, this way things will remain always tidy (why you should add a course on the very root of the site?).

See:

Windows and MySQL issues and tips

Kotti can be installed on Windows but I strongly suggest to adopt a Unix-like server with Postgresql instead of MySQLas storage layer: 

Tests

All the software is tested. Very happy with the py.test framework.

See:

Other third party Kotti plugins

I've used the following third party plugins that can be used on a standard Kotti environment:
See also the full list of available plugins:

Photoes and credits

http://www.mip.polimi.it/enAll the screenshots shown in this article are taken from the "MIP Politecnico di Milano's graduate school of business" website:
So the MIP's website backend is powered by Pylons/Pyramid and Kotti CMS, I'll write a non-technical case study soon. In the mean time many thanks to:
  • MIP
    • Simona Strepparola, Head of Communication
    • Gabriele Bedani, Microsoft Sysadmin
    • all the MIP staff
  • Truelab 
    • Riccardo Bonini, project manager
    • Ruben Barilani, web developer
    • Caio Ceccon, web developer (Faculty and Staff implementation). Python developer that joined us for only 5 days but enough to become good friends! He never touched a Pyramid/Pylons or Kotti application but he was able to be productive in a couple of days, this proves the Pyramid/Pylons developer friendliness
    • Davide Moro, it's me
    • Andrea Sironi, creative director

Results

You can consider Kotti as a very good, secure, flexible, battle tested and easy to approach solution for important customers.

All Kotti posts published by @davidemoro

Next steps

Reading this article you should find all the bricks you need if you want to implement a public website decoupled from its backend with Kotti.

Now I'm assembling the above bricks in order to provide an "easy to install" solution with the same pattern I've described here. This is my roadmap:
It can be considered a good starting point for:
  • CMS-ish application
  • a generic framework for building generic web apps or single page web applications (remember, Kotti is not only a CMS, it could be considered as a framework)
So stay tuned and if you like this work please consider to contribute with
  • code
  • testing
  • issue reporting
  • Github stars
  • spreading the word 
or why not sponsorships!

And if you want to know more about Kotti and you are attending +EuroPython Conference 2015 in Bilbao don't miss the Andreas Kaiser's talk "Standing on the Shoulders of Giants: The Kotti Web Application Framework". I'll join the sprint (remotely) at the end of EuroPython so see you on IRC (http://webchat.freenode.net/?channels=kotti). If you want to stay tuned follow https://twitter.com/KottiCMS.

Davide Moro: Introducing substancek. A Kotti project

by davide moro at 2015-08-23T22:03:10Z

Let me introduce
Let me introduce substancek, a Kotti (http://kotti.pylonsproject.org) project.

What it is substancek?

substancek is:
  • Kotti (web application framework) + decoupled admin interface
It is only an additional layer upon the following opinionated stack:
with the following motto:
"""(even) better development experience and complete frontend freedom"""
and introduces (or better promotes) the concept of private admin area (backend) decoupled from the public side (frontend) of your web applications built with Kotti.

In other words it is a set of technologies addressed under the substancek brand that let you extend Kotti in order to use it just as a private backend administration area for your application data.

So you are still using plain Kotti with an additional package (at least kotti_backend depending on what you need).

If you want to know more I've discussed here benefits and why frontend decoupled from the backend pattern. See http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html

      substancek name

      Tribute to:
      • substanced (http://www.substanced.net). Carefully designed, well documented Python web development framework based on Pyramid/ZODB. It shares the concept of management views decoupled from the retail views for each published resource
      • kotti (http://kotti.pylonsproject.org). The Kotti framework

      When substancek is for you

      Any project of any size (from micro to XXL) involving content management that needs:
      So if you project needs (or in future iterations) one or more:
      • complex security policies
      • workflows
      • hierarchical data support
      • or even intranet/extranet like collaboration areas
      you might consider even more substancek (kotti_backend + Kotti + Pyramid + SQLAlchemy).

      For example:
      • very small applications. For example a just one view public json endpoint for published news-like resources consumed by a third party app with an admin interface for editing contents
      • heavy Javascript based applications with modern frontend tools (eg: SPA + REST) with a decoupled admin interface
      • content management solutions
        • blog
        • ecommerce
        • intranets
        • large CMS-ish applications
      Note well: if you don't need workflows don't be scared because there is no overkill. You can use a one state workflow or no workflow at all for example. No hierarchical data? Use not nestable resources and so on. If it comes out later that you need them it will be quite easy converting your code.

      Alternatives

      You can use plain Kotti, without the substancek's kotti_backend setup. Or if you prefer noSQL try the excellent substanced (substanced + Pyramid + ZODB). Both solutions are lightweight, tested, well documented and easy to learn. Alternatively if you need really a minimal and unopinionated solution you might use plain Pyramid.

      Do you need something more? You might consider to use Plone (https://plone.org) as a framework.

      Anyway the good news is that Python is plenty of good options.

      substancek architecture details

      As already told you the private admin area (backend) and the rest of the application (frontend) are two complete different applications with different settings, different views and shared authentication.

      Assuming you are going to use PasteDeploy to run your application, let's consider the following configuration files setup:
      • backend-dev.ini, configures the private admin interface based on Kotti thanks to the kotti_backend plugin
      • frontend-dev.ini, configures your application you are developing (a full CMS frontend implementation or a microapp with just one retail view).
      • development.ini (optional), mount the backend-dev.ini and frontend-dev.ini applications in the same process (/admin for the private admin interface and / for your application). Alternatively you can run the frontend and backend using two processes waiting for requests on different ports and play with rewrite rules.

      backend-dev.ini

      [app:kotti]
      use = egg:kotti

      ...
      pyramid.includes =
      pyramid_debugtoolbar
      pyramid_tm
      kotti_backend.views.override_root_view


      kotti.configurators = kotti_tinymce.kotti_configure
      kotti_backend.kotti_configure

      kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml

      kotti_backend.goto_frontend = 1
      This is a normal Kotti setup with:
      • enabled the (optional) root view override for our admin interface. In other words the default root view will be @@contents instead of the standard Kotti's hello page (see pyramid.includes)
      • added kotti_backend to kotti.configurators
      • override the Kotti's default workflow with the one provided by kotti_backend (kotti.use_workflow). Playing with the additional pview permission you can decide your resources visibility on the frontend. See the workflow definition here https://github.com/Kotti/kotti_backend/blob/master/kotti_backend/workflows/simple_backend.zcml
      • enable a "Goto frontend" link (kotti_backend.goto_frontend) for easy switching from admin interface and frontend 
      See more options on the kotti_backend's README file:

      frontend-dev.ini

      [app:main]
      use = egg:Kotti
      ...

      kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml

      kotti.configurators =
      your_package.kotti_configure

      kotti.base_includes =
      kotti
      kotti.views
      On the frontend configuration file we share the same workflow in use on the admin interface (kotti.use_workflow).

      One of the most important configuration is the kotti.base_includes override: here we decide what will be loaded on our application. We omit all the Kotti views loaded by default in the standard setup and we load what we want to include where:
      • kotti, loads the kotti "core"
      • kotti.views (optional), load some view discriminators and utils defined by Kotti if you need them
      • your_plugin.your_includes, load your includes registering the views needed by your application
      The kotti.configurators typically auto includes your package and tell what should be included in your application (pyramid.includes). See the Kotti documentation for more info.

      In other words:
      "what is not loaded, it doesn't exist"
      so the final result is that there is nothing exposed on the frontend except what you decide to load and extreme control. You can register just only one view application or configure a complex setup for a CMS-like application: it's up to you registering only the views your application needs and no more. This way you can use completely different frontend frameworks, different versions of Javascript libraries, you have no css/js conflicts and no need to hide unneeded things and you decide which resources will be published on the frontend.

      See also another advance usage pattern "Using Kotti as a library" http://kotti.readthedocs.org/en/latest/developing/advanced/as-a-library.html

      development.ini

      # See http://pythonpaste.org/deploy/#paste-composite-factory
      [composite:main]
      use = egg:Paste#urlmap
      / = config:frontend-dev.ini
      /admin = config:backend-dev.ini


      [server:main]
      use = egg:waitress#main
      host = 127.0.0.1
      port = 5000
      The (optional) development.ini shows how to configure a composite application with different mount points. You can change /admin with /cms or /whateveryouwant depending on your needs.

      Examples

      You can checkout the https://github.com/substancek/substancek_cms_theme package if you want to see in action a (quite complex) example.

      I'm going to provide more and simpler examples (eg: a pretend micro application), see the roadmap.

      What are the substancek related packages

      Here you can see the whole substancek ecosystem:
      • kotti_backend, generic package that turns Kotti to a private admin area. This is the keystone for every substancek like project
      • CMS-like applications
        • substancek_cms (roadmap), a Kotti CMS distribution with an alternative frontend theme based on SASS, html/templates minification and assets optimization based on Yeoman tools. It will be based on the kotti_project experiment available here https://github.com/davidemoro/kotti_project
        • substancek_cms_theme, an example public side CMS implementation based on Kotti built with SASS and html/templates minification
        • kotti_actions, link actions backend implementation for header, footer and main navigation links
        • kotti_boxes, portlets and box backend implementation
        • kotti_es, elastic search support (to be refactored)
      • common utilities
        • pyramid_html_minifier, generic Pyramid package that introduces Chameleon templates minification with no overhead. Required by substancek_cms_theme.
        • build_commands, generic Python package that defines a set of initialization commands on install (eg: python setup.py npm/bower/gulp) for frontend stuff. Required by substancek_cms_theme.

      Who's using substancek technologies

      MIP - graduate school of business

      The MIP (Politecnico di Milano graduate school of business - www.mip.polimi.it/en) uses substancek technology inside for the private admin interface. This approach is so flexible that let you use Kotti as a private admin content management area and even implement your public views using other frameworks or non Python languages (for example PHP+Symfony2).

      See:

      substancek_cms_theme

      This is a work in progress opinionated CMS frontend implementation that reuses existing Kotti templates and logics.

      The look and feel is the same you get with a standard Kotti installation but it shows how to integrate and distribute a Python package integrated with a Yeoman setup (http://yeoman.io) that provides:
      • production vs development setup. In production mode will be used assets and templates from the dist
      • Chameleon templates minification with no overhead thanks to pyramid_html_minifier when you are in "production" mode
      • SASS
      • image/assets optimization
      • build commands for npm/bower/gulp initialization thanks to the generic package build_commands
      Once installed you'll see the admin interface visiting http://localhost:5000/cms.
      See the code here: https://github.com/substancek/substancek_cms_theme

      Next steps

      If you want to contribute there is a lot to do:
      • contributions to the Kotti core (for example @@contents pagination, REST, etc and new ideas)
      • creation of the substancek_cms package that puts together the substancek_cms_theme default theme plus the most useful third party Kotti plugins like news, events, etc with decoupled admin interface and Vagrant/Ansible automated setup based on kotti_project for easy installation/evaluation
      • creation of simple pretend packages, add more examples (microapp, blog)
      • a toolbar available on the frontend just of editor users for improved usability (the same shown in the MIP case history) implemented with a Pyramid tween
      • more work on substancek_cms_theme, implement advanced features shown in the MIP case history like link managers and portlets and refactor the Yeoman folder removing what is not strictly needed
      • scaffolding, help people create new projects with decoupled admin interface using Kotti as a framework
      • substancek dedicated github page
      Contributions, feedback or pings like "hey, I'm going to use Pyramid/Kotti for the my next project" will be very appreciated!

      Documentation

      All Kotti posts published by @davidemoro

      Twitter links

August 21, 2015

Plone.org: Announcing the Plone Conference 2016 selection process

2015-08-21T13:58:53Z

With Plone Conference 2015 drawing near, it is time to begin planning for our next conference in 2016

August 17, 2015

Plone.org: 2015 Plone Conference - Early Bird Registration Almost Over

2015-08-17T15:30:00Z

Early Bird registration expires at the end of the month.

August 15, 2015

Jazkarta Blog: Most Seriously Amazing

by Sally Kleinfeldt at 2015-08-15T02:33:27Z

Another one of our websites is in the running for a seriously amazing award. Literally. The North American Orchid Conservation Center and their Go Orchids site has been nominated to compete in the Smithsonian Summer Showdown, where the public votes on the “most seriously amazing” thing at the Smithsonian – and they have made it to Round 2! Click here if you’d like to add your vote to help them make it to the final round. You can vote once a day, now through Tuesday.

And hey, if you just can’t resist voting for the earth-sized Event Horizon Telescope instead, I won’t hold it against you. It’s all amazing stuff. But my heart lies with the exotic plants.

Go Orchids!

#siunderdogs


Tagged: #sishowdown, orchids, smithsonian

August 14, 2015

Benoît Suttor: How I made my wedding site

by Benoît Suttor at 2015-08-14T13:35:04Z

So I'll getting maried !

I decide to make a website for my wedding with a list of gift, my honneymoon, presentation of my witnesses and so on.

I was looking for a litlle CMS with a "list of gift" (an online shopping) which can be installed on cheap and reliable hosting (An it's when I loose Plone)

Pyramid vs Django

I started looking on Pyramid (because I'm a Plone/Zope dev). I thought Kotti, but I didn't find a way to make easily gift, and I thougt project looks cool, but it'was maybe a little young for my kind of requirements. I didn't find good solution on pyramid for a wedding list. 

Such as I have some exprience in Django, And in my daily work, we started intereset on Geonode for GIS project.

-> I started looking on Django !

Django CMS vs Mezzanine

Django CMS and Django CMS e-commerce plugin. But it seems this project is a almost dead ? Last commit on github make me septic.

With little search, I found Mezzanine and Cartridge. I try it and It seems perfect for my porject, So I choose it !

Hosting

My first choose was OVH, because it's very cheap (5€ / month). But with little search, it is almost impossible to create a complex Django site (by complex, I mean a "Mezzanine" Django site, and it's not very complex). I pursued my searching... And I found Webfaction. They have local pythons, postgres, 600Go data for 10 € / month. It looks perfect for me, except they do not manage domain name directly. So I host my wedding site on webfaction and my domain name on OVH.

Maybe I could made an heroku Django website, but I was little affraid about complexity.

 

Next step is to create an online shop with Kotti or with Pyramid !

Benoît Suttor: New recipe, collective.recipe.buildoutcache

by Benoît Suttor at 2015-08-14T13:33:31Z

This recipe generate a buildout-cache archive. We use pre-generated buildout-cache folder for speed up buildout duration.

Introduction

This recipe generate a buildout-cache archive. We use pre-generated buildout-cache folder for speed up buildout duration. The archive contains one single buildout-cache folder. In this folder, there are 2 folders:

  • eggs: contains all eggs use by your buildout except eggs which have to be compiled.
  • downloads: contains zip eggs which must be compiled (as AccessControl, lxml, Pillow, ZODB, ...)

Before starting a buildout, we download and extract buildout-cache and use it on our buildout. We add eggs-directory and download-cache parameters on buildout section like this:

[buildout]

eggs-directory = buildout-cache/eggs download-cache = buildout-cache/downloads

 

Use case

In our organization, we have a Jenkins server. We created a Jenkins job which generate buildout-cache.tar.gz2 every night and push it into a file server.

We also use Docker, our Dockerfiles download and untar buildout-cache before starting buildout, so creation of docker image became very faster !

 

How it works

Simply, you have to add an parts with this recipe on your buildout project.

Like this :

[buildout]

parts = ... makebuildoutcachetargz [makebuildoutcachetargz] recipe = collective.recipe.buildoutcache

You can use some parameters for changing name of your archive, use another working directory than ./tmp or use another buildout file than buildout.cfg for eggs downloads, See https://github.com/collective/collective.recipe.buildoutcache.

For recipe installation you can make this command line:

./bin/buildout install makebuildoutcachetargz

And start recipe script:

./bin/makebuildoutcachetargz

 

Conclusion

Use collective.recipe.buildoutcache and decrease time lost with your buildout ;)

August 12, 2015

Benoît Suttor: Probes into Plone and Zope

by Benoît Suttor at 2015-08-12T18:50:24Z

On all teams, we need to check if Plone turns well. We need some probe to be sure our Plone site runs well !

Introduction

On all teams, we need to check if Plone turns well. We need some probes to be sure our Plone site runs well !

With Jean-François Roche, we started to have a look on Products.ZNagios. This product allow you to have some probes from Zope, you can ask your instance (live):

  • Number of unresolved conflict on Zope
  • CPU usage
  • DB sizes
  • Memory percent
  • Uptime of Zope
  • ...

You can access to the probes with a thread which listen on Zope  on port 8888 (in this conf). You just have to add zope-conf-additional in your buildout like this:

[instance]
...
zope-conf-additional =
<product-config five.z2monitor>
  bind 0.0.0.0:8888
  </product-config>

If you want more information on this, you can see documentation of five.z2monitor package.

collective.monitor

I created this package for adding some probes into Plone. I created probes as Products.ZNagios. We used a zope interface for registering all probes (zc.z3monitor.interfaces.IZ3MonitorPlugin). In this package, I added these probes:

  • count users
  • count valid users (user logged during 3 last months)
  • check if smtp is set up
  • last login time of a user
  • last time a plone or zope object was modified

How use it

Adding collective.monitor in your buildout in eggs and zcml instance section

[instance]
...
eggs +=
...
collective.monitor
zcml +=
...
collective.monitor

And also adding zope-conf-additional as explain above.

After this little config, you can access to probes with different way

1. bin/instance

After starting instance (bin/instance fg) you can access to probes with

./bin/instance monitor dbinfo main
./bin/instance monitor objectcount
./bin/instance monitor stats./bin/instance monitor help

2. netcat

After starting instance (bin/instance fg) you can access to probes with

echo 'dbinfo main' | nc -i 1 127.0.0.1 8888

3. telnet

After starting instance (bin/instance fg) you can access to probes with

$ telnet 127.0.0.1 8888
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
last_modified_zope_object_time
2015/08/11 11:49:48.540729 GMT+2
Connection closed by foreign host.

Conclusion

With this package, you can make stats on your instance.

We use diamond to collect and put informations from probes on graphite.

It's very helpful for having state of our infrastructure.

 

August 09, 2015

David "Pigeonflight" Bain: Removing rogue members from your Plone site

by David Bain at 2015-08-09T13:13:00Z

I recently had one of my Plone sites get hit by a "join form attack". Basically a spam bot which started adding new members to my site (we don't use a captcha at the moment). I ended up with far more members in the site than members of the organization. I started deleting them manually using the Zope Management Interface > acl_users > source_users, but that became tedious quickly. So I wrote a

August 08, 2015

Manuel Reinhardt: The Mime Type Magic Show

by Manuel Reinhardt at 2015-08-08T12:40:14Z

"Articles of Association" seems like a perfectly fine description for a file object. However, trying to set this string as the description using a custom form failed for me in a Plone instance. After saving, the description returned an empty string. Other strings ("123", "My ridiculous test description", etc.) were saved just fined. After a while I could narrow it down to the string "Article". If the description started with this string and had at least one following character, we ended up with an empty string.

A descent into the depths of Archetypes and Plone core finally revealed the reason: No mime type was supplied for the description, so the MimetypesRegistry tried to guess it from the beginning of the string. There are a number of hard coded "magic numbers" in the magic module, which aren't actually always numbers but sometimes strings, including "Article". If the beginning of the description matched one of these strings, a mime type other than text/plain was guessed, and things went south from there. Without a match, "text/plain" was assumed and all was jolly.

The problem only surfaced after years in production. This is understandable as most of the "magic" strings would almost never appear at the beginning of a description string ("MM\x00\x2a", "<xbel", etc.). However, some of them very occasionally might happen to make it there ("Article", "Only in ", "import ", etc.).

The solution was to explicitly specify a mime type in the form with a hidden field.

<input type="hidden" name="description_text_format" value="text/plain" />

This is read by Archetypes and prevents any guessing.

To me, this problem also begs the question, "How smart should software try to be?". I'd love to see a piece of software that is smart enough to "understand" input without being explicitly programmed to do so. However, we're a long way from there, and often enough attempts at writing "magic" functions that try to be clever lead to unexpected problems that are a pain to debug. Maybe the problem is connected to the programming principles of Single Responsibility and Modularity. As a human developer I see the big picture and it's obvious to me that a description is always a plain text string. The MimetypesRegistry only looks at the string that is passed in, without any knowledge of what it is used for. If it saw (and understood) the big picture it might come to the same conclusion as the human developer. But as I said, we're a long way from there.