Planet Plone

This is where developers and integrators write about Plone, and is your best source for news and developments from the community.

April 25, 2015

Gil Forcada: 1st WPOD recap

by gforcada at 2015-04-25T20:45:11Z

Last Friday the first WPOD happened around the globe.

Here is what was done on the Berlin gathering:

Hopefully other participants around the globe will share their achievements as well!

Are you already planning the next WPOD? Mark it on your calendar May 29th.

Happy hacking!

April 24, 2015

Abstract Technology: Plone Open Garden 2015 - the right chance for discussing and planning

by Giacomo Spettoli at 2015-04-24T15:34:05Z

Reaching its 9th edition, the PLOG is growing and getting wiser. This was the year of the Strategic Summit with 5 days of talking, planning and preparing the future of Plone in the next 5 years.

Abstract Technology: Plone Open Garden 2015 - the right chance for discussing and planning

by Giacomo Spettoli at 2015-04-24T15:34:05Z

Reaching its 9th edition, the PLOG is growing and getting wiser. This was the year of the Strategic Summit with 5 days of talking, planning and preparing the future of Plone in the next 5 years.

April 20, 2015

eGenix: eGenix mxODBC Zope DA 2.2.1 GA

2015-04-20T08:00:00Z

Introduction

The eGenix mxODBC Zope DA allows you to easily connect your Plone CMS or Zope installation to just about any database backend on the market today, giving you the reliability of the commercially supported eGenix product mxODBC and the flexibility of the ODBC standard as middle-tier architecture.

The mxODBC Zope Database Adapter is highly portable, just like Zope itself and provides a high performance interface to all your ODBC data sources, using a single well-supported interface on Windows, Linux, Mac OS X, FreeBSD and other platforms.

This makes it ideal for deployment in ZEO Clusters and Zope hosting environments where stability and high performance are a top priority, establishing an excellent basis and scalable solution for your Plone CMS.

>>>   mxODBC Zope DA Product Page

News

The 2.2.1 release of our mxODBC Zope/Plone Database Adapter product is a patch level release of the popular ODBC database interface for Plone and Zope. It includes these enhancements and fixes:

Feature Updates:

  • Fully compatible with Zope External Methods and Zope's built-in transaction mechanism: use Connection Objects in Python modules set up as Zope External Methods to all the features of the mxODBC Zope DA from Python and even tap into the low-level features of mxODBC within Zope's transaction mechanism.
  • Full support of Stored Procedures: when using Zope External Methods, you have full access to SQL stored procedures, which allows direct integration with external database APIs.

Driver Compatibility Enhancements:

Fixes:

  • Documented use of allow_hosts in buildout.cfg

mxODBC Zope DA 2.2.0 was released on 2014-12-11. Please see the mxODBC Zope DA 2.2.0 release announcement for all the new features we have added.

The complete list of changes is available on the mxODBC Zope DA changelog page.

Upgrading

Users are encouraged to upgrade to this latest mxODBC Plone/Zope Database Adapter release to benefit from the new features and updated ODBC driver support. We have taken special care not to introduce backwards incompatible changes, making the upgrade experience as smooth as possible.

Customers who have purchased mxODBC Plone/Zope DA 2.2 licenses can continue to use their licenses with this patch level release.

For major and minor upgrade purchases, we will give out 20% discount coupons going from mxODBC Zope DA 1.x to 2.2 and 50% coupons for upgrades from mxODBC Zope DA 2.x to 2.2. After upgrade, use of the original license from which you upgraded is no longer permitted. Patch level upgrades (e.g. 2.2.0 to 2.2.1) are always free of charge.

Please contact the eGenix.com Sales Team with your existing license serials for details for an upgrade discount coupon.

If you want to try the new release before purchase, you can request 30-day evaluation licenses by visiting our web-site or writing to sales@egenix.com, stating your name (or the name of the company) and the number of eval licenses that you need.

Downloads

Please visit the eGenix mxODBC Zope DA product page for downloads, instructions on installation and documentation of the packages.

If you want to try the package, please jump straight to the download instructions.

Fully functional evaluation licenses for the mxODBC Zope DA are available free of charge.

Support

Commercial support for this product is available directly from eGenix.com.

Please see the support section of our website for details.

More Information

For more information on eGenix mxODBC Zope DA, licensing and download instructions, please write to sales@egenix.com.

Enjoy !

Marc-Andre Lemburg, eGenix.com

April 17, 2015

Gil Forcada: Testing pull requests and multi-repository changes

by gforcada at 2015-04-17T20:40:06Z

At Plone we use Continuous Integration (with Jenkins) to keep us aware of any change made on any of our +200 of packages that break the tests.

Thus making it feasible to spot where the problem was introduced, find the changes that were made and report back to the developer that made the changes to warn him/her about it.

A more elaborate step by step description is on our CI rules, be sure to read them!

At the same time though, we use GitHub pull requests to make code reviews easy and have a way to provide feedback and context to let everyone give their opinion/comment on changes that can be non-trivial.

Sadly, pull requests and multi-repository changes can not be tested directly with Jenkins, yes, there is a plugin for that, but our CI setup is a bit (note the emphasis) more complex than that…

Fortunately we came up with two solutions (it’s Plone at the end, we can not have only one solution :D)

Single pull requests

If you have a pull request on a core package that you want to test follow these steps:

  1. Get the pull request URL
  2. Go to http://jenkins.plone.org and login with your GitHub user
  3. Go to pull-request job: http://jenkins.plone.org/job/pull-request (you can see it always at the front page of jenkins.plone.org)
  4. Click on the Build with Parameters link on the left column
  5. Paste the pull request URL from #1 step
  6. Click on Build

Once it runs you will get an email with the result of the build. If everything is green you can add a comment on the pull request to let everyone know that tests pass.

Note: it’s your responsibility to run that job with your pull request and that changes made on other packages after tests started running can still make your pull request fail later on, so even if the pull-request job is green, be sure to keep an eye on the main jenkins jobs as soon as you merge your pull request.

Example: Remove Products.CMFDefault from Products.CMFPlone (by @tomgross)

Pull request: https://github.com/plone/Products.CMFPlone/pull/438

Jenkins job: http://jenkins.plone.org/job/pull-request/80

Multi-repository changes

When the changes, like massive renamings for example, are spread over more than one repository the approach taken before doesn’t work, as the pull-request Jenkins job is only able to change one single repository.

But we, the CI/testing team, have another ace on our sleeves: create a buildout configuration in the plips folder on buildout.coredev (branch 5.0) that lists all your repositories and which branch should be used, see some examples.

Once you have that cfg file, you can politely ask the CI team to create a Jenkins job for you. They will make a lot of clever things to make that work on jenkins (3 lines change plus following some instructions) and sooner or later a new Jenkins job will show up on the PLIPs tab on jenkins.plone.org.

Rinse and repeat!

Extra bonus and caveats

All Jenkins jobs, be it the pull-request, PLIPs and of course the core jobs, are configured to send an email to the one that triggered the job, so don’t worry about how long do they take to run, once they are finished you will get notified.

The caveat is that the above is only valid for changes targeting Plone 5. We didn’t put the extra effort to make it sure it also works for pull requests (either single or multi-repository) aimed at Plone 4.3. It’s quite trivial to add it for multi-repositories, a bit more work to make it run on single pull requests, still feasible to do if there’s enough people asking for it.

Hopefully the amount of pull requests for Plone 4.3 will decrease more and more as Plone 5 is getting closer and closer one pull request at a time :)

Now there’s no excuse on pushing changes to master without having tested them before on jenkins.plone.org!

Proposals on improvements and suggestions are always welcome on the issue tracker for jenkins.plone.org GitHub repository. Help on handling all those issues are, of course, also welcomed!

Happy testing!

eGenix: Python Meeting Düsseldorf - 2015-04-29

2015-04-17T08:00:00Z

The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.

Ankündigung

Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:

Mittwoch, 29.04.2015, 18:00 Uhr
Raum 1, 2.OG im Bürgerhaus Stadtteilzentrum Bilk
Düsseldorfer Arcaden, Bachstr. 145, 40217 Düsseldorf


Neuigkeiten

Bereits angemeldete Vorträge

Johannes Spielmann
       "Nachrichtenprotokolle in Python"

Matthias Endler
       "The State of PyPy"

Charlie Clark
       "Die Kunst des Nichtstun: Eine Einführung in Profiling"
       "et_xmlfile: Valides XML schreiben mit niedrigem Speicherbedarf"

Marc-Andre Lemburg
       "SSL in Python 2.7.9"
       "YouTube Feed mit feedparser auswerten"

Weitere Vorträge können gerne noch angemeldet werden. Bei Interesse, bitte unter info@pyddf.de melden.

Startzeit und Ort

Wir treffen uns um 18:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden.

Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.

Über dem Eingang steht ein großes “Schwimm’in Bilk” Logo. Hinter der Tür direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.

>>> Eingang in Google Street View

Einleitung

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.

Einen guten Überblick über die Vorträge bietet unser PyDDF YouTube-Kanal, auf dem wir Videos der Vorträge nach den Meetings veröffentlichen.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf:

Programm

Das Python Meeting Düsseldorf nutzt eine Mischung aus Open Space und Lightning Talks, wobei die Gewitter bei uns auch schon mal 20 Minuten dauern können :-)

Lightning Talks können vorher angemeldet werden, oder auch spontan während des Treffens eingebracht werden. Ein Beamer mit XGA Auflösung steht zur Verfügung. Folien bitte als PDF auf USB Stick mitbringen.

Lightning Talk Anmeldung bitte formlos per EMail an info@pyddf.de

Kostenbeteiligung

Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.

Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.

Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.

Anmeldung

Da wir nur für ca. 20 Personen Sitzplätze haben, möchten wir bitten, sich per EMail anzumelden. Damit wird keine Verpflichtung eingegangen. Es erleichtert uns allerdings die Planung.

Meeting Anmeldung bitte formlos per EMail an info@pyddf.de

Weitere Informationen

Weitere Informationen finden Sie auf der Webseite des Meetings:

              http://pyddf.de/

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

April 16, 2015

Mikko Ohtamaa: Inspecting thread dumps of hung Python processes and test runs

by Mikko Ohtamaa at 2015-04-16T20:39:24Z

Sometimes, moderately complex Python applications with several threads tend to hang on exit. The application refuses to quit and just idles there waiting for something. Often this is because if any of the Python threads are alive when the process tries to exit it will wait any alive thread to terminate, unless Thread.daemon is set to true.

In the past, it use to be little painful to figure out which thread and function causes the application to hang, but no longer! Since Python 3.3 CPython interpreter comes with a faulthandler module. faulthandler is a mechanism to tell the Python interpreter to dump the stack trace of every thread upon receiving an external UNIX signal.

Here is an example how to figure out why the unit test run, executed with pytest, does not exit cleanly. All tests finish, but the test suite refuses to quit.

First we run the tests and set a special environment variable PYTHONFAULTHANDLER telling CPython interpreter to activate the fault handler. This environment variable works regardless how your Python application is started (you run python command, you run a script directly, etc.)

PYTHONFAULTHANDLER=true py.test

And then the test suite has finished, printing out the last dot… but nothing happens despite our ferocious sipping of coffee.

dotdotdotmoredotsthenthenthedotsstopappearing
..

How to proceed:

Press CTRL-Z to suspend the current active process in UNIX shell.

Use the following command to send SIGABRT signal to the suspended process.

kill -SIGABRT %1

Voilá – you get the traceback. In this case, it instantly tells SQLAlchemy is waiting for something and most likely the database has deadlocked due to open conflicting transactions.

Fatal Python error: Aborted
Thread 0x0000000103538000 (most recent call first):
  File "/opt/local/Library/Fra%                                                                                                                                                                     meworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 154 in _eintr_retry
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 236 in serve_forever
  File "/Users/mikko/code/trees/pyramid_web20/pyramid_web20/tests/functional.py", line 40 in run
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 921 in _bootstrap_inner
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 889 in _bootstrap
Current thread 0x00007fff75128310 (most recent call first):
  File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/SQLAlchemy-1.0.0b5-py3.4-macosx-10.9-x86_64.egg/sqlalchemy/engine/default.py", line 442 in do_execute
...
  File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/SQLAlchemy-1.0.0b5-py3.4-macosx-10.9-x86_64.egg/sqlalchemy/sql/schema.py", line 3638 in drop_all
  File "/Users/mikko/code/trees/pyramid_web20/pyramid_web20/tests/conftest.py", line 124 in teardown
...
  File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/_pytest/config.py", line 41 in main
  File "/Users/mikko/code/trees/venv/bin/py.test", line 9 in <module>

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

April 13, 2015

Gil Forcada: WPOD

by gforcada at 2015-04-13T21:04:19Z

WPOD: World Plone Office Day

During this year’s PLOG I presented the simple idea behind WPOD:

  • every last Friday of the month
  • meet somewhere (remotely is fine as well, see below)
  • hack on Plone instead of your regular work
  • Rinse and repeat

That’s it, as simple and as easy as it can be.

Mark on your calendars every last Friday of the month, you have an appointment with the Plone community to bring Plone one step further ahead!

WPOD in Berlin

Preparations are being made for the first ever WPOD in Berlin that my company will gladly host. If you happen to be around Berlin, please contact me telling that you are coming!

The location is Hegelplatz 1, 10117 Berlin.

You are welcome during all day long. Plonistas are expected to come during the morning, enjoy some lunch together, and hack away until late afternoon.

WPOD around the world

If you happen to not be in Berlin, fear not, an irc channel will be available (#sprint on irc.freenode.net) so you can feel the same experience as in any other city hosting a WPOD.

Credit where credit’s due

That’s not an original idea of mine, nor is something that I thought of myself alone, Philip Bauer already tried to present the very same idea on last year’s Plone Conference in Bristol.

Later on, during the Bycicle sprint in Berlin, Stefania and I discussed about it and defined the format as it will start with.

Thanks to them for their bright minds and clever ideas!

Future

Within 10 days the first WPOD will happen, Plonistas will hack/plan/design away and Plone will get better and better.

I hope that other cities and individuals alike will start participating on WPOD, the more we are the bolder the change we will make.

There are some plans to put all the relevant information regarding WPOD on plone.org, either on the current website, or even better on the newer plone.org that is on the making (watch here for tickets ready to be fixed by any of you!).

Happy hacking!

Update: a meetup has been created, please RSVP there.

April 10, 2015

Davide Moro: kotti_multilingual

by davide moro at 2015-04-10T22:41:05Z

kotti_multilingual is a package still in an early stage of development that adds to the Kotti CMS (http://kotti.pylonsproject.org/) multilingual capabilities. It is neither feature complete nor can be considered API stable: so things will change!

You can find various fixes on this fork (a PR is still in the review phase):

How it works

First of all you should add a LanguageRoot folder in your root site. It is like the standard folderish Document, but with an editable Language attribute where you set the language code (eg: en, it, etc).

Once you have created two or more language folders (with >2 languages there is a problem with the translation link actions at this time of writing) you can add your contents and translate them.

The translate menu prompts a translate into italian action from  /en/welcome
If you click on the translate into action, it will create a translated instance in /it/welcome (you can rename it later in /it/benvenuto or whatever you like) and you'll be redirected to a regular edit form prefilled with the english fields values.

Once saved, you can switch to the existing translation and navigate among languages as shown in the following picture:

You can switch to the existing English translation
kotti_multilingual supports the quite advanced concept of language independent fields: a field whose values should be inherited by translations, only editable on the root translation.

You can see for example a select widget in edit mode on the root translation:
And the same field in readonly mode on the translated object:

See the kotti_multilingual.widget.i10n_widget_factory code for more info.

Code examples

And now code examples.

resources.py

You can define language independent fields in your type_info attribute on your resource.
class YourResource(...):

    ...

    type_info = Document.type_info.copy(
        ...
        language_independent_fields= ['course_sku',],
        )

views/edit.py

The edit form does not require changes, you just need to apply the i10n_widget_factory on your language independent fields (in some particular cases you need a bit more complex setup when you have to deal with not null column, required fields, etc). In these particular cases you'll have to play with get_source (kotti_multilingual.api.source) and put the widget in readonly mode. If you experience problems cloning select widgets you might have to migrate to deferred widgets (that creates a new widget instance each time) and set the widget mode in readonly when needed.

from kotti_multilingual.widget import i10n_widget_factory
...
from kotti_multilingual.api import get_source 
@colander.deferred
def deferred_widget(node, kw):
    request = kw['request']
    context = request.context

    ...

    widget = SelectWidget(values=available_tags, multiple=True)
    if get_source(context) is not None:
        widget.readonly = True
    return widget


class YourResourceSchema(colander.Schema):

    course_sku = colander.SchemaNode(
        colander.String(),
        title=_(u"Course SKU"),
        missing=u"",
        widget=i10n_widget_factory(TextInputWidget),
        )


class YourResourceAddForm(ImageAddForm):
    schema_factory = YourResourceSchema
    ...

    def get_bind_data(self):
        bind_data = super(YourResourceAddForm, self).get_bind_data()
        # we tell to the i10n_widget_factory that this is an addform,
        # so our widgets will be shown as usual in edit mode
        bind_data['addform'] = True
        return bind_data

Final thoughts

Yes, it is a very very young package but very promising!
It is not complete and probably it never will be complete because SQLAlchemy is huge and I think it is not possible to cover all the possible SQLAlchemy combinations.

For example this fork includes support for the SQLAlchemy's association_proxy feature and language independent fields (in this case the copy_properties_blacklist attribute on your resource is your friend).

This is open source, dude: if you need something that is not yet covered, just fork kotti_multilingual, implement the missing parts and share with others!

Maurits van Rees: PLOG Friday evening report-out

by Maurits van Rees at 2015-04-10T17:55:22Z

Report-out of today

  • Some cleanup tasks for the future identified, fifteen to twenty. Made list of benefits connected to each task. Like: pep8, remove skins usage from core Plone. Risks. Probably setup web page for people or companies to donate money to spend on cleanup, bit like Plone Intranet Consortium. Workers that work for, for example, half the normal wage. Because these are mostly boring tasks, that no one really wants to pick up, not very rewarding of its own. Individual, or group who organises itself. Sponsor can pay you as normal. Do not set up a big organisation. Trust each other.
  • Through the web customization, without being horrible. Looked at collective.jbot. Some security issues that need to be considered. That package needs Plone 5 support. ACE editor is useful. Resource folder to add. jbot folder in your diazo theme. Advertise as add-on, not in core, as the approved way to allow these kinds of hacks, with round-tripping. Maybe move to core in the end.
  • Increase number of Asian Canadians in Plone. :-) So: diversity meeting. Some are under represented. White males dominate the Plone scene, or at least at the PLOG. But there are some families here, which is very good. Non-native English speakers should feel welcome. At future events, we ask you to look at for newcomers. Connect new people with old-timers. Prioritize first-time speakers at events, help and encourage them. Expand range of talks, like how to run a small business, be a free-lancer. Simple things to make ourselves more attractive to new people.
  • Documentation. Explained how we do it.
  • Trainings. Three people are willing to give a training at the next conference. Fulvio, Fred, Philip. Beginner, integrator, developer. Master class maybe? Training the trainers. Enable new trainers, tips and tricks, how to set it up and market it. So: we are going to have a Plone Training Team, with me as lead. Increase visibility when people give trainings. Monthly hangouts.
  • Translations. Fixed lots of i18n issues. You can start to translate now for Plone 5. We need help figuring out how to extract messages from Javascript.
  • Communication with the community. Collection of activity to get into newsletter. Get teams to report regularly and consistently, also about help they may need. Teams should fill out a document about themselves on plone.org. New information in newsletter. Job openings. Recent launches. Contact Christina. Sponsorship. Social media plan, record upcoming events in a calendar. We like to avoid twisting your arm for information that should be shared with the outside world.
  • Mosaic is working in Plone 5. Want to do a release this weekend. Alpha release. Various things that do not work yet, but at least you can try it. It changes various things in the rendering engine, like with tiles. Philip: I was mind blown again and again when seeing it today, lost for words.
  • Release team. Commit to doing bugfix releases monthly. Let other people release packages. Write nicer combined changelog for each release, more welcoming, more exciting for new developers.
  • Plone RESTapi. Created package to use http verbs in the ZPublisher, delegating to browser views. plone.restapi can build on top of that.

General report-out of this week

  • Cleaning up bits of the backend, like portal skins, tools, and also simply pep8.
  • RESTapi, preparation for frontend.

A bit scary at the beginning of the week, complaining about what does not work, or all the big work that still needs to be done. But there is a plan for the long term future, with sane steps for short and middle term. So rough roadmap is possible and totally compelling. More energy for people who contribute. We can be brave for the next five years, to see a brighter future.

Big cheer for everybody!

Tomorrow: overflow day. You can have more discussions. Or visit Sorrento or other bits of the surroundings. Paul will start compiling some documents in a kind of roadmap, and people are invited to help. Open space about Plone booth at Bilbao. Plone RESTapi.

Maurizio: the board really helped this year, with declaring this a Strategic Summit, and helping in organizing. Thank you all!

http://zope.maurits.vanrees.org/mvr/weblog/images/standup-plog-2015

[Image by Fred van Dijk.]

by Maurits van Rees at 2015-04-10T09:48:01Z

On the Trello board there is now a column for teams. Feel free to edit it. And add yourself to a team if you feel lonely and want to join the fun. :-)

Report-outs from yesterday:

  • Javascript / RESTful api. First needed is the RESTful api. Afterwards Javascript front-end. Talked about technical challenges for REST, like supporting http verbs in the ZPublisher without defaulting to webdav, Ramon actually has a working prototype for this, too good to be true. Roadmap can be found on https://github.com/plone/plone.restapi/milestones. It will only work for dexterity. How can we make it happen. We can discuss that here and online. Second session was on Javascript front-end. Brainstorming about what each Javascript framework actually does. If you compare things, they should be similar, which they really are not. We do not have to make a future decision now. We currently have patternslib, and mockup on top of that. Patternslib may help you to not have to write Javascript, but you should really learn Javascript anyway if you want to be a web developer. In half a year there will be even more Javascript frameworks, we can decide later. Sorry, there was no pillow fight. We have smart people who can implement a solution. We still have a solid, integrated stack. Outsiders who may want to build on top of Plone will not look deeply into out stack, at how difficult it may be: when is the last time you really dived into the SOLR code? You simply use it, without worrying too much, so we should not worry to much.
  • Plone Intranet. Reviewing where we are and how to move forward to ship with a sane architecture. We decided to organize four one-week sprints in the next ten weeks. One online, others in Bristol, Berlin, Arnhem.
  • Hackable Plone, Through The Web Plone (TTW). The need for hackability (tinkering with it as power user) was widely supported. Finishing up Mosaic is an important, maybe vital task. Through the web editable .po files, to change jargon. [Look at prototype in collective.storedtranslations, Maurits.] Editable labels per content type, even in the same dexterity behavior; would be really cool, but we have no idea if that is easy or difficult. Repeatable export and import of content and configuration, in the Plone UI please; per add-on exportable config would be great, but might be difficult. Making view, viewlets, theme snippets through the web; make that be seen as a normal supported practice, instead of being frowned upon. Through the web installable add-ons? collective.jbot allows through the web jbot. Some few but loud people are against jbot, but most seem to think it is a fine method for overriding. portal_view_customizations can be tricky, in some cases breaking things simply by customizing it without changes. Being able to do a similar and less breakable thing through the Plone UI would be good. Undo or better versioning of such changes is helpful.
  • Branding, positioning. Micro managing has not worked. Various regional and cultural differences that do not work. See Fred's talk of yesterday. We could work towards a common brand view. Get help from community and also from outside. We know someone who uses Plone and does Jungian Archetypes for a living, he was actually here at PLOG last year, so maybe we can ask him to help. plone.com discussions, a bit too much American, enterprise has different connotations. Whatever brand identity we have, does not necessarily influence how individual companies market themselves. Press releases on time can be hard with volunteers, we may explore other ways. Agreement among US companies to hire marketing people. Not a finished discussion. Board and current marketing team will pick this up, and be sure to ask others for their input.

Gil: Berlin Sprint

With Stefania we plan to organize a sprint in Berlin (not the one Guido mentioned above). Working on Mosaic. The coming weeks we will send a formal announcement. Middle of September this year.

Fred: Six Thinking Hats

See presentation at Trello.

This is about Six Thinking Hats, by Edward de Bono. Edward has thought a lot about thinking. Gotta love the meta approach. Creative, lateral, structured thinking, brainstorming, etcetera. Some say it is pseudo science. See for yourself.

Meetings can be discussions, about arguments, ending in 'yes, but...'. A suggestion, followed by risk, idea, emotion, fact, cause, effect, all through each other. Familiar? We have seen it. You cannot compare a fact with an emotion. Six Thinking Hats is about parallel thinking. One thing at the same time. First all think about ideas, without facts or emotions. Do not feed the egos. No showing off.

So what are those hats?

  • White: facts, numbers, what do or don't we know, which questions need asking, pretend you are a computer. Examples of data, research. No argumentation, no judgement. Need research, then store it for later. Somebody's facts can be another person's opinion; can you explain, based on your experience why something is 'for a fact' wrong?
  • Red: emotion, but feeling. Fire, warmth, intuition. We don't know why we don't agree, but we just don't. You don't have to prove what you are feeling, argument about it. Emotions are there, the red hat just let's them surface.
  • Black: risk, disadvantage. Critical thinking. Cautions. Being critical is easy, basic survival instinct to avoid getting eaten by a lion, you will not argue with it. It is important. Sit down with six optimists and pick a new framework... not so good.
  • Yellow: optimism, benefits. Look at the bright side. How can we do this. How will the future then look in one year. Proposals, suggestions, constructive thinking. No arguments, but do ground it with facts. Best case scenario.
  • Green: creative solutions. Edward de Bono has written twenty other books about this. Get new ideas. Brainstorming, no judgement. Thought experiments. Postpone judgement. Come up with at least five different alternatives, like five Javascript frameworks. Reverse the proposal to come up with a new proposal. Provocation: choose a random word (banana) and associate with the current proposal in mind.
  • Blue: control, meta. Blue sky. Overview. Helicopter view. Think about thinking. Meta. Organise the meeting. Which hat order do you start with in this meeting? Role of the chairman probably, or some other dedicated blue person. Observation.

Deal or no deal: if you wear the hat, stick to that thinking direction. Everybody wears the same hat at the same time. Do not say "you are too emotional", but say "you are wearing the red hat". It comes across as less hostile.

How do you use it? You can use it alone or in a group. Start without hats, but then separate the hats when you are stuck: you do not have to use it all the time. Limit the time per hat.

Why use it? It makes things easier. No mix-up of emotions. Think outside your own comfort zone: you may naturally be more black or more yellow. And of course shorter meetings.

The group leader watches the process and decides that people should now put on a specific color.

White hat: what about facts presented as opinions. Use "E-prime": English without the verb "to be". So not "no one is reading news letters", but "in my experience, this does not happen." Start with yourself.

Let's try it! Now!

First try: Let's say Plone wants a new logo. There is a proposal. Discuss it now. Proposal is four squares of blue, yellow, red, green...

Second try: we support Python 3 at the end of 2016.

[Can't summarize these tries, but it was fun and interesting to do.]

Alexander Loechel: Patterns for working Project Teams

It project management, team motivation. Novel from Tom DeMarco: Deadline. In a fictional way he describes software projects and what can go wrong. Other books: Peopleware - Productive Projects and Teams. And Adrenaline Junkies - Template Zombies.

Conclusion on why so many IT projects fail: the major problems of our work are not so much technological as sociological in nature.

He makes lots of points, with patterns and anti-patterns.

My personal conclusion: Plone community intuitively does most of his points right. Keep calm and carry on.

Eric Steele: Hey, when is Plone version X going to be released?

I get this question all the time. It mostly takes so long because I am busy releasing Plone...

Check Jenkins, auto-checkouts, check changelog, etc. By the time I am through the long list of checks for the long list of to-be-released packages, the list has grown by at least ten...

By 2020, Plone will dominate PyPI with over 99 percent of the packages being for Plone, and our cyborgs will take over the world.

Nathan: Security Team

Some of the core people are on it. There is some fatigue on the team, because it is a lot of work when there really is a problem. If your company can help, that would be cool and smart. We need someone who knows Plone really well.

April 09, 2015

Maurits van Rees: PLOG Thursday RESTapi current status

by Maurits van Rees at 2015-04-09T16:01:44Z

RESTapi current status

Timo started with some proof of concept implementations. See https://github.com/plone/plone.restapi

If it would not work with for example the ZPublisher, then that would be bad, so we should look into that. Let it support http verbs, like POST, GET, PUT, DELETE, instead of assuming it is a webdav request when it is not POST or GET.

Aren't people moving away from that, just using GET parameters? Staying close to REST seems best, Angular and other frameworks can handle it. Workflow actions will use POST anyway.

You will always transform stuff between the saved data and presented data, like saving a uuid and presenting a normal url. You save something and then may get back an object with different values.

Several levels of RESTfulness.

  1. Resources
  2. RPC calls
  3. HTTP verbs
  4. hypermedia

If we only go for the second level, we could just use the json api. We should play around with the third level, to see if we can make it work.

There is a risk that we break webdav when we fix the ZPublisher. We may have to do that. Webdav works according to some, is buggy for others, or not working at all. For webdav you could look at Accept headers, or discover webdav in some way like that.

Take a look at the dexterity transmogrify code and see if we can take some export code from that. Also https://github.com/collective/plone.jsonapi.core. And look at json schema.

We thought about authentication, but the first phase is just about reading. In a web browser the current authentication will work fine. For non browser visits we need something else, but that can be done later.

The edit schema may differ from the add schema or the view schema. David Glick has written code in javascript for creating a form based on such a schema, using ReactJS and ReactForms.

So we may not need z3c.form then. But z3c.form also does data transformation and validation, you would still need that. If your schema is defined in json, you could use some json schema handling and validation in the backend as well. That is long term.

If you GET a page, you want a json with all the data you might want to see there, so title and fields of this object, list of items if it is a folder, portlets.

Timo: I have limited time to work on this. I do have a project where I am using it. Good if it can work on Plone 4.3 for this. But if it would only work on Plone 5 that would not be a deal breaker.

Hypermedia is there: you can click through the site with json. The json exposes other urls that you could click.

There is a live demo linked on the github page: https://github.com/plone/plone.restapi. You can install a Mozilla json plugin to look at it.

If companies would be willing to give developers money or time for this, that could be helpful. Maybe there is appetite to pool resources. The API design needs to be done before we can ask someone to really dive in and commit. It could feel strange that one person gets paid and others work on it for free, although I wouldn't mind, for me it is a lack of time.

Javascript front-end

Good to get some options out there, get understanding with more people about what we are actually talking about, so if we make a decision it is more informed, and knowingly agreed upon by more people. What are limitations of Angular, React, Patternslib, etcetera? What do we expect from a javascript front-end.

Plone Intranet is using Patternslib and it will be around in 2020.

People will build multiple javascript front-ends for Plone, with whatever framework they like.

Can we come up with a matrix of several frameworks in the next session?

[Well, we tried, but your note taker gave up.]

Maurits van Rees: PLOG Thursday morning talks

by Maurits van Rees at 2015-04-09T11:21:08Z

Report-out from yesterday afternoon sessions:

  • ZCatalog: ripping it out? Setup performance test first, to get baseline understanding of where issues are. Make ZCatalog more pluggable. Enhance visibility of collective.solr in our documentation, good for larger deployments, improve its documentation.
  • New plone.org work: halfway finished, most content that we want to keep can be transmogrified, css and responsive tweaks needed, personal profile pages with metrics to show what you are doing as member of the Plone community. PloneSoftwareCenter for the add-ons will be lost. Perfection is the enemy, it would never be finished then. Use collective.roster and collective.workspace. Plone 4. Started a working group to manage this and see it finished: Víctor, Gil, Christina, Fulvio and me, meeting every two weeks. Some of you may be called upon to help. Give us a shout if for example you know about LDAP.
  • Marketing/positioning: listed some competitors, talking about sectors, different ways we use Plone. Plone is obviously a very good CMS, but also a platform, a set of tools, that can be used to serve various targeted needs. Unique set of features. Discussion will continue this afternoon. [Fred will present some ideas / pep talk this morning.]

Víctor Fernández de Alba: Activity stream and conversation engine (MAX)

Víctor, Carles, Ramon. We have built a WhatsApp like application on Pyramid. Website. Push notification to app on phone.

First commit in August 2011. Initially designed as key feature for BarcelonaTech university concept of social intranet. But limited resourced for this. We did it bit by bit.

Activity stream with the basic key concepts of activity, comments, likes, favorites, upload images and files.

Conversations: one on one, in group, also with images and files. Doing this realtime using STOMP RabbitMQ plugin.

Push notifications to IOS and Android, including apps for those two.

Aggregate external sources: Twitter.

Authentication is done with oAuth 2.0. Resource Owner Password Credentials workflow using a server we created: osiris. See https://pypi.python.org/pypi/osiris

Activity stream stores activity from users and applications. Application can 'impersonate' a user to feed the stream with useful information, for example when uploading a file somewhere. MongoDB for storing info.

Subscriptions are made against contexts, something with a unique URL. Everything aggregated on the timeline of the user.

Realtime conversations and private messaging, going to RabbitMQ, then to conversation queues, then via an API (WSGI client) to MongoDB. Support for sending images and files too.

Infrastructure. Front-ends: MAXUI.js on IOS and Android. oAuth server. API. MongoDB plus RabbitMQ. Queues and consumers designed for huge loads.

API has 88 RESTful endpoints. Over 600 tests. Powered by Pyramid, optimized for gevent.

Performance: 4000 concurrent users: 100 messages per second.

LDAP integration. Deploying it to Moodle.

What's next: follow people, contextless activity streams. Documentation not only in Catalan.

Resources:

Please give feedback. Maybe you want to use it? If we get enough interest, we can convince our managers to let us go to EuroPython. ;-) We would like to get traction to continue to develop this.

Alexander Pilz: Design First Driven Process

The Plone Intranet Consortium is using this. Let's explain how we look at it, how we define it. Give some insight.

Why is design first necessary?

Part 1: the backend

The developer's fun is the integrator's nightmare.

New UI with Plone as backend: you end up with two separate projects: a backend project (Python) and a frontend project (css and javascript).

The Traveling Integrator Problem. Plone alone works fine. Each add-on you add increases the complexity. The more add-ons I install, the more I need to redo the user interface: one adds an action in a viewlet, another in a portlet, another in a content menu dropdown, so the end user does not know how to use them all.

People do not want to work with an intranet, they have to.

Design is not something on top of add-ons and Plone. It is something that should be integrated in the whole. For that to work, you need to start with the design.

In the Intranet Consortium we started with six companies who were using eight workspace implementations.

Part 2

  • Platform approach: an add-on that is as flexible as possible, the least assumptions.
  • Product approach.
  • Developers approach: first build it, then make it nice.
  • Designers approach: make it nice, then develop for it. Emphasis put on the end user.

"Design First" process. Product owner and designer (and only optionally a developer) sit and detail the requirements. You need a designer who can do html and css. A patterns library so that designers can add UI behavior, interaction design, even though nothing yet happens on the backend. You directly use the actual design as the diazo theme.

It is hard to keep developers from designing. They actually like doing that a bit. You have to hit them on the fingers: don't make design decisions yourself, talk with the designer.

Visual versus interaction design. Designers are not just there to make things nice. Theme is the visual design. But there is more: interaction. Usability, user testing. Theming is okay for websites. For intranets there is much less need for branding: you do not need to be reminded for which company you work.

Fred: Branding

Framing effect: what you see, influences what you think later.

I am not a marketeer or communication specialist (anymore), so I shouldn't give this talk. This is just an intro, not a lecture.

You can look at sectors, audiences, regional, etcetera, but it gets complicated. Maybe we have to agree to disagree. Also the wording, top down: marketing, strategy, branding, positioning, campaigning, advertising, communicating, copy writing. Look up a bit. Get out of our comfort zone.

Brands give you emotions and create an identity. They frame you. You see someone and you have an immediate idea: is this person nice or not.

Selling cars. How? They can try to sell horsepower, color, style. Or safety, speed. But deeper down: freedom. Cultural differences. BMW: fun, experience. Audi: technology. Those messages are usually not there explicitly, but it sifts through. Foundation on which they put their other marketing and communication expressions. You don't have to be vocal about your brand identity. Should be consistent, not contradicting. If Paul suddenly shows up in a business suit, I would ask: where is Paul.

This is not about features. Does Paul have portlets? Do I have tiles?

You cannot pick an identity and be done. Time frame of three to five years where you cultivate it. Maybe we already have an identity.

How to describe it. Mission statement, About Us section, mostly describing what you are actually not in the case of corporations. Slogans, persona. Archetypes. No, not the next version of dexterity, but Jungian Archetypes. Twelve in total, like outlaw, sage, magician, lover, hero. An abstraction. Independent of culture: hero evokes the same emotion in America as in Asia.

Example: a sage. Finding truth, wisdom. Weakness: study details forever. Think of Gandalf, Yoda, Dumbledore. BBC: if someone hits a co-worker, you fire him, because it damages your image.

There is more documentation on Jungian Archetypes than you can find on Plone.

This maybe touchy, feely, fuzzy stuff. We are tough, rational developers. So we might need other specialists that actually see this as an exact science.

Back to Plone. Problems have since 2004 been documentation and marketing.

Global positioning or global marketing is not bad, but we are too diverse. If you choose one niche to target, you loose another one.

Try to find a more generic but globally recognizable Brand Identity. Seek help. Derive a brand strategy from this identity. Sharpen the identity that we already have. Focus. Gravitational areas. Schwerpunkte in German. Even distributions can have their own sub branding with aligned personalities. A feeling of what is important in that sector. Look at a competitor, how are they positioning themselves, and can you take the opposite side?

Look at the Plone logo. Close your eyes and be quiet for ten seconds. What do you feel? Probably not much. We have not attached a brand to the logo yet.

A new plone.com and plone.org is nice, but we also need a brand.

Danger of ping-ponging between 'we need to focus on this' and 'no, on that' or 'we do not know how to do this and have no time and money'.

Edward de Bono writes about group dynamics. Imaging six hats with different colors. You can say in a meeting: 'put on your black hat and look at the bad parts' and later 'put on your yellow hat and look at the nice parts', or 'blue hat to look at process'. If six colored hats are in the same discussion at the same time, you will just go back and forth and do not end up anywhere, just go around in circles. This is true for marketing discussions and for technical discussions about our backend or frontend. See http://en.wikipedia.org/wiki/Six_Thinking_Hats

'Plone gives you peace of mind' may sound good to Europeans. Americans may like more 'Plone is stable 24/7.'

Who are you? Mira Lobe / Susi Weigel: I am me. Read it here, nice story: http://www.jungbrunnen.co.at/data/medialibrary/2013/04/Ich_bin_ich_englisch.pdf

JC Brand: Patternslib, Mockup

Patterns make it possible to first create an interaction design, without creating backend code first. Cornelis Kolbach creates designs for sites that still look and work fine without javascript. The patterns are there so it works more nicely or faster when you have javascript enabled, but it is not in the way. It allows a designer to create an interactive design without needing to write your own javascript.

I have replaced mockup stuff with patterns stuff. Mockup is now a collection of Plone-specific patterns, for example for the query string widget used in collections. Patternslib has the non Plone-specific stuff. There is some duplication that we want to resolve. That is the direction in which we are going.

Patternslib is very lightweight. Not a 'kitchen sink' framework. Trying to keep it as vanilla, plain javascript as possible. Lots of third party javascript libraries could be easily integrated. AngularJS maybe not that easily.

Isn't it yet another too Plone specific thing? Patternslib was developed outside of Plone, by Cornelis and Wichert. It has zero dependency on anything in Plone. It is simple conceptually. There is not that much to maintain.

You should usually reuse patterns, instead of creating your own. You may need a new one from time to time, but if you have that often, something is wrong.

Time: not to start a pillow fight, but Angular does much the same thing. But where do you see Mockup in three or four years? In combination with a json api?

JC: We have patterns in Plone now and might as well use them. We need to inform people about how it works, because you sometimes see a pattern where you think: this misses the point. If we use it, we might as well use it in a sane way. Patternslib is not a competition for Angular. We do not want complicated javascript in Plone, which is first and foremost a Python framework. No closely tied javascript client to the Python server.

Alex: Patterns is a simple solution that works at the moment. And it puts us more in the correct javascript mindset. That mindset is more important than the exact technology. Plone as framework should be ready for a pure javascript front-end, whatever it is.

Timo: Three parts that we should do. Cleanup backend, write RESTful api, get some javascript frontend.

Paul: Keep in mind how these things would work for add-on writers. Do they need to learn alien technology or rewrite their add-ons every three years? Also, Mockup documentation should somehow be moved to docs.plone.org.

Timo: Patterns are there in Plone 5. First thing is RESTful api, but that does not change anything yet.

Roel: On the Four Digits Anniversary Sprint we will have an API track, good to have people there who want to work on it.

Martijn Faassen: Server Templating in Morepath 0.10

by Martijn Faassen at 2015-04-09T11:05:00Z

Introduction

I just released Morepath 0.10 (CHANGES)! Morepath is a modern Python web framework that combines power with simplicity of use. Morepath 0.10's biggest new feature is server-side templating support.

Most Python web frameworks were born at a time when server-side templating was the only way to get HTML content into a web browser. Templates in the browser did not yet exist. Server templating was a necessity for a server web framework, built-in from day 1.

The web has changed and much more can be done in the browser now: if you want a web page, you can accomplish it with client-side JavaScript code, helped by templates, or embedded HTML-like snippets in JavaScript, like what the React framework does. Morepath is a web framework that was born in this new era.

Morepath could take a more leisurely approach to server templating. We recommend that users rely on client-side technology to construct a UI -- something that Morepath is very good at supporting. For many web applications, this approach is fine and leads to more responsive user interfaces. It also has the benefit that it supports a strong separation between user interface and underlying data. And you could still use server template engines with Morepath, but with no help from the framework.

But there is still room for server templates. Server-generated HTML has its advantages. It's the easiest way to create a bookmarkable traditional web site -- no client-side routing needed. For more dynamic web applications it can also sometimes make sense to send a server-rendered HTML page to the client as a starting point, and only switch to a client-side dynamic code later. This is useful in those cases where you want the end-user to see a web page as quickly as possible: in that case sending HTML directly from the server can still be faster, as there is no need for the browser to load and process JavaScript in order to display some content.

So now Morepath has now, at last, gained server template support, in version 0.10. We took our time. We prototyped a bit first. We worked out the details of the rest of the framework. As we will see, it's nice we had the chance to spend time on other aspects of Morepath first, as that infrastructure now also makes template language integration very clean.

The basics

Say you want to use Jinja2, the template language used by Flask, in Morepath. Morepath does not ship with Jinja2 or any other template language by default. Instead you can install it as a plugin in your own project. The first thing you do is modify your project's setup.py and add more.jinja2 to install_requires:

install_requires=[
  'more.jinja2',
],

Now when you install your project's dependencies, it pulls in more.jinja2, which also pulls in the Jinja2 template engine itself.

Morepath's extension system works through subclassing. If you want Jinja2 support in your Morepath application, you need to subclass your Morepath app from the Jinja2App:

from more.jinja2 import Jinja2App
class App(Jinja2App):
    pass

The App class is now aware of Jinja2 templates.

Next you need to tell your app what directory to look in for templates:

@App.template_directory()
def get_template_directory():
    return 'templates'

This tells your app to look in the templates directory next to the Python module you wrote this code in, so the templates subdirectory of the Python package that contains your code.

Now you can use templates in your code. Here's a HTML view with a template:

@App.html(model=Customer, template='customer.jinja2')
def customer_default(self, request):
    return {
      'name': self.name,
      'street': self.street,
      'city': self.city,
      'zip': self.zip_code
    }

The view returns a dictionary. This dictionary contains the variables that should go into the customer.jinja2 template, which should be in the templates directory. Note that you have to use the jinja2 extension, as Morepath recognizes how to interpret a template by its extension.

You can now write the customer.jinja2 template that uses this information:

<html>
<body>
  <p>Customer {{name}} lives on {{street}} in {{city}}.</p>
  <p>The zip code is {{zip}}.</p>
</body>
</html>

You can use the usual Jinja2 constructs here.

When you access the view above, the template gets rendered.

Chameleon

What if you want to use Chameleon (ZPT) templates instead of Jinja2 templates? We've provided more.chameleon that has this integration. Include it in install_requires in setup.py, and then do this to integrate it into your app:

from more.chameleon import ChameleonApp
class App(ChameleonApp):
    pass

You can now set up a template directory and put in .pt files, which you can then refer to from the template argument to views.

You could even subclass both ChameleonApp and Jinja2App apps and have an application that uses both Chameleon and Jinja2 templates. While that doesn't seem like a great idea, Morepath does allow multiple applications to be composed into a larger application, so it is nice that it is possible to combine an application that uses Jinja2 with another one that uses Chameleon.

Overrides

Imagine there is an application developed by a third party that has a whole bunch of templates in them. Now without changing that application directory you want to override a template in it. Perhaps you want to override a master template that sets up a common look and feel, for instance.

In Morepath, template overrides can be done by subclassing the application (just like you can override anything else):

class SubApp(App):
    pass
@SubApp.template_directory()
def get_template_directory_override():
    return 'override_templates'

That template_directory directive tells SubApp to look for templates in override_templates first before it checks the templates directory that was set up by App.

If we want to override master.jinja2, all we have to do is copy it from templates into override_templates and change it to suit our purposes. Since a template with that name is found in override_templates first, it is found instead of the one in templates. The original App remains unaffected.

Implementation

In the introduction we mentioned that the template language integration code into Morepath is clean. You should be able to integrate other template engines easily too. Here is the code that integrates Jinja2 into Morepath:

import os
import morepath
import jinja2
class Jinja2App(morepath.App):
    pass
@Jinja2App.setting_section(section='jinja2')
def get_setting_section():
    return {
        'auto_reload': False,
    }
@Jinja2App.template_loader(extension='.jinja2')
def get_jinja2_loader(template_directories, settings):
    config = settings.jinja2.__dict__.copy()
    # we always want to use autoescape as this is about
    # HTML templating
    config.update({
        'autoescape': True,
        'extensions': ['jinja2.ext.autoescape']
    })
    return jinja2.Environment(
      loader=jinja2.FileSystemLoader(template_directories),
      **config)
@Jinja2App.template_render(extension='.jinja2')
def get_jinja2_render(loader, name, original_render):
    template = loader.get_template(name)
    def render(content, request):
        variables = {'request': request}
        variables.update(content)
        return original_render(template.render(**variables), request)
    return render

The template_loader directive sets up an object that knows how to load templates from a given list of template directories. In the case of Jinja2 that is the Jinja2 environment object.

The template_render directive then tells Morepath how to render individual templates: get them from the loader first, and then construct a function that given content returned by the view function and request, uses the template to render it.

Documentation

For more documentation, see the Morepath documentation on templates.

Let us know what you think!

April 08, 2015

Davide Moro: Pip for buildout folks

by davide moro at 2015-04-08T21:53:49Z

... or buildout for pip folks.
... or buildout for pip folks.

In this article I'm going to talk about how to manage software (Python) projects with buildout or pip.

What do you mean for project?
A package that contains all the application-specific settings, database configuration, which packages your project will need and where they lives.
Projects should be managed like a software if you want to assure the needed quality:
This blog post is not:
  • intended to be a complete guide to pip or buildout. If you want to know more about pip or buildout
  • talking about how to deploy remotely your projects

Buildout

I've been using buildout for many years and we are still good friends.
Buildout definition (from http://www.buildout.org):
"""
Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later. 
"""
With buildout you can build and share reproducible environments, not only for Python based components.

Before buildout (if I remember well the first time I get started to use buildout was in 2007, probably during the very first Plone Sorrento sprint) it was a real pain sharing a complete and working developing environment pointing to the right version of several repositories, etc. With buildout it was questions of minutes.

From https://pypi.python.org/pypi/mr.developer.
Probably with pip there is less fun because there isn't a funny picture that celebrates it?!
Buildout configuration files are modular and extensible (not only on per-section basis). There are a lot of buildout recipes, probably the one I prefer is mr.developer (https://pypi.python.org/pypi/mr.developer). It allowed me to fetch different versions of the repositories depending on the buildout profile in use, for example:
  • production -> each developed private egg point to a tag version
  • devel -> the same eggs point to the develop/master
You can accomplish this thing creating different configurations for different profiles, like that:

[buildout]

...

[sources]

your_plugin = git git@github.com:username/your_plugin.git

...

I don't like calling ./bin/buildout -c [production|devel].cfgwith the -c syntax because it is too much error prone. I prefer to create a symbolic link to the right buildout profile (called buildout.cfg) and you'll perform the same command both in production or during development always typing:

$ ./bin/buildout

This way you'll avoid nasty errors like launching a wrong profile in producion. So use just the plain ./bin/buildout command and live happy.

With buildout you can show and freeze all the installed versions of your packages providing a versions.cfg file.

Here you can see my preferred buildout recipes:
Buildout or not buildout, one of the of the most common needs it is the ability to switch from develop to tags depending on you are in development or production mode and reproduce the same software later. I can't figure out to manage software installations without this quality assurance.

More info: http://www.buildout.org

Pip

Let's see how to create reproducible environments with develop or tags dependencies for production environments with pip (https://pip.pypa.io/en/latest/).

Basically you specify your devel requirements on a devel-requirements.txt file (the name doesn't matter) pointing to the develop/master/trunk on your repository.

There is another file that I call production-requirements (the file name doesn't matter) that it is equivalent to the previous one but:
  • without devel dependencies you don't want to install in production mode
  • tagging your private applications (instead of master -> 0.1.1)
This way it is quite simple seeing which releases are installed in production mode, with no cryptic hash codes.

You can use now the production-requirements.txt as a template for generating an easy to read requirements.txt. You'll use this file when installing in production.

You can create a regular Makefile if you don't want to repeat yourself or make scripts if you prefer:
  • compile Sphinx documentation
  • provide virtualenv initialization
  • launch tests against all developed eggs
  • update the final requirements.txt file
For example if you are particular lazy you can create a script that will create your requirements.txt file using the production-requirements.txt like a template.
This is a simple script, it is just an example, that shows how to build your requirements.txt omitting lines with grep, sed, etc:
#!/bin/bash

pip install -r production-requirements.txt
pip freeze -r production-requirements.txt | grep -v mip_project | sed '1,2d' > requirements.txt
When running this script, you should activate another Python environment in order to not pollute the production requirements list with development stuff.

If you want to make your software reusable and as flexible as possible, you can add a regular setup.py module with optional dependencies, that you can activate depending on what you need. For example in devel-mode you might want to activate an entry point called docs (see -e .[docs] in devel-requirements.txt) with optional Sphinx dependencies. Or in production you can install MySQL specific dependencies (-e .[mysql]).

In the examples below I'll also show how to refer to external requirements file (url or a file).

setup.py

You can define optional extra requirements in your setup.py module.
mysql_requires = [
'MySQL-python',
]

docs_requires = [
'Sphinx',
'docutils',
'repoze.sphinx.autointerface',
]
...

setup(
name='mip_project',
version=version,
...
extras_require={
'mysql': mysql_requires,
'docs': docs_requires,
        ... 
},

devel-requirements.txt

Optional extra requirement can be activated using the [] syntax (see -e .[docs]).
You can also include external requirement files or urls (see -r) and tell pip how to fetch some concrete dependencies (see -e git+...#egg=your_egg).
-r https://github.com/.../.../blob/VERSION/requirements.txt
 
# Kotti
Kotti[development,testing]==VERSION

# devel (to no be added in production)
zest.releaser

# Third party's eggs
kotti_newsitem==0.2
kotti_calendar==0.8.2
kotti_link==0.1
kotti_navigation==0.3.1

# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git#egg=kotti_actions
-e git+https://github.com/truelab/kotti_boxes.git#egg=kotti_boxes
...

-e .[docs]

production_requirements.txt

The production requirements should point to tags (see @VERSION).
-r https://github.com/Kotti/Kotti/blob/VERSION/requirements.txt
Kotti[development,testing]==VERSION

# Third party's eggs
kotti_newsitem==0.2
kotti_calendar==0.8.2
kotti_link==0.1
kotti_navigation==0.3.1

# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git@0.1.1#egg=kotti_actions
-e git+https://github.com/truelab/kotti_boxes.git@0.1.3#egg=kotti_boxes
...

-e .[mysql]
requirements.txt
The requirements.txt is autogenerated based on the production-requirements.txt model file. All the installed versions are appended in alphabetical at the end of the file, it can be a very long list.
All the tag versions provided in the production-requirements.txt are automatically converted to hash values (@VERSION -> @3c1a191...).
Kotti==1.0.0a4

# Third party's eggs
kotti-newsitem==0.2
kotti-calendar==0.8.2
kotti-link==0.1
kotti-navigation==0.3.1

# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git@3c1a1914901cb33fcedc9801764f2749b4e1df5b#egg=kotti_actions-dev
-e git+https://github.com/truelab/kotti_boxes.git@3730705703ef4e523c566c063171478902645658#egg=kotti_boxes-dev
...


## The following requirements were added by pip freeze:
alembic==0.6.7
appdirs==1.4.0
Babel==1.3
Beaker==1.6.4
... 

Final consideration

Use pip to install Python packages from Pypi.

If you’re looking for management of fully integrated cross-platform software stacks, buildout is for you.

With buildout no Python code needed unless you are going to write new recipes (the plugin mechanism provided by buildout to add new functionalities to your software building, see http://buildout.readthedocs.org/en/latest/docs/recipe.html).

Instead with pip you can manage also cross-platform stacks but you loose the flexibility of buildout recipes and inheritable configuration files.

Anyway if you consider buildout too magic or you just need a way to switch from production vs development mode you can use pip as well.

Links

If you need more info have a look at the following urls:
Other useful links: