Planet Plone

Ross Patterson: Buildout Performance Improvements

Fri, 27 Jan 2012 18:08:32 +0000

I've been baffled by the amount of time running buildout takes. In particular, it seemed like a lot of duplicate work was being done when multiple parts had identical or very similar sets of required eggs/distributions. After a bunch of profiling and timing, I've found there are a number of ways to increase your buildout performance.

Use buildout.dumppickedversions version 0.5 or later:: If you're using buildout.dumppickedversions, use version 0.5. This release includes a fix I contributed which yields a 3-4 fold decrease in runtime if buildout is run with just the -N option and no -v options. If your're using buildout-versions instead of buildout.dumppickedversions, unfortunately it suffers from the same problem. The author knows about this and may address it but until then you may want to switch to buildout.dumppickedversions.
Use zc.buildout version 1.5 or later:: This seems to yield a 2-3 fold decrease in buildout runtime when run with -N and addresses a similar hot loop as found in buildout.dumppickedversions. In fact, it's resolution of required distributions seems to be much more efficient in general. If you're stuck with 1.4, I've contributed the same fix as for buildout.dumppickedversions to the 1.4 branch so it should make it into a next release if someone makes a new 1.4 release.
Use the -N option:: With the above newest releases of zc.buildout and buildout.dumppickedversions, you can now exercise a lot of control of your run times depending on what command-line options you use. In particular, make sure newest is false and that verbosity is not increased. Both of these can be influenced by options in the buildout configuration, so keep an eye out for that, but if no such options are interfering then this means just invoking buildout like $ bin/buildout -N -c buildout.cfg. IOW, use one -N option and no -v options. Before the improvements in zc.buildout 1.5 and the fix to buildout.dumppickedversions, a lot of time was wasted doing work that would only actually be used if -v had been given.
Clean out your egg cache and use virtualenv --no-site-packages:: Some of the remaining inefficiencies in zc.buildout are proportional to the number of distributions available on sys.path. IOW if you reduce the number of distributions to be scanned, you can increase performance. If you're using a shared eggs-directory or have a buildout that has been around for a long time, there may be a lot of different versions of the same packages in eggs-directory. By cleaning those out you reduce the number of distributions that buildout needs to scan. I recommend just emptying your eggs-directory and then letting buildout re-download all the eggs it actually needs now. I takes a while once, but then it's done. Similarly, using virtualenv --no-site-packages can reduce the number of dists buildout needs to scan.
Help me get zc.buildout/branches/env-cache merged and released:: Having said the above about cleaning out the egg cache, buildout, distribute, and setuptools shouldn't be scanning these paths multiple time anyways. Along with package indexes, these paths are global in nature and so the dists found there should only be scanned for once and cached globally. I've started this work on the env-cache branch of zc.buildout. In my timing tests, this branch seems to yield another ~60% decrease in run time. The zc.buildout tests, however, are a PITA and Jim Fulton is a very busy man with little time to evaluate this branch. If anyone else can help me shepherd this branch through to merge and then release, that would be great. I've written up how to use this branch with your existing buildouts.

With all this in effect I took a production buildout and another development buildout from 2 minute run times down to 20 second run times. This really makes re-running buildout something I no longer feel an urge to avoid. So use zc.buildout 1.5.2, buildout.dumppickedversions 0.5, -N and do what you can to help get the env-cache branch merged.

Six Feet Up: Optimize Your Plone Development by Packing the ZODB

Fri, 27 Jan 2012 11:30:00 +0000

What does it mean to "pack" a database? In shortest form, it means to remove unused records in order to reduce the memory required to hold the database.

Think of it like this: each time you delete a page in your website, a copy of that page is kept in the database. These copies are kept in case you need to get it back later. Over time, as you create and delete content and make other changes in your website, the number of unused objects can grow and grow, expanding the memory needed to operate the website, and slowing down operations. When you pack the database, the system finds the unused objects and removes them. Performing this job regularly helps to keep the size of a database manageable.

Perform database packs regularly

What is the benefit of scheduling routine database packs? If the database is not being packed on a regular basis, the amount of resources and time to perform a database pack may increase exponentially. Making sure that the packs are happening on a scheduled basis will save time in the long run. These are tasks that can easily be added into your buildout and then released into a production environment.

As an example, we recently had a Plone site that was using RelStorage backed with MySQL that had not been packed in over a year. The size of the MySQL dump grew to nearly 8GB and took 45 minutes to complete. This means that any time a release was made, there would be 45 minutes of lead time just to make a backup of the data.

Having this large database does not only impact the ability to swiftly do a release. If a new bug is discovered that can only be re-produced on the production environment, you certainly don't want to be debugging it there. Instead, you will want to be able to quickly bring back the data to your testing instance, hence the benefit of keeping the size of the database to a manageable size.

Setting up the automatic pack

When using a ZEO server, either from plone.recipe.zeoserver or plone.recipe.zope2zeoserver, a zeopack script is automatically generated for you with all the necessary parameters to run the pack. This can simply be run from the command line:

$ cd path/to/buildout
$ bin/zeopack

Now your database is packed, that was easy! As programmers and systems administrators, we know that doing things manually will eventually fail. How do we automate this process? Simple, we add a cron job to take care of the work for us. Here is an example configuration that gets everything set up for us:

[buildout]
parts = instance zeoserver cron-pack
extends = http://dist.plone.org/release/4.1.3/versions.cfg

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
zeo-address = 8100
zeo-client = True
# Set the shared blob option so blobs work with zeoserver
shared-blob = on
blob-storage = ${buildout:directory}/var/blobstorage
eggs =
    Pillow
    Plone

[zeoserver]
recipe = plone.recipe.zeoserver
blob-storage = ${instance:blob-storage}
zeo-address = ${instance:zeo-address}
# keep history for 7 days instead of the default 1
pack-days = 7

[cron-pack]
recipe = z3c.recipe.usercrontab
times = @weekly
command = ${buildout:bin-directory}/zeopack

After running this buildout, there will be a cron job added to the cron of the user that ran the buildout. This job will run weekly and pack the database, leaving 7 days of history.

RelStorage

Now for a more complex scenario using RelStorage. The zeopack script will no longer work, but fear not, RelStorage comes with a script called zodbpack that can do the job.

First some setup to make the generation of the conf file work:

$ cd path/to/buidout
$ mkdir templates
$ mkdir etc
$ touch templates/zodbpack.conf.tmpl

This sets up a directory for our zodbpack.conf template, and an output directory named etc. In the zodbpack.conf.tmpl, place the following code:

<relstorage>
  <mysql>
    host ${settings:mysql-host}
    db ${settings:mysql-db}
    user ${settings:mysql-user}
    passwd ${settings:mysql-passwd}
  </mysql>
</relstorage>

Now here is our example buildout configuration that sets up Plone to use RelStorage with MySQL:

[buildout]
parts = instance relstorage zodbpack-conf cron-pack
extends = http://dist.plone.org/release/4.1.3/versions.cfg

[settings]
# settings for RelStorage connection
mysql-host = localhost
mysql-db = mydbname
mysql-user = root
mysql-passwd = 
# options for the cron job
pack-days = 7

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
eggs =
    Pillow
    Plone
    RelStorage
    MySQL-python
rel-storage =
    type mysql
    host ${settings:mysql-host}
    db ${settings:mysql-db}
    user ${settings:mysql-user}
    passwd ${settings:mysql-passwd}
    blob-dir ${buildout:directory}/var/blobstorage

# Generate the RelStorage scripts
[relstorage]
recipe = zc.recipe.egg
eggs =
    ${instance:eggs}
    RelStorage
    MySQL-python

[zodbpack-conf]
recipe = collective.recipe.template
input = ${buildout:directory}/templates/zodbpack.conf.tmpl
output = ${buildout:directory}/etc/zodbpack.conf

[cron-pack]
recipe = z3c.recipe.usercrontab
times = @weekly
command = ${buildout:bin-directory}/zodbpack -d ${settings:pack-days} ${zodbpack-conf:output}

In this example, we are generating a conf file for the zodbpack script to use with the [zodbpack-conf] part. Since the RelStorage information will be used in two places, and might be different on each deployment platform, there is a [settings] section that both the [instance] and [zodbpack-conf] parts take advantage of.

Once this buildout has run, it will be set up similar to our conventional buildout. The cron job will pack the RelStorage database once a week, keeping 7 days of history.

Conclusion

Packing your Plone site's database on a regular basis is as simple as adding some configuration to buildout. There is really no reason you shouldn't be keeping up on this maintenance task with your Plone sites, especially when you run complex websites or intranets. It will save you time during your development and release cycles by making sure the database is at its leanest.

ACLARK.NET, LLC: Hello Plone theming

Wed, 25 Jan 2012 05:15:50 +0000

In this post, we introduce a new tutorial aimed at bridging the gap between the incredibly easy to use and enormously powerful Diazo theming engine for Plone, and bootstrapping your first new theme. You do not need to have any Python skills whatsoever to perform these tasks.

New Plone themes are:

HTML/CSS/JavaScript/images (Sound familiar? You don’t need to know any Plone to understand this part.)
A set of rules that map your HTML/CSS/JavaScript/images to Plone content. (This part is Diazo-specific; and while Diazo is not part of Plone, integration is provided by a Python package called plone.app.theming.)

The Diazo rules can get complex, but it’s very easy to get your first rule done and working and then off you go. You may find this new tutorial on pythonpackages.com. Enjoy, and get busy theming Plone.

Plone.org: Plone Symposium East 2012 Calls for Training Proposals - Due by Feb 13, 2012

Tue, 24 Jan 2012 05:57:16 +0000

The folks at Weblion have begun organizing Plone Symposium East 2012 (PSE12), and have put out the call for trainers who would like to hold training sessions in the two days prior to the May event - on May 20-21st. It is an opportunity to teach others how to be successful with open source.

Interested trainers have time to submit their proposals between now and February 13. You can find out more about the proposals process by visiting http://weblion.psu.edu/symposium/training-proposal

There is limited classroom space, so get your proposal in TODAY!

Connexions Blog: Want a free Connexions T-Shirt?

noreply@blogger.com (Daniel Williamson) — Tue, 24 Jan 2012 02:04:51 +0000

Next week, we are going to be having a contest!

Here's the scoop.

Next week, January 30th - February 3rd, 2012, we will ask a Connexions related trivia question on twitter everyday. First to answer correctly by @replying to us on Twitter gets a free t-shirt!

This week we will be sharing some of the facts you will need to know in order to answer the questions correctly. So, keep an eye out for hints on twitter and facebook.

Unfortunately, we are only able to ship domestically. If you have any questions about this contest, please email dcwill@cnx.org.

Hector Velarde: Adding test users programmatically using collective.recipe.plonesite

Mon, 23 Jan 2012 23:45:00 +0000

I'm writing some functional tests for a Plone project and I need to add a group of test users every time I create a test site.

collective.recipe.plonesite is a cool Buildout recipe that enables you to create and update a Plone site as part of a buildout run.

I added the following lines to my buildout configuration:

parts =
    …
    plonesite

[plonesite]
recipe = collective.recipe.plonesite
profiles = my.project:default
post-extras = ${buildout:directory}/acceptance-tests/add_test_users.py

The code of the add_test_users.py script is pretty straightforward:

"""This script will add a number of test users to a Plone site.

You can used it in post-extras option of collective.recipe.plonesite. It will
be evaluated after running QuickInstaller and GenericSetup profiles.

@param portal: The Plone site as defined by the site-id option
"""

import logging
logger = logging.getLogger('collective.recipe.plonesite')

test_users = [
    # (username, password, group),
    ('username1', 'password1', 'group1'),
    ('username2', 'password2', 'group2'),
    ('username3', 'password3', 'group3'),
    ]

for username, password, group in test_users:
    if username not in portal.acl_users.getUserIds():
        try:
            portal.portal_registration.addMember(username, password)
            portal.portal_groups.addPrincipalToGroup(username, group)
        except ValueError:
            logger.warn('The login name "%s" is not valid.' % username)
        except KeyError:
            logger.warn('The group "%s" is not valid.' % group)

Enjoy!

(Next time I'll give you my first impressions on SeleniumLibrary, a web testing library for Robot Framework, I'm using to write the tests.)

Four Digits: Ladies and gentleman, we 've got it

Mon, 23 Jan 2012 15:10:00 +0000

The Plone conference 2012 will take place in Arnhem, the Netherlands. The foundation members have had their advisory vote and the majority favored Arnhem. Last weekend, the Board made the final decision, so now we can start the preparations. Needless to say we are very happy to be elected.

We already put together a team together that will take care of initial planning and preparations. Of course, during the conference, we will need all the help we can get so all of the Four Digits employees will be assigned to a specific task. We had our first post-proposal planning meeting today and are in the works of making the planning up to the event.

We will keep you posted on all the relevant information in the months to come. Here, on Twitter and we will utilize the Dutch local and (if possible) national media. Apart from organising a great conference, we want to promote Arnhem as one of the best cities in the Netherlands. The company is located here and all the Four Digits members live in Arnhem.

The Plone conference 2012 will be held in the very heart of Arnhem centre, within walking distance of pubs, restaurants, shops, and public transport. In fact, the venue is very close to the office. We are going to our best to make sure you all have a great time here. Check out our proposal for more info on both the conference and the city of Arnhem.

And just to get you warmed up, a few of the many thing to see here. As mentioned in Mark Corum's article, Arnhem has a famous bridge that played a key role during the second world war in The Netherlands. Be sure to see the movie A Bridge Too Far featuring Sean Connery, Gene Hackman and Robert Redford (which sadly wasn't filmed in Arnhem).

Arnhem also houses one of the biggest zoo's in Holland, as well as the only historical open air museum in the country. In all, it's a beautiful city with lots of nice shops, cosy bars en a lot of parks to take a stroll in.

For updates keep track of this blog, and the soon to launch Plone conf website. For now: we thank the community for the opportunity to host the conference and we thank Abstract and Fanelli Consulting for putting in a great bid. Hope to see all you guys in October!

Netsight Developers: Full Text Code Searching

Mon, 23 Jan 2012 15:08:28 +0000

Last night I saw a tweet re-tweeted by David Glick about a post from one of the original guys who worked on Google Code search, Russ Cox. He has released some code for doing code search based upon the ideas used by Google Code Search.

Google shut down Google Code search recently and his code allows you to do similar hybrid index/regex searching on files locally.

I have to confess, I never really used Google Code Search. I guess probably due to until actually reading the post above I didn't realise that it could do regular expression searching. This is a 'big thing' when searching in code. I really don't know how I missed that fact.

I am a bit of a (lapsed) search and information retrieval geek myself. My final year project at university was a full text mailing list indexer/search system (similar to gmane). This was based heavily on the seminal work mentioned in Russ' post Managing Gigabytes by Witten, Moffat, and Bell and also by Modern Information Retrieval by Baeza-Yates and Ribeiro-Neto and the paper that influenced much of the internet as we know it now, Brin and Page's The Anatomy of a Large-Scale Hypertextual Web Search Engine.

Russ Cox has released the code to his tool at http://code.google.com/p/codesearch/ and you can download binaries for OSX, FreeBSD, Linux, and Windows.

Here is a quick rundown on using it to index and then search my local development directory on my Macbook. This allows me to quickly search through both all the code in the projects I'm working on, but all my egg cache, which contains all the code to Plone itself. Great for when you see a particular bit of markup, or error message and need to work out where it is coming from in the code.

Firstly download the binaries from the link above, or compile it yourself. Then you need to get it to index your directories. In my case everything I do in under /Development:

dhcp10:~ matth$ cindex /Development
2012/01/23 14:47:26 index /Development
2012/01/23 14:49:08 flush index
2012/01/23 14:49:08 merge 7 files + mem
2012/01/23 14:49:23 520694590 data bytes, 82743702 index bytes
2012/01/23 14:49:23 merge /Users/matth/.csearchindex /Users/matth/.csearchindex~
2012/01/23 14:49:39 done

And then you can use it to search through the code:

dhcp10:~ matth$ csearch "def authenticate"
/Development/buildout-cache/eggs/AccessControl-2.13.4-py2.6-macosx-10.7-x86_64.egg/AccessControl/userfolder.py:    def authenticate(self, name, password, request):
/Development/buildout-cache/eggs/AccessControl-2.13.4-py2.6-macosx-10.7-x86_64.egg/AccessControl/users.py:    def authenticate(self, password, request):
/Development/buildout-cache/eggs/AccessControl-2.13.4-py2.6-macosx-10.7-x86_64.egg/AccessControl/users.py:    def authenticate(self, password, request):
/Development/buildout-cache/eggs/Paste-1.7.5.1-py2.6.egg/paste/auth/basic.py:    def authenticate(self, environ):
/Development/buildout-cache/eggs/Paste-1.7.5.1-py2.6.egg/paste/auth/digest.py:    def authenticate(self, environ):
/Development/buildout-cache/eggs/Paste-1.7.5.1-py2.7.egg/paste/auth/basic.py:    def authenticate(self, environ):
/Development/buildout-cache/eggs/Paste-1.7.5.1-py2.7.egg/paste/auth/digest.py:    def authenticate(self, environ):
...

One current issue is there is no way to exclude directories from the index, so you get .svn directories in the results

dhcp10:~ matth$ csearch "sspi.ServerAuth"
/Development/netsight.windowsauthplugin/netsight/windowsauthplugin/windowsauthplugin/.svn/text-base/krbtest.py.svn-base:            sa = sspi.ServerAuth('Negotiate')
/Development/netsight.windowsauthplugin/netsight/windowsauthplugin/windowsauthplugin/.svn/text-base/plugin.py.svn-base:            sa = sspi.ServerAuth('Negotiate')
/Development/netsight.windowsauthplugin/netsight/windowsauthplugin/windowsauthplugin/krbtest.py:            sa = sspi.ServerAuth('Negotiate')
/Development/netsight.windowsauthplugin/netsight/windowsauthplugin/windowsauthplugin/plugin.py:            sa = sspi.ServerAuth('Negotiate')
/Development/py24nsp/sanofi/src/netsight.windowsauthplugin/netsight/windowsauthplugin/.svn/text-base/krbtest.py.svn-base:            sa = sspi.ServerAuth('Negotiate')
/Development/py24nsp/sanofi/src/netsight.windowsauthplugin/netsight/windowsauthplugin/.svn/text-base/plugin.py.svn-base:            sa = sspi.ServerAuth('Negotiate')
/Development/py24nsp/sanofi/src/netsight.windowsauthplugin/netsight/windowsauthplugin/krbtest.py:            sa = sspi.ServerAuth('Negotiate')
/Development/py24nsp/sanofi/src/netsight.windowsauthplugin/netsight/windowsauthplugin/plugin.py:            sa = sspi.ServerAuth('Negotiate')
/Development/py26nsp2/plone41demo/src/netsight.windowsauthplugin/netsight/windowsauthplugin/.svn/text-base/krbtest.py.svn-base:            sa = sspi.ServerAuth('Negotiate')
/Development/py26nsp2/plone41demo/src/netsight.windowsauthplugin/netsight/windowsauthplugin/.svn/text-base/plugin.py.svn-base:            sa = sspi.ServerAuth('Negotiate')
/Development/py26nsp2/plone41demo/src/netsight.windowsauthplugin/netsight/windowsauthplugin/krbtest.py:            sa = sspi.ServerAuth('Negotiate')
/Development/py26nsp2/plone41demo/src/netsight.windowsauthplugin/netsight/windowsauthplugin/plugin.py:            sa = sspi.ServerAuth('Negotiate')

You can however restrict the search to a path (regex):

dhcp10:~ matth$ csearch -f /Development/py26nsp/cplonline/src/netsight.cpl "def auth"
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/.svn/text-base/models.py.svn-base:    def authenticate(self, password):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/models.py:    def authenticate(self, password):

As it can take regular expressions, you can do more advanced searches, e.g. 'all methods that take an argument called password':

dhcp10:~ matth$ csearch -f /Development/py26nsp/cplonline/src/netsight.cpl "def.*\(.*password"
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/__init__.py:def fetchandload(ftpcmd, loadcmd, ftpcache, host, user, password):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/evidence.py:    def __init__(self, uri, user=None, password=None):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/models.py:    def __init__(self, name, login, password, company_id, role=None):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/models.py:    def _hash_password(self, password, salt):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/models.py:    def set_password(self, password):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/models.py:    def authenticate(self, password):
/Development/py26nsp/cplonline/src/netsight.cpl/netsightcpl/tests/base.py:    def login(self, login='admin', password='admin'):

So thanks to Russ for releasing this. It certainly is much quicker for search through large directories of code than using grep or ack

Reinout van Rees: Website changes: sphinx code, layout, twitter bootstrap, less

Sun, 22 Jan 2012 21:49:00 +0000

I've made a couple of changes to my website (and thus weblog):

I moved from svn to git.
I made my weblog-related sphinx customizations public.
I'm using twitter bootstrap as the css framework instead of the yahoo one I've been using before.
I changed the layout. Bigger font, less clutter. More bare-bones. Most of the looks (font type, headers) is from the bootstrap css, but the ordering is mine. I especially like the bigger font.

Svn to git

I'm getting used to git at the office. And my website was in svn. The svn is on me and my brother's server. And we're going to re-build that server in the near future. And not having to set up svn there saves quite some time.

So I moved part of the code to github and the actual content to a simple directory in our own server. Git works just fine with a repository URL like ssh://vanrees.org/~/git/websitecontent. Just init a repo in ~/git/websitecontent on the server and off you go.

Sphinx as weblog: code is now available (somewhat)

My site is static html now for two years or so. Generated with http://sphinx.pocoo.org/. No Plone (which it was when I worked with Plone) and no Django (which I'd like, as I work with it, but my current text-only setup works real fine and Sphinx is an essential Django documentation ingredient, too).

I also make my weblog with Sphinx. I need a couple of custom restructuredtext tags for that (like ..tags:: for handling tags). I basically point a script at a sphinx directory that contains my entries and it generates index.txt files for all the directories with a list of entries. And it generates tags/TAGNAME.txt files for the tags. Afterwards, I just run Sphinx and everything gets generated OK.

I got some questions in the past about my code, but never followed up by publishing my terribly Reinout-specific code. It is still Reinout-specific, but I've published it now. It might provide some ideas. The full code is at https://github.com/reinout/reinout.vanrees.org . Again: this is not something you can pip install. Best you can do is borrow and copy at the moment. (Threat the Python code in there as public domain).

Most of interest are probably https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/rst.py which provides the custom restructuredtext (and thus sphinx) tags. And https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/weblog.py which creates all the index and tags files.

It is not well-documented and quite specific to my case. But there are some docstrings and the code is nicely PEP8 compliant :-) If you need any info, feel free to contact me.

Twitter bootstrap css

I have to re-visit the layout of the websites we make at my work (Nelen & Schuurmans). One of the things we want is to have a more attractive layout. And after some browsing, twitter bootstrap struck me as a good basis.

And, as I do often, I first experiment on myself. So I set out to change the layout of my own website by using twitter bootstrap. I'm not even using half of bootstrap's possibilities, but I'm quite happy with the result. The basic typography is fine. There are some handy helpers for common things like labels (which I used to liven up the tags below the weblog headers on my weblog). And I quite like the simple header at the top of the page: just a simple bar.

And there's another good thing about twitter bootstrap: less. See below.

Less

What coffeescript is for javascript, less is for css. A simpler syntax that helps you write better and less duplicated css.

I like it! Take for example the headings of this entry. Hover over one and you'll see a paragraph token at the end. Sphinx adds this so you can link to individual sections in a bigger file. Sphinx also has some css that hides the paragraph token normally and only shows it when you hover over the heading:

a.headerlink {
    visibility: hidden;
}

h1:hover > a.headerlink,
h2:hover > a.headerlink,
h3:hover > a.headerlink,
h4:hover > a.headerlink,
h5:hover > a.headerlink,
h6:hover > a.headerlink,
dt:hover > a.headerlink {
    visibility: visible;
}

I restricted that to:

.headerlink {
  visibility: hidden;
  }

h1, h2, h3, h4, h5, h6, dt {
  &:hover {
    .headerlink {
      visibility: visible;
      }
    }
  }

It looks just like css, as you see in that first .headerlink rule. The second one has the special less stuff. The ampersand means an extra condition, so the &:hover effectively adds :hover to each of the h1..``dt``. This really shines if you have a couple of other extra conditions on the same set of elements.

Anyway, head over to http://lesscss.org/ where they have a better explanation.

Closing off

I'm happy with the change. One extra change that I didn't mention yet: I've removed deliverance from the mix. Deliverance is a tool that I used to change the layout of a couple of inputs (sphinx, docbook2html, latex2html) into my vanrees.org layout. It did that on the fly. But the versions kept changing and the versions I had were getting old and hard to update. And I had to run a wsgi process to run it, even though the actual website was all plain html.

I'm back to plain html now. I've implemented the layout in a Sphinx template. Simple to do, as it is just jinja2, which has the same syntax as Django. And the docbook and latex html exports are just that. I'll leave them that way. It isn't that important to have my logo on those pages :-)

One extra advantage: the site ought to be faster!

Plone.org: Plone Conference 2012 Coming to Arnhem, the Netherlands

Sun, 22 Jan 2012 07:00:00 +0000

Arnhem, the Netherlands will host the 2012 Plone Conference, the Plone Foundation announced today following a vote of the foundation membership. The Arnhem proposal, championed by Dutch Plone Developers Four Digits, beat out a competing proposal for the conference to be held in Paris, France which was made by Abstract and Fanelli Consulting.

The Plone Conference 2012 will be held in Arnhem on October 10-12, with pre-conference training on October 8-9 and a post-conference sprint October 13-14.

Arnhem, which is located the east Netherlands between the Veluwe nature reserve and the Rhine river, is well known for its art and culture. Its compact city center places many attractions in easy walking distance. Once known as World War II's "A Bridge Too Far," Arnhem's reputation as one of the greenest cities in Europe and famous hospitality to visitors from around the globe makes a stellar location for this year's conference.

One of Four Digits' goals for hosting Plone Conference 2012 is "putting Arnhem on the map for Plone," but the folks hosting this year's conference are far from unknowns in the Plone community. In addition to organizing the very successful and fun Living Statutes Sprints in 2010 and 2011, they've participated Plone conferences and sprints around the globe since 2007. Their members also are well-known as "do-ers" throughout the Plone community and are represented on the Plone Foundation Board of Directors. Four Digits develops open source software and web applications for companies and organisations, ranging from relatively simple websites to complex systems. They have a staff of eight people, none of whom are addressed as "manager" or "supervisor."

"I cannot say how incredibly excited I am and we all are here at Four Digits. We think Arnhem is a great place to host the Ploneconf and are very much looking forward to October," said Sjoerd van Elferen, a member of the Four Digits team.

More information on the conference, including registration, calls for speakers, and conference specifics, will be coming soon from ploneconf.org .

Jazkarta Blog: Agile Development with Plone, Revisited

Fri, 20 Jan 2012 18:39:54 +0000

The announcement of the 2012 Plone Symposium East (thank you Weblion! always a great gathering) is reminding me of the talk I gave there last year on Agile Development with Plone. Here at Jazkarta we’re engaged in another large project, similar in many ways to the one that inspired that talk. Because our project management tools and techniques have evolved a bit since last year, I figured it was time for an update.

The Projects

Last year at this time, we were just getting started on a major website redesign for the University of Minnesota Press. There’s lots of information about this project in my Plone Conference presentation How To Get a Fabulous Website on a Modest Budget Using Plone. Briefly, it involved a major data conversion from a FileMaker database into Plone, a small number of complex custom content types, extensive design work, and a featured “explore” section powered by eea.facetednavigation.

Our new project is a major website redesign for Dumbarton Oaks Research Library and Collection, a research institute of Harvard University and the home of a little known but outstanding museum and garden in Washington, D.C. We began with a discovery phase last summer, and have been working on implementation since November.

What’s the Same

Both of these are fixed budget projects with a large number of desired features, and at the outset many of these were inadequately defined for development purposes. In both cases there are multiple stakeholders with differing priorities, but there is an intelligent, proactive project owner who is trusted and empowered to make decisions. The Jazkarta team is the same – Carlos de la Guardia in Mexico, Alec Mitchell in California, and me in Massachusetts – and none of us ever work full time on any of our projects.

This all adds up to the same basic project management challenge: how to adapt agile techniques to manage a fixed price, flexible scope project with a distributed, part time team.

What’s Different

Happily, the Dumbarton Oaks project does not involve a complicated data import from a FileMaker database. For UMP this required a collective.transmogrifier script, many many many trial and error imports, and over 60 hours of Jazkarta development time. Dumbarton Oaks currently has a static HTML website, which is much simpler to import. Funnelweb made this easy – only a few trial imports and some content massaging were required. We’re not a quarter through the project but all the old content is already on staging. Total development effort: 10 hours.

Because the content import was relatively easy and the project mostly requires custom features and content types that are independent of each other, we have been able to divide up the work into several releases. This gives us a series of smaller planning horizons and allows us to be more agile (more on this later). Each release can potentially be deployed to production – and even if we decide not to, new content is developed on staging which gives the new features a thorough vetting. No matter how much you thought you tested, there are always problems that don’t get discovered until the software sees serious use. Having a series of small production deployments avoids piling all the bug fixes and theme tweaks into a single mega deployment phase at the end of the project.

Our Evolving Agile Development Practices

Our basic development process remains the same:

Discovery
Define schedule and resources (how the budget will get spent)
User story development
Story estimation (we use planning poker)
A series of development iterations alternated with evaluation periods
Production deployment

During discovery the project owner and I distributed the feature requirements into a series of releases, based on a combination of priorities and keeping all the stakeholders happy. (Everyone got something in almost every release.) This high level release plan allows us to do our story development and estimation one release at a time, which saves time and money – especially considering that all the developers participate in planning poker sessions. Since we know the customer wants more than their fixed budget will get them, and we also know they will change their mind about what they want as the project progresses, it makes sense to postpone the effort of writing and estimating stories we may never get to.

As in previous projects, I took the first cut at turning the project requirements into stories, then discussed and revised the stories with the project owner. Because I have intimate knowledge of Plone, I can translate requirements into stories that make sense to develop. For example, we don’t need to spend time defining stories for CMS functionality or for things that Plone has great add-ons for (like staff directories and bibliographies). Some things may turn out to need tweaking, but it is better to give the customer stock Plone and add-on features to try out first. They may decide to live with the stock features and put their development dollars into new features instead of making changes.

Our communication tools remain the same: Skype, IRC, chat, Google Calendar, and Google Documents. This combination is the secret sauce that allows a distributed team to work as if they were in the same room. We did switch from ClueMapper (based on svn and trac) to github for source code repository and ticket tracking. We miss a few management features, but the developers are much happier working in a dvcs.

Our development iterations remain the same – one week each, followed by a week of evaluation during which the customer has time to test and reflect on what they want to do next. We kick off each iteration with a planning meeting, which surfaces any questions about the stories we’re tackling and allows us to do task breakdown and assignment while we are all in the virtual room of a Skype conference call. During the iteration we have a short standup meeting every day to keep things on track – everyone says what they did yesterday, what they’ll do today, and raises any blocking issues. My notes from the standups are written in a Google spreadsheet which is shared with everyone. This gives the project owner access to detailed information about how things are going, if they miss a standup meeting.

The biggest change we’ve made for this project is to adopt ScrumDo for story development and iteration planning. ScrumDo is an open source agile project management tool that’s written in Python. We are using the hosted, SaaS version that’s at scrumdo.com – accounts are free for up to 10 users and 10 projects. ScrumDo has a friendly drag and drop UI that makes it easy for for project owners to plan iterations, edit stories, add comments, and see results. Unlike Pivotal Tracker, it allows flexible iteration scheduling and story and task assignments. Google Docs provide better simultaneous editing for multiple users and more flexible formatting, but ScrumDo makes it really easy to add and change tasks, assign them, and mark them as complete. If it managed estimated and actual hours I’d totally love it.

ScrumDo added a Planning Poker tool last March, and this has been a real time saver. It’s a web page where the scrum master selects a story to size and everyone puts down a story point card. Once you play your card you can see the others, and after discussion and reaching consensus, the scrum master assigns the winning value to the story. What set of cards to use is one of the ScrumDo project settings (we use modified Fibonacci).

ScrumDo Planning Poker interface

The Value of Agile Project Management

The planning and standup meetings required by an agile approach do take time. For the UMP project, the meetings and project management activities combined used almost 40% of the total budget. But what is the alternative? Written requirements “thrown over the wall” to developers are a recipe for failure. The customers and the developers have to talk to each other to understand the project and its development trade offs. I’ve never had a customer complain that I’m using too much project management time when I’m running a project in this way. The value of open and frequent communications and iterative development cycles is very clear to them. They get full ownership of the project, choose exactly what it will deliver at every step along the way, and in the end they are delighted to get the system they envisioned.

Connexions Developers Blog: Connexions Dev Update for Jan. 20, 2012

noreply@blogger.com (Ed Woodward) — Fri, 20 Jan 2012 14:32:24 +0000

Lots of info to share since last week's update was missed.

CCAP

We tested the Physics book PDF generation and the Word Importer changes in QA last week. The code was using FOP and Docbook to create more professional looking PDFs. Our plan was to release this code and in a couple of weeks release the Prince XML version of the PDF generation. The urgency was caused by the lack of harddrive space on the development server where the content is being created. We have now fixed the space issue so it was decided not to release the FOP version of the PDF code. We are focused on the new Prince XML version.
Documentation - We have some new styles needed for the Sociology book to our documentation of the new markup.
Coding is continuing on the PrinceXML version of the PDF. You can see a sample of the Sociology layout at http://mountainbunker.org/~schatz/sociology.pdf. It has Physics content, but the color scheme of Sociology.
Some minor issues have been brought up by our content team regarding Math display in the Physics book. Some of this will be fixed in the Prince XML code, but others will have to be fixed when we do the migration to production by tweaking the MathML generated by the Word Importer.

CNX Conference Preparation

As the CNX Conference approaches, we are preparing for the sprints that will happen after the conference.
The Bug List for the Sprint has been completed.
Documentation for the Sprint has started.

Donate Button Change

The donate button now has a minimum donation of $1. The original minimum was $10. We hope this will lead to more donations.

iPhone App

An Apple Developer Program for iOS was finally purchased last week. It has taken months for all the documentation problems to get ironed out.

Consortium Tech Committee Meeting

The Connexions Consortium Tech Committee held its monthly meeting last week.
Discussions included current Connexions dev work, OERPub API update, Sprint topics and other member updates. Check the minutes for more details.

OERPub API

We are currently testing a new release of the OERPub API which uses Sword to add and modify Connexions content.
The latest release allows users to add content to a Lens, add Featured Links to a module and to create a collection in a Workspace or Workgroup. The code should be released early next week.
The latest documentation for the API is at https://trac.rhaptos.org/trac/rhaptos/wiki/TechnicalDocumentation/Code/Sword

HTML Editor Discussion

We have been talking within our team and on the Rhaptos list about the possibility of using HTML5 Microdata in the new WYSIWYG editor to document non-HTML elements in CNXML in the editor.
HTML5 has some traction because of support from schema.org, but it also has problems such as very limited browser support.
The thread on the Rhaptos list has more information.

Four Digits: Fingers crossed during vote

Wed, 18 Jan 2012 10:44:32 +0000

This week, the community decides wether the Plone Conf 2012 will be held in Arnhem or Paris. Two companies turned in a prososal, so we have a 50/50 chance of winning the vote. If all goes well, the decision will be made public on monday the 23th of january.

In october or november, the 2012 edition of the Plone Conference will take place and one thing is certain: it will be in Europe. Four Digits wrote a proposal and so did Abstract and Fanelli Consulting from Paris, France.

The procedure: from 16 until the 20 of januari, the foundation members give their advisory vote. After that, the board of directors meet to discuss the result. They will make the formal desicion, although it is expected that the board follows the advice of the foundation members. As said, monday 23rd of januari is D-Day for both Arnhem and Paris

The members as well as the Board are facing a challenge; both Arnhem (The Netherlands) and Paris (France) are great cities with a rich history. Two great venues for a conference.

As for now, it is out of our hands. We are convinced that the community will pick the right place for the conference and no matter what the result is: Four Digits is going to be there.

Jorgen Modin: How to make a massive Zope/Plone undo through the web

Mon, 16 Jan 2012 12:03:49 +0000

Summary:

http://localhost:8080/manage_UndoForm?PrincipiaUndoBatchSize:int=660

....where 660 should be replaced with the number of transactions you need to go back. Then click like crazy.

I had the need to back a Zope site in time several months, October 2011 to be specific.

I had deleted a page (after its due date) for a Plone course December 2011 and kept one for December 2007. I did not, repeat not, want to enter everything that has changed in the contents of that course in between those two dates again. There are backups for the entire system but they are rotated, and I caught this late.

The course page is a compound content type consisting of sub objects for dates, pricing and other things. The missing course page had resided in a folder hierarchy that had also been deleted. The easiest would therefore be to just back the entire server in time, rather than trying to find specific edits to undo.

The undo page in the ZMI batches undoable transactions in lots of 20. My plan was to revert everything back to October 2011 (working on a copy of course). So, 20 transactions is not enough. I need a mega batch! I know there are command line scripts were I can find the byte offset for the date I am interested in and then use truncate, but haven't done it for ages (ok, once in 2002), and can the effect still be achieved from the ZMI? It will be an append but that's OK.

If you look at the "Earlier transactions" and "Later transactions" links, you can see that the start and end transactions are encoded as cgi parameters. The batch size is in other words, not hard coded. So, if the url for "Later transactions" on the second page is

http://localhost:8080/manage_UndoForm?first_transaction:int=0&last_transaction:int=20&PrincipiaUndoBatchSize:int=20

We can just change those numbers. So I clicked through "Earlier transactions" until I reached October, noted where I were in transaction numbers and did something like this:

http://localhost:8080/manage_UndoForm?first_transaction:int=0&last_transaction:int=660&PrincipiaUndoBatchSize:int=20

I got a big page and I did a lot of clicking of check boxes, and hit "Undo" - worked like a charm. I then copied over the oh so precious text to the production server.

"PrincipiaUndoBatchSize" I did not notice until now, it did not seem to overrule the other numbers; I did not change that one.

Hmmm, must go back and try something.... Yup, this works too:

http://localhost:8080/manage_UndoForm?PrincipiaUndoBatchSize:int=660

Hector Velarde: Setting the right permissions on your blobstorage directory

Fri, 13 Jan 2012 19:08:00 +0000

Have you ever been annoyed by a message saying that the blobstorage directory of your instance has an insecure mode setting?

That happens to me all the time, so today I spent a couple of minutes trying to figure out how to fix it.

In ZODB/blob.py we have the following:

class FilesystemHelper:
    # Storages that implement IBlobStorage can choose to use this
    # helper class to generate and parse blob filenames.  This is not
    # a set-in-stone interface for all filesystem operations dealing
    # with blobs and storages needn't indirect through this if they
    # want to perform blob storage differently.

    …

    def create(self):
        if not os.path.exists(self.base_dir):
            os.makedirs(self.base_dir, 0700)
            log("Blob directory '%s' does not exist. "
                "Created new directory." % self.base_dir)
        if not os.path.exists(self.temp_dir):
            os.makedirs(self.temp_dir, 0700)
            log("Blob temporary directory '%s' does not exist. "
                "Created new directory." % self.temp_dir)

        if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
            layout_marker = open(
                os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
            layout_marker.write(self.layout_name)
        else:
            layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
                          ).read().strip()
            if layout != self.layout_name:
                raise ValueError(
                    "Directory layout `%s` selected for blob directory %s, but "
                    "marker found for layout `%s`" %
                    (self.layout_name, self.base_dir, layout))

    def isSecure(self, path):
        """Ensure that (POSIX) path mode bits are 0700."""
        return (os.stat(path).st_mode & 077) == 0

    def checkSecure(self):
        if not self.isSecure(self.base_dir):
            log('Blob dir %s has insecure mode setting' % self.base_dir,
                level=logging.WARNING)

Then, the only thing you need to do is run chmod 700 var/blobstorage (owner can read, write and execute) in your installation directory.

Why this directory is created with a different setting (755) is a mystery to solve another day.