The announcement of the 2012 Plone Symposium East (thank you Weblion! always a great gathering) is reminding me of the talk I gave there last year on Agile Development with Plone. Here at Jazkarta we’re engaged in another large project, similar in many ways to the one that inspired that talk. Because our project management tools and techniques have evolved a bit since last year, I figured it was time for an update.

The Projects

Last year at this time, we were just getting started on a major website redesign for the University of Minnesota Press. There’s lots of information about this project in my Plone Conference presentation How To Get a Fabulous Website on a Modest Budget Using Plone. Briefly, it involved a major data conversion from a FileMaker database into Plone, a small number of complex custom content types, extensive design work, and a featured “explore” section powered by eea.facetednavigation.

Our new project is a major website redesign for Dumbarton Oaks Research Library and Collection, a research institute of Harvard University and the home of a little known but outstanding museum and garden in Washington, D.C. We began with a discovery phase last summer, and have been working on implementation since November.

What’s the Same

Both of these are fixed budget projects with a large number of desired features, and at the outset many of these were inadequately defined for development purposes. In both cases there are multiple stakeholders with differing priorities, but there is an intelligent, proactive project owner who is trusted and empowered to make decisions. The Jazkarta team is the same – Carlos de la Guardia in Mexico, Alec Mitchell in California, and me in Massachusetts – and none of us ever work full time on any of our projects.

This all adds up to the same basic project management challenge: how to adapt agile techniques to manage a fixed price, flexible scope project with a distributed, part time team.

What’s Different

Happily, the Dumbarton Oaks project does not involve a complicated data import from a FileMaker database. For UMP this required a collective.transmogrifier script, many many many trial and error imports, and over 60 hours of Jazkarta development time. Dumbarton Oaks currently has a static HTML website, which is much simpler to import. Funnelweb made this easy – only a few trial imports and some content massaging were required. We’re not a quarter through the project but all the old content is already on staging. Total development effort: 10 hours.

Because the content import was relatively easy and the project mostly requires custom features and content types that are independent of each other, we have been able to divide up the work into several releases. This gives us a series of smaller planning horizons and allows us to be more agile (more on this later). Each release can potentially be deployed to production – and even if we decide not to, new content is developed on staging which gives the new features a thorough vetting. No matter how much you thought you tested, there are always problems that don’t get discovered until the software sees serious use. Having a series of small production deployments avoids piling all the bug fixes and theme tweaks into a single mega deployment phase at the end of the project.

Our Evolving Agile Development Practices

Our basic development process remains the same:

1. Discovery
2. Define schedule and resources (how the budget will get spent)
3. User story development
4. Story estimation (we use planning poker)
5. A series of development iterations alternated with evaluation periods
6. Production deployment

During discovery the project owner and I distributed the feature requirements into a series of releases, based on a combination of priorities and keeping all the stakeholders happy. (Everyone got something in almost every release.) This high level release plan allows us to do our story development and estimation one release at a time, which saves time and money – especially considering that all the developers participate in planning poker sessions. Since we know the customer wants more than their fixed budget will get them, and we also know they will change their mind about what they want as the project progresses, it makes sense to postpone the effort of writing and estimating stories we may never get to.

As in previous projects, I took the first cut at turning the project requirements into stories, then discussed and revised the stories with the project owner. Because I have intimate knowledge of Plone, I can translate requirements into stories that make sense to develop. For example, we don’t need to spend time defining stories for CMS functionality or for things that Plone has great add-ons for (like staff directories and bibliographies). Some things may turn out to need tweaking, but it is better to give the customer stock Plone and add-on features to try out first. They may decide to live with the stock features and put their development dollars into new features instead of making changes.

Our communication tools remain the same: Skype, IRC, chat, Google Calendar, and Google Documents. This combination is the secret sauce that allows a distributed team to work as if they were in the same room. We did switch from ClueMapper (based on svn and trac) to github for source code repository and ticket tracking. We miss a few management features, but the developers are much happier working in a dvcs.

Our development iterations remain the same – one week each, followed by a week of evaluation during which the customer has time to test and reflect on what they want to do next. We kick off each iteration with a planning meeting, which surfaces any questions about the stories we’re tackling and allows us to do task breakdown and assignment while we are all in the virtual room of a Skype conference call. During the iteration we have a short standup meeting every day to keep things on track – everyone says what they did yesterday, what they’ll do today, and raises any blocking issues. My notes from the standups are written in a Google spreadsheet which is shared with everyone. This gives the project owner access to detailed information about how things are going, if they miss a standup meeting.

The biggest change we’ve made for this project is to adopt ScrumDo for story development and iteration planning. ScrumDo is an open source agile project management tool that’s written in Python. We are using the hosted, SaaS version that’s at scrumdo.com – accounts are free for up to 10 users and 10 projects. ScrumDo has a friendly drag and drop UI that makes it easy for for project owners to plan iterations, edit stories, add comments, and see results. Unlike Pivotal Tracker, it allows flexible iteration scheduling and story and task assignments. Google Docs provide better simultaneous editing for multiple users and more flexible formatting, but ScrumDo makes it really easy to add and change tasks, assign them, and mark them as complete. If it managed estimated and actual hours I’d totally love it.

ScrumDo added a Planning Poker tool last March, and this has been a real time saver. It’s a web page where the scrum master selects a story to size and everyone puts down a story point card. Once you play your card you can see the others, and after discussion and reaching consensus, the scrum master assigns the winning value to the story. What set of cards to use is one of the ScrumDo project settings (we use modified Fibonacci).

ScrumDo Planning Poker interface

The Value of Agile Project Management

The planning and standup meetings required by an agile approach do take time. For the UMP project, the meetings and project management activities combined used almost 40% of the total budget. But what is the alternative? Written requirements “thrown over the wall” to developers are a recipe for failure. The customers and the developers have to talk to each other to understand the project and its development trade offs. I’ve never had a customer complain that I’m using too much project management time when I’m running a project in this way. The value of open and frequent communications and iterative development cycles is very clear to them. They get full ownership of the project, choose exactly what it will deliver at every step along the way, and in the end they are delighted to get the system they envisioned.

This week, the community decides wether the Plone Conf 2012 will be held in Arnhem or Paris. Two companies turned in a prososal, so we have a 50/50 chance of winning the vote. If all goes well, the decision will be made public on monday the 23th of january.

In october or november, the 2012 edition of the Plone Conference will take place and one thing is certain: it will be in Europe. Four Digits wrote a proposal and so did Abstract and Fanelli Consulting from Paris, France.

The procedure: from 16 until the 20 of januari, the foundation members give their advisory vote. After that, the board of directors meet to discuss the result. They will make the formal desicion, although it is expected that the board follows the advice of the foundation members. As said, monday 23rd of januari is D-Day for both Arnhem and Paris

The members as well as the Board are facing a challenge; both Arnhem (The Netherlands) and Paris (France) are great cities with a rich history. Two great venues for a conference.

As for now, it is out of our hands. We are convinced that the community will pick the right place for the conference and no matter what the result is: Four Digits is going to be there.

I had the need to back a Zope site in time several months, October 2011 to be specific.

I had deleted a page (after its due date) for a Plone course December 2011 and kept one for December 2007. I did not, repeat not, want to enter everything that has changed in the contents of that course in between those two dates again. There are backups for the entire system but they are rotated, and I caught this late.

The course page is a compound content type consisting of sub objects for dates, pricing and other things. The missing course page had resided in a folder hierarchy that had also been deleted. The easiest would therefore be to just back the entire server in time, rather than trying to find specific edits to undo.

The undo page in the ZMI batches undoable transactions in lots of 20. My plan was to revert everything back to October 2011 (working on a copy of course). So, 20 transactions is not enough. I need a mega batch! I know there are command line scripts were I can find the byte offset for the date I am interested in and then use truncate, but haven't done it for ages (ok, once in 2002), and can the effect still be achieved from the ZMI? It will be an append but that's OK.

If you look at the "Earlier transactions" and "Later transactions" links, you can see that the start and end transactions are encoded as cgi parameters. The batch size is in other words, not hard coded.  So, if the url for "Later transactions" on the second page is

http://localhost:8080/manage_UndoForm?first_transaction:int=0&last_transaction:int=20&PrincipiaUndoBatchSize:int=20

We can just change those numbers. So I clicked through "Earlier transactions" until I reached October, noted where I were in transaction numbers and did something like this:

http://localhost:8080/manage_UndoForm?first_transaction:int=0&last_transaction:int=660&PrincipiaUndoBatchSize:int=20

I got a big page and I did a lot of clicking of check boxes, and hit "Undo" - worked like a charm. I then copied over the oh so precious text to the production server.

"PrincipiaUndoBatchSize" I did not notice until now, it did not seem to overrule the other numbers; I did not change that one.

Hmmm, must go back and try something.... Yup, this works too:

http://localhost:8080/manage_UndoForm?PrincipiaUndoBatchSize:int=660

class FilesystemHelper:
    # Storages that implement IBlobStorage can choose to use this
    # helper class to generate and parse blob filenames.  This is not
    # a set-in-stone interface for all filesystem operations dealing
    # with blobs and storages needn't indirect through this if they
    # want to perform blob storage differently.
    …
    def create(self):
        if not os.path.exists(self.base_dir):
            os.makedirs(self.base_dir, 0700)
            log("Blob directory '%s' does not exist. "
                "Created new directory." % self.base_dir)
        if not os.path.exists(self.temp_dir):
            os.makedirs(self.temp_dir, 0700)
            log("Blob temporary directory '%s' does not exist. "
                "Created new directory." % self.temp_dir)
        if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
            layout_marker = open(
                os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
            layout_marker.write(self.layout_name)
        else:
            layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
                          ).read().strip()
            if layout != self.layout_name:
                raise ValueError(
                    "Directory layout `%s` selected for blob directory %s, but "
                    "marker found for layout `%s`" %
                    (self.layout_name, self.base_dir, layout))
    def isSecure(self, path):
        """Ensure that (POSIX) path mode bits are 0700."""
        return (os.stat(path).st_mode & 077) == 0
    def checkSecure(self):
        if not self.isSecure(self.base_dir):
            log('Blob dir %s has insecure mode setting' % self.base_dir,
                level=logging.WARNING)