Robert Cowham's Weblog

Review of BCS Agile SCM Event on 24 November 2008

This is a brief review of the BCS CMSG Agile SCM event on 24th November 2008.

Please note that the presentations are on line from the link above. Below are some extra notes to go with the presentations.

All in all a very good and informative day!

Michael Azoff (The Butler Group) - The Agile Difference

Presentation here

Michael did a good job of setting the scene and introducing various terms around agile etc.

He talked about the wider context - and how people need to stay engaged in projects. Quoted an example of a US Healthcare provider where management did not get sufficiently involved in the new billing system because it was not "exciting enough". The company had sufficient problems that it lost 2/3 of its value, and ended up having to pay out $200m to clients etc.

Such things promote the need for a dashboard - simple summary of project status with the ability to drill down. Data is always available.

Regarding SCM he noted nothing in the indexes of various books on Agile methods that relates to SCM - an oversight!

However, came up with a couple of useful links:

Henrik Kniberg's Version Control for Multiple Agile Teams
Also his Scrum and XP from the Trenches
Henrik's Blog

Richard Erwin (Microsoft) - SCM and Agile Practices within Microsoft Developer Division

Presentation here

Another excellent talk and the slides well worth reading. Interesting to see Microsoft's "dog fooding" of their own product and the benefits and improvements of their process between Visual Studio 2005, 2008 and 2010 (already out in previews).

Some points:

Most companies using adhoc processes, and having problems getting a handle on all the information that is coming in. Showed off some of the dashboards and similar feedback options within VSTS.
Within Microsoft there are 4 major divisions:
- Windows
- Office
- Developer Tools
- SQL Server
Their largest instance is 17m source files, 500k items, 3 Tb data
VSTS 2010 has been "dog fooding" since 2007. Problems don't go unnoticed as a result - he mentioned the great summer '08 outage!
There is a slide showing usage of VSTS within the various divisions. Developer Tools is completely VSTS. However Windows and Office are still using SourceDepot for version control (although using VSTS being used for Planning and Bug Tracking). Richard responded to one question that the internal tools (SourceDepot) were "not available" outside Microsoft. On further prompting, he admitted that SourceDepot was a rebadged version of Perforce - based on the source code that Microsoft bought in 1999 - one of those "industry secrets" that has leaked out over the years!
A successful import from the Office division is the concept of "Feature Crews"
Concept of Bug Debt - carry no debt in feature model. One of the slides shows the burn down chart from 25,000(!) bugs for VS 2005 � they did get it close to zero for release, but life was hellish! The comparable chart for VSTS 2008 is much flatter.
Not surprisingly the cost of hotfixes or sevice packs is massive - its a major incentive to avoid those - hence major quality gates when teams push code back to the mainline.

Branching Model

It's a fairly standard Mainline model with large team branches. There's a huge amount of automated testing. Teams take it in turns to "publish" to the Mainline, and have to pass very significant quality gates to do so.

Andrew Tunnicliffe (London Underground) - Towards Visualizations of Configuration Management

Presentation here

Andrew wanted to leave us with the message - "Configuration Management isn�t just for Christmas!"

Some notes:

His presentation was a little different in that it looked at the complexities of managing what according to the Royal College of Engineers is the most complex system in the UK.
Software is very much only one element - in addition to hardware, staff/people are vital. This includes training needs etc - you can't introduce new stuff without it. Takes 2 years to train 100 drivers. How do you manage change in that time?
In one morning they went from 4 trains to 5 trains per period � this is a step change of 25% in one go - you can't introduce 10% of a new train :)
S/w control and signalling all changing.
He called it saftety critical agile.
Of course in a public project of this size, there are huge political pressures to deliver.
A lot of the value in visualisation is for people at the edges rather than those in the trenches. Train driver just needs to know that something has changed.

The examples he showed us are a tool based around MS Project with various extra attributes, and a Network diagram tool. It can also publish to the web.

Interesting to see how something relatively simple and straight forward could make a significant difference to acceptance by people - the CM information is much more easily visible.

Finbarr Joy (Upco) - All Things to All Men: Keeping it simple?

Presentation here

Finbarr wanted to avoid any perception of giving a "sales pitch", inspite of Upco's sponsorship of the event - paid for a nice lunch!

The main focus of his talk was around build and deployment frameworks - making it easy to setup, customize and manage.

He mentioned the experience of Upco in doing development for their clients and wanting to scratch their own itch and improve their processes. The only drawback for me was the lack of specifics in the talk - it would have been useful to see some before and after figures showing how this approach has worked in practice. A couple of useful nuggets:

Is your change management really change prevention � or perceived as that?
Projects change a standard pattern many times which leads to incompatible processes and much rework.

He finished by pointing at the open source project Kundo which is well worth checking out:

Kundo provides a structured, convention based approach for Java builds. Kundo has a pluggable, extensible architecture; it harnesses the power and flexibility of Groovy and Ant to provide a highly configurable Java build framework.

Kundo achieves this flexibility with a plug-in architecture that attaches behaviours (provided by Kundo plug-ins) to build lifecycle phases. Kundo consists of a kernel and a set of foundation plug-ins. The kernel is responsible for the orchestration of the multiple collaborators within the build system.

Conceptually similar to the approach taken by the Apache Maven project, the Kundo implementation is simpler (the kernel library jar file is ~ 60Kb) and, in our humble opinion, offers greater flexibility; if you want to simply wire an Ant into a buildfile and use it, you can. Build lifecycles are defined within a build 'recipe'. A recipe declares the plug-ins required to perform a build. Each Kundo plug-in, much like a Maven plug-in, encapsulates a discreet set of build management (or deployment, release management etc etc) logic and has its own versioning/release cycle.

Sean Cody (Bank of America) - SCM � the Agile Keystone

Sean's talk has unfortunately not yet had the slides approved to be released externally. It was an excellent talk and well received.

He talked about some of the background and issues of both SCM (software configuration management) and Agile.

In particular for agile:

Not always clear what it means
Is it a project management approach or just technical practices?
Fragilism?!
Image problem:
- Hard to manage and control
- Too many cowboys
- An excuse for no documentation
- "Decline and Fall of Agile"

He used the metaphor of the keystone which is vital to arches, domes and other architectural designs. A lack of good SCM renders Agile ineffective.

Key components for Bank of America:

Story management server
SCM server
Continuous Integration server
Testing server
Wiki for publishing results

Story management is the most important function for users. The key thing is to track the evolution of a story from inception to release. This means the following are all tied to stories (with no exceptions):

All SCM check-ins
Build records
Testing records

The business can see the details of the stories and indeed click through to view all aspects of development should they so wish. This visibility allows development to prove its value to the business. It also makes for easy traceability - audit is not a problem.

Sean is keen on tools that are easy to integrate, even though customisation comes with the cost of maintenance. For the SCM tool the following are mandatory:

Atomic check-ins
Change history
Performance
Reliability
Simple to use

These aren't new requirements - but they even more important for Agile development. They (together with the framework) help address some of the criticisms of Agile:

Lack of structure
Lack of documentation
Management of scope creep
Meeting enterprise change requirements
Meeting audit and regulatory requirements

He left with a word of warning: cultural change is hard to implement - tools can only do so much - your people are important!

Custom Perforce GUi Installers (for large sites)

For quite a while there has been an option to create your own custom installer for Perforce deployments (including automation). This can be particularly useful for larger sites with lots of new users coming along regularly.

The customizable element of the scripted installer is a configuration file named perforce.cfg. The Perforce Administrator edits this file, and places it on a shared network drive along with the expanded contents of the perforce.zip file. Perforce client programs can then be installed by running setup.exe from each desktop.

One of the annoyances with this approach is that you have to expand the .zip file and put things on a network drive. It is not easy to do this sort of thing within a single executable for example (which makes it hard to put on your intranet and get users to double click).

Well, having played around with various installers and options, I have discovered a fairly simple way of doing this via a self extracting executable (SFX), and the open source 7-Zip program.

Instructions

Create a directory structure:

some root dir
- 7-zip - contains a batch file, a couple of config files and perforce.cfg - see below for contents.
- p4winst - contains expanded p4winst.zip (without the perforce.cfg)
- p4vinst - contains expanded p4vinst.zip (without the perforce.cfg)

Currently (as of 2007.2) the perforce.cfg file is the same for both P4Win and P4V.

You need to get a couple of files from 7-zip.org/download.html

Download 7-Zip executable and install, say in d:\apps\7-Zip (as referenced in batch file)

Download 7z Library, SFXs for installers, Plugin for FAR Manager anand unzip to the 7-Zip install directory mentioned above.

Edit perforce.cfg to customize for your installation.

Create a p4winst.conf and p4v.conf (which must both be UTF-8 - Notepad can save this format, as can Notepad++ or other editors) in that directory.

Run make_installers.bat from the command line in the appropriate directory and check that the following are created (which you can put on an intranet or whatever):

p4winst.exe

p4vinst.exe

You can then send your users a single exectuable which unzips itself and automatically runs the perforce setup.exe with the included perforce.cfg - QED!

make_installers.bat

This contains something like the following:

:: Batch file to package up P4Win using .zip file and the 7-Zip freely available 
:: Zip program (which can create SFX - Self Extracting Archives). 

@echo off 

:: May need to customize the following 
set ZIP_DIR="d:\apps\7-zip" 

call :make_install p4winst 
call :make_install p4vinst 
goto :exit 

:make_install 

set INSTALLER=%1 

:: First create new clean versions of the zip files (including current directory version of perforce.cfg in preference) 
if exist %INSTALLER%.7z del %INSTALLER%.7z 
if exist %INSTALLER%.exe del %INSTALLER%.exe 

%ZIP_DIR%\7z a %INSTALLER%.7z ..\%INSTALLER%\* perforce.cfg

:: Copy SFX file plus config file plus zipped installer into a single SFX .exe
copy /b %ZIP_DIR%\7zsd.sfx + %INSTALLER%.conf + %INSTALLER%.7z %INSTALLER%.exe 
if not errorlevel 1 goto :EOF 
if exist %INSTALLER%.exe goto :EOF 

echo *** ERROR: failed to create %INSTALLER%.exe 
goto :exit 

:exit

p4winst.conf (and p4vinst.conf)

These are simple UTF8 format file with contents similar to:

;!@Install@!UTF-8!
Title="P4Win Custom Installer"
BeginPrompt="Do you want to install P4Win?"
RunProgram="setup.exe"
;!@InstallEnd@!

Subconf 2007 - Subversion and the Enterprise

This article is a result of my visit to Subconf in Munich a couple of weeks ago.

The first thing to say is that Subconf was very enjoyable and interesting and my compliments to the organisers. There were some excellent speakers and it was good to chat to various vendors too. I also had a very good time at the dinner that evening - the only slight fly in the ointment was the S-Bahn strike the following morning which meant a rather earlier taxi to the airport than I might have wished!

The two keynotes by Brian Behlendorf and Karl Fogel respectively were particularly good. Indeed it was good to get them at the conference since both guys seem to be scaling down their Subversion involvement - Brian has stepped down as CTO at Collabnet and Karl is running QuestionCopyright.org (although remains as President of Subversion Corporation.

Some of the slides are now available online.

Brian's Keynote

Brian mentioned his involvement in Apache and some of the lessons learnt about creating a community that worked. These lessons were then applied by Collab.net to get Subversion going. If you create a high quality community, you will create high quality software.

Need to be nice to people to avoid a "fork" - Development leaders exhibit good communication skills, and can bring different ideas together.
Conscious effort to bring new developers along the path: from �consumers�, to bug reporters, to patch submitters, to active contributors.

Brian was asked the question in his talk about Linus's somewhat inflammatory video about Git where various sideswipes about brain-dead CVS and Subversion people are made. Brian had no particular axe to grind about this and said that the Subversion camp bore no ill will to the distributed crowd and were rather surprised at some of the venom coming back - a very measured and mature response.

Brian mention the difference between a centralised model and a distributed model - there is a lot that enterprise like about the control of a centralised model. For example, laptops get stolen (he lost his a few weeks ago), and a laptop might contain a whole distributed repository. Enterprises prefer more control!

Martin Doettling's Intro to Karl

The key points of this were:

Estimated user base now exceeds 2 million
10x growth since 1.4
Large numbers of enterprise users

Karl's Keynote - How Stuff Happens

Karl picked up on some of Brian's points about the community, and showed various ways in which they have been able to get hackers to create software that the enterprise can use!

There is a very comprehensive guide to how to contribute - 43 pages of it! This is where people start.
Some principles:
- Make it easy to do things right
- Make it rewarding to do things right
- Influence proportional to effort
  - Moving from contributor to partial-committer to commiter
  - Tracking no of patches etc by user - all automated with links (see Contribulyzer)

Other Notes

I couldn't stay for the second day, but some other points of interest:

There are increasing numbers of tools based around Subversion - Collabnet, Polarion, CodeBeamer etc
Subversion command line is increasingly irrelevant as people use it through Tortoise, Eclipse etc.
Merge tracking there at long last with 1.5 just around the corner
Really being used in the industry
SVK provides an interesting distributed method

I also had some chats with Collabnet around the idea of perhaps the BCS CMSG working with them to put on an event in London next year - definitely something to look into.

Well worth a visit as a conference.

Directing vs. Managing Configurations

In a recent CM Crossroads article on defining Agile SCM I included the quote from Charles Handy that I recently picked up on my travels through an airport:

Go to the theatre and look at the programme. Everyone connected with the performance is listed, no matter how small their contribution. People like to be recognised as individuals. The word "manager" is reserved for those in charge of things, not people, the stage manager and the lighting manager. The people who are in direct communication with the customer, the actors, are directed, not managed, by someone who actually leaves the scene once the project is under way. He or she trusts the cast to go it alone, and as often as not, they improve on the production once the director departs. Trust inspires. One more thing - at the end of each performance they receive an expression of appreciation from their audience, direct feedback from the people who matter. No waiting for the annual performance appraisal.

Charles Handy, Myself and Other More Important Matters, Arrow Books, 2007 - ISBN 9780099481881

This obviously applies to more than just the directing vs. managing debate (e.g. feedback/appreciation), but really hit home for me as an expression of what those of us in the Configuration Management arena need to be doing. We should be directing our colleagues and developers rather than managing them (although we can of course manage actual configurations as before!). We need to get out and about amongst our congregation of developers and others, and infect them with enthusiasm and skills for the delights and benefits of good CM.

There is a huge danger, which I come across fairly frequently, of sitting in your ivory tower wondering why developers aren't doing what you know is "the right thing". They haven't got the message, they aren't doing what they should, and management isn't helping you to beat them up with a big stick to force them into doing it. Sometimes it seems we are just on a hiding to nothing...

In CM we need to aspire to the director (servant-leader) mold - we provide a support service for people to get on with their jobs and we help them do it, but it is they that do (most of) it, not us. There is obviously some level at which CM is still responsible for certain things - in particular processes and systems, and for auditing to ensure things have been done correctly.

Thus I am more and more convinced that, particularly for Agile teams, CM should be the responsibility of everyone not just the CM person(s). This means that as CM professionals our job is:

to sell the ideas, principles and tools of sound CM while supporting people in getting their jobs done and delivering value to the business (not just ticking CM boxes)
to solve people's day-to-day issues through education and tool/automation support
to show how it is easier to do "the right thing" than it is not to do it - a little discipline is worth a ton of lost time due to bugs introduced, or things suddenly not working
to get out and about and work with developers, pairing with them and demonstrating good CM practice, discovering those little niggling things that cause lost time and productivity or pain and solving them
to ensure that our own working practices are suitable role models (avoid "do as we do not as we say...")
to focus on continuous process improvement - there is almost always something more we can do (but again focus on value)

There are many challenges, and in an Agile world, it is more about working with people to define just enough process to be useful, and not too much to get in the way. I was fascinated to hear a vendor representative complaining that some of their customer had developers who were downloading and using Subversion for their day to day work as opposed to using the corporate tool. I wondered what had happened to make the tool or the configured process so unfriendly as to force the developers down this path.

Perforce Branch Naming and Repository Structure Guidelines

One of the the things that comes up just about every time I go in to do some Perforce related consulting for a client is that of repository structure - how do you map some particular branching scheme (combination of branching patterns) onto the repository?

This of course necessary since Perforce uses the repository structure to represent branches as well as other directory structures. This way of recording branches is feature one might occasionally wish were implemented otherwise, but it does have its attractions, and is the same model as used in Subversion and also Microsoft's new Team Foundation Server product (as an aside, it is perhaps not surprising that Microsoft chose this model since it is a not very well kept "secret" that their in house SCM tool called SourceDepot, introduced during the development of Windows 2000, was a re-badged copy of Perforce. Team Foundation Server is obviously a total rewrite on top of SQL Server, but they seem to be comfortable with the "branch in path space" model).

Some clients need help working out a good branching scheme, and some have relevant experience with other tools so are comfortable with that part, but many need help with the mapping. I have covered a little of this in my article Introduction to Perforce Branching, but this article expands on that.

For their flexibility, I would generally recommend people use branch specs (e.g. created by "p4 branch" command) for branching since you can add view mappings to address not propagating renames, and branch extra components. However, if you can keep the branch spec to a single view mapping then that is great! (Note that the P4V GUI allows integration via drag-and-drop - sometimes convenient, but can be somewhat dangerous...).

Guidelines

A repository branch structure is about creating an appropriate information hierarchy (designing an information architecture). So here are some guidelines that I have found useful over the years and seen successfully used in many organisations (and I certainly didn't invent any of them!). Maybe in the future I can come up with some appropriate pattern like descriptions, but for now, here they are:

Standardise naming conventions - make branch directories obvious
Keep branch spec mappings and client workspace spec mappings as simple as possible (and permissions)
Make it as easy as possible for users (new and experienced) to visualise the branching scheme as they click down through the hierarchy
Find the right balance between broad & shallow vs. narrow & deep hierarchies
Keep branches of a similar type at similar depths in the structure
Make branch directories sort sensibly
Plan for the future: 2, 3, 5 years down the line - include things like the year (and optionally month) somewhere in the path to provide a natural sorting.
Provide Guidelines and Delegate!
Remember - there is no one best solution, but there are many good enough solutions!

Some of what comes below is also covered by Laura Wingerd in her recent book "Practical Perforce" (O'Reilly) - you really need to get this book if you are responsible in any way for Perforce usage at your company! Her quote which sums things up nicely is:

"There is no reason that the repository structure should match the release lineage, but when it doesn�t it causes confusion to new users [and cognitive dissonance to all users]."

Now while I agree with the intent of what Laura has written, I find her suggested option is perhaps a little too restrictive, and it doesn't take into account organisations with many branches at a particular level (and indeed many different types of branches).

Standardise Naming Conventions - Make Branch Directories Obvious

Laura gives a very simple example in her book of naming convention - just use uppercase for the branch component. Thus she has:

//depot/MAIN/...
//depot/REL1/...
//depot/SOME_PROJECT/...
//depot/ANOTHER_PROJECT/...

This is simple and can work very well (although it rather suggests that you don't use uppercase for any other paths which might be unnatural in certain companies for certain items). Another alternative is to append (or prepend) something to the name indicating that it is a branch directory, e.g. ".branch" or or "-branch", or ".br" for those who don't like typing! Thus the above becomes:

//depot/main.br/...
//depot/rel1.br/...
//depot/some_project.br/...

I am happy to have other suggestions recommended!

Sensible Sorting

Directory (and thus branch) names should sort sensibly at any point in the hierarchy, thus

use leading zeros for numeric components, such as 01 instead of 1 if you are going to have more than 10, or indeed 4, 5 or 6 leading zeroes where necessary
reverse the order of dates using yyyy-mm rather than mm-yyyy

Examples:

//depot/main.br/...
//depot/rel_01_00.br/...
//depot/rel_02_00.br/...
//depot/rel_02_01.br/...
//depot/rel_02_02.br/...

//depot/dev/2007_06_some_project.br/...

Broad & Shallow vs. Narrow & Deep

At any one level the number of choices (sub-directories/branches) should be appropriate and naturally limiting (note that the structure needs to be as useable in several years time as it is now). For example, a structure which ends up with thousands of branches in a single directory will become unwieldy over time. Divide these branches into sub-directories using appropriate naming (and sorting). For example if you wish to create lots of branches for individual change requests then the obvious structure is very wide and shallow at some point in your repository:

~/rel 	/CR000001/...
	/CR000002/...
	:
	/CR000099/...
	/CR000100/...
	:
	/CR009100/... 
	:

Instead of that, try:

~/rel 	/CR0000xx	/01/...
			/02/...
			:
			/99/...
	/CR0001xx	/01/...
			:
	/CR0091xx/	/01/... 
			:

This naturally splits things up and also sorts at all levels.

As a corollary of the above you should split things up even if they don�t have a �natural� dividing component. One alternative is to use the date as an element of the path (or even a prefix) as YYYY or YYYY-MM to give some sort of natural sorting by date. Examples being:

~/2006-01/some_project/...

Or without introducing another level but using a prefix (which is often just a matter of personal preference):

~/2006-01-another_project/...

In 2 or 5 years time it should be obvious to any user which are the which are the �new� ones where they are likely to be spending most of their time, and which are the �old� branches which can typically be ignored.

Hiding or Retiring Old Branches

It is possible �hide� old branches by removing read permissions on those repository paths (leaving them visible to superusers or perhaps an "archive" user). There are some potential performance implications to doing a lot of this (the more lines you add to the protections table the more work the server has to do for commands).

Often the easiest way is just to make them easily ignored via an appropriate hierarchical naming convention as shown above.

Provide Guidelines and Delegate Responsibility

Larger organisations may have a number of teams or units each of which is responsible for products with sometimes quite different lifecycles and thus branching schemes. In these cases it makes sense to set the guidelines and principles and delegate to the teams the work of defining the precise conventions to be used.

Automation

If your repository structure is simple enough and regular enough then it is easy for tools or scripts to:

create them (perhaps with input from a user), including adding a mapping to the current client workspace view
have triggers that validate that they are being created in the correct place (e.g. don't allow a branch with wrong naming convention to be created in certain directories or levels of the repository). This prevents user error.
work out the relationship between branches automatically and thus be able to produce simple reports for things like changes that need to be propagated etc. While this is often done by means of some configuration file (stored in the repository), and indeed this is sometimes necessary anyway, it is easier if it can be automatically deduced from the structure (so that a new branch "popping up" is automatically included into the reporting mechanism).
make your protections table easier to maintain

There is No One Best Design!

And remember that this is always true - there are always many ways that are good enough, and the final choice often comes down to personal preference.

However, every type of branch should have a �home�, so that when a user creates a type of branch your CM Plan should have simple guidelines so the user is never at a loss as to where to create one. This requires identifying the types of branches likely to be used.

You need to do enough thinking and planning to get at least 80% of it right. It is of course possible to re-structure your repository (by branching from old name to new name and deleting the old within the same atomic changelist). However, this can cause you quite a few problems if the branch you are re-structuring has relationships with lots of other branches (e.g. release branches being maintained).

People who are aware of the issues and have experience, can slot in a new branch type fairly easily (though they can still end up making sub-optimal choices that they regret).