There have been some interesting posts recently on the challenges faced by Microsoft in developing Vista. It has of course fairly recently made it out of the door which in itself is a major achievement. However, it has become clear that many features originally scheduled for inclusion have been dropped (WinFS for example). The sheer challenge of developing such a vast system is immense, and a key part of any such development effort is the change, configuration and release management.

Some fairly recent blog postings have highlighted some of the issues involved, and seeing as SCM is right in the mix, I though it worth a review.

Overview of Vista - Paul Thurrott's Supersite for Windows

The sheer size and scope of Windows Vista makes it difficult to review, to digest, and to understand. If you step back too far, it doesn't look very impressive at all: It's like XP with a spit-shine. But if you get too close, it's easy to get lost in the seemingly never-ending lists of new features.

A previous article by Paul bemoaned various lost features.

Joel Spolsky wrote an article How many Microsofties does it take to implement the Off menu? reflecting on UI choices and how less is usually more.

This attracted a comment (or was it coincidental): Moishe Lettvin's Windows Shutdown Crapfest posting:

The most frustrating year of those seven was the year I spent working on Windows Vista, which was called Longhorn at the time. I spent a full year working on a feature which should've been designed, implemented and tested in a week..

The key SCM-related part of Moishe's post is:

In small programming projects, there's a central repository of code. Builds are produced, generally daily, from this central repository. Programmers add their changes to this central repository as they go, so the daily build is a pretty good snapshot of the current state of the product..

In Windows, this model breaks down simply because there are far too many developers to access one central repository. So Windows has a tree of repositories: developers check in to the nodes, and periodically the changes in the nodes are integrated up one level in the hierarchy. At a different periodicity, changes are integrated down the tree from the root to the nodes. In Windows, the node I was working on was 4 levels removed from the root. The periodicity of integration decayed exponentially and unpredictably as you approached the root so it ended up that it took between 1 and 3 months for my code to get to the root node, and some multiple of that for it to reach the other nodes. It should be noted too that the only common ancestor that my team, the shell team, and the kernel team shared was the root.

Joel Spolsky then chimed in:

Of all the things broken at Microsoft, the way they use source control on the Windows team is not one of them...

When you're working with source control on a huge team, the best way to organize things is to create branches and sub-branches that correspond to your individual feature teams, down to a high level of granularity. If your tools support it, you can even have private branches for every developer. So they can check in as often as they want, only merging up when they feel that their code is stable. Your QA department owns the "junction points" above each merge. That is, as soon as a developer merges their private branch with their team branch, QA gets to look at it and they only merge it up if it meets their quality bar.

I think there are various interesting things about this discussion.

Managing Complexity and Reducing Dependencies

From the days of The Mythical Man Month: "Adding manpower to a late software project makes it later", it has been clear that the challenges of managing large scale systems are many. The whole book is a joy to read, and the Wikipedia articles referenced give a good highlight.

I particularly like the idea of Conceptual Integrity to help keep things simpler both in development and control.

Microsoft's products would appear almost by design to have a very high level of interdependencies between them (although this is changing: "And what happened is as the projects got larger and larger, we introduced too many complex interdependencies on early software, more so than we could really digest throughout the system," said David Treadwell, corporate vice president of the .Net Developer Platform group).

One advantage of the dependencies is that if your customer buys one product then they pretty much have to take a slew of accompanying products too.

Branches and Sub-Branches - Your SCM Tool

Back to Joel's comment that the best way to organise things is branches and sub-branches. This is a fairly classical SCM approach to the problem which has many benefits.

It is interesting to note that Joel mentions SourceDepot which Microsoft uses internally, and which is a re-badged version of Perforce circa 1999 (when win2k was in development), with some Windows specific improvements such as memory usage, but from what I can gather no fundamental algorithm improvements. Perforce of that era only had the ability to easily propagate changes between directly related parent-child branches. Things like grand-parent <-> grand-child propagation skipping over intervening parent were possible but tricky - the tool didn't default to handling it. (Note that I am interested in the original comment about there being a tree of repositories, as opposed to a tree of branches - if the former, then it makes things much more difficult to improve merging across separate repositories).

Microsoft's new Team Foundation Server tool (uses a similar branching model and terminology but totally re-architected it would appear), does not yet support good common ancestor detection either.

Over the years Perforce has addressed this in various ways, and from 2005 it is working well and with reasonable performance at large sites. Thus in Perforce you can pull changes from one sub-branch to another without going via the parent, and changes propagated back to the common parent will not cause major problems for either sub-branch. There still remains a very large question as to whether "uncontrolled" propagation of changes is a good idea (as Laura Wingerd puts it - "why we don't allow driving through hedges" - Ch7), and my advice is certainly to think carefully and plan your normal strategy (with tightly locked down exceptions maybe permissible). It still comes back to managing complexity: the more you allow changes to be propagated all over the place in an ad-hoc fashion, the harder it is to track what has gone one.

As mentioned by Joel, Accurev is a very interesting looking tools and their stream browser certainly looks attractive, and has raised the bar in terms of features for the SCM vendors. My impression is that Perforce still has the advantage in terms of scalability (see Google paper) and performance, but I would certainly put Accurev on my evaluation list if advising a client in this area.

The 3rd BCS CMSG Conference is now open for registrations with a good lineup of speakers, and an excellent looking venue.

A number of people commented on problems getting to Cambridge (not so convenient from various parts of the UK), so hopefully Oxford will fit the bill a little better. In addition, we can up the budget a little for a slgithly more commercial venue.

Some other differences this time around:

• PB Projects is doing the bulk of the heavy lifting in terms of organising details etc - yes it costs us, but it seems to be working fairly well. and takes some (though not all - e.g. sorting out timetable/speakers) of the load off the committee.
• Only just realised that offering a group discount might help bring in a few more people - we more interested in having more people there than making a vast profit (haven't yet managed to get the committee fact finding trip to the Caribbean off the ground to spend our current surplus, but am working on it...)

So, the first registrations are already in, and look forward to plenty more!

Meanwhile, am hopefully getting a little breather to allow rather more blogging than of late - have a stack of topics piled up awaiting that spare moment!

I had some fun with the Agile SCM Column at CM Crossroads this month:

A Christmas (Configuration) Carol (Abridged)

We have endeavored in this Ghostly little tale, to raise the Ghost of an Idea, which shall not put our readers out of humour with themselves, with each other, with the season, or with us. May it haunt their houses pleasingly, and no one wish to lay it.

Their faithful Friends and Servants

C.D., R.C., B.A & S.B.

Charlie's Ghost

In which the ghost of (Configuration) Charlie appears to our heroine SCM Sally to help her in her dealings with VP of Development Ben ('Benezer) Scrooge.

Continued...

Sometimes inspirations strikes, though it is interesting what turns out in the end - had a couple of different ideas (and a lot more quotations from the original), but it evolved into what you see. Not quite as provocative as a previous column:

National Treasures of Agile Development

Introduction

Recent research has discovered a very interesting cache of papers about a little known Tribe called the Agile Developers. The first document is a draft dated July 4th (the year is illegible).

(Document starts)

When in the Course of human events, it becomes necessary for some developers to dissolve the bands which have connected them with others, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.

We hold these truths to be self evident,

Continued...

Have been meaning to do some work on P4Python recently, and the first thing I realised I should do is to update the test harness.

This is based around the unittest module so is fairly standard Python. Does some fairly standard things with test suites etc, and provides a reasonably good example of how to use the code itself, so acting as some level of documentation of P4Python.

The old version assumed a pre-existing Perforce server installed and running with some known content. This was fine for my own personal testing but had a couple of problems:

• it assumed the training repository which is fine for Perforce Consulting Partners but which isn't available to ordinary mortals (so they couldn't run the test harness locally)
• was a snapshot with existing users and client workspaces so required a license to use - not good for everyone
• it required too much manual setup before running

Obviously a candidate for tidying. So recent work done:

• Change to use the new Perforce Sample Repository which anyone can download and install (works without a license too)
• Change to automatically create and run a new server instance on a fresh install from the sample repository download (automatically unzip etc)

The end effect is fairly nice and automated, and provides a much better ongoing resource to anyone wishing to do work on P4Python. Please note that it currently assumes Windows environment, but will insert a few checks to make platform independent shortly.

To have a look you can either:

• download the complete P4Python sources
• browse the test module
• just browse the Perforce Public Depot using your favourite method: //guest/robert_cowham/perforce/API/python

Note that credit for various aspects should go to Ravenbrook from their work on P4DTI - I stole various techniques from their implementation of a test harness.

It also serves as a reasonable example of a test harness for Perforce scripting, and am very happy to receive comments and improvements (there are no doubt some Python gurus out there who can suggest some improvements at points in the code). I use something fairly similar as a test harness for the VSStoP4 scripts which are actually written in Perl (too horrible to write such a thing in Perl was my feeling)!

Would like to encourage people to take their own copies and give back some improvements in the general framework at least...

This is a personal review of the week in Wales with Inaba sensei and other teachers organised by Tetsushinkan dojo. It is not a blow by blow account of everything that took place, but a personal summary with various highlights of things that were important for me.

First a quick overview:

• An advanced guard of the Japanese contingent arrived on Thursday (not including Inaba sensei)
• Udegawa san taught the normal Friday evening class at Tetsushinkan
• Saturday everyone decamped the 4-6 hours to Wales (this included Inaba sensei and Endo san who only arrived that day from Poland where they had been teaching the previous week). Various other overseas visitors flew in that day and also drove down.
• Sunday was the official opening and first practice
• Standard schedule during the week
  • Morning practice 9.30 - 12.30
  • Afternoon activities in groups ranging from canoeing to horse riding to cliff walks
  • A couple of Instructor's sessions 6-7.30pm for invited people (others free to watch)
  • A couple of meetings with instructors to review links with Shiseikan, including future courses
• On Tuesday evening we greeted Toyama guji (the Chief Priest of the Meji Jingu shrine who was "on tour" in Europe) with his entourage to Wales. He came to watch practice on Wednesday and kindly hosted a meal on the Wednesday evening for an invited group.
• Final practice on Friday, with an excellent party that evening in a restaurant in Newport
• Return to London on Saturday
• Japanese all left Sunday

Summary

It was a truly excellent week. I came away with masses of food for thought and further reflection regarding both life and budo studies. It was a great pleasure to renew old friendships and to make some new friends. I also feel a renewed personal challenge to do more with these insights!

The teaching started with a formal opening ceremony conducted by Mori san, a priest from the Meiji Jungu shrine and included prayers and harai tachi - cleansing of the dojo and the spirits of those present. Inaba sensei talked about his wish to demonstrate the importance of this aspect as part of his teaching. Shinto is a relatively recent name for practices which go back thousands of years. The gods and spirits come and sit in the branches of the himorogi, part of the small shrine set up during practice. He does not view this as a religious practice and certainly has no wish to convert everyone to Shintoism! However, he wished to demonstrate the importance of setting the tone for the week - in a way a sort of misogi for all those present.

Those of you who have seen Inaba sensei will know that talking and lecturing form quite a part of his teaching. This can be a bit frustrating at first for those used to other styles of teaching where much more emphasis is placed on practice. Many of Inaba sensei's messages keep recurring over the years, but as often happens, various things made more sense to me this time - maybe I have changed in the meantime and so now hear them differently!

Why Are you Practicing?

As with the initial ceremony, this was something Inaba sensei returned to at several points. We should understand why we are practising - what do we wish to protect or defend? What are we prepared to do in the face of attack, for example on our loved ones? Answers to this question inform all the rest of our studies, and thus really are fundamental. Study and thought will be well repaid.

Relationship with Kunii Sensei

Inaba sensei gave some more details on his relationship with Kunii Sensei (the 18th generation Headmaster of the Kashima Shinryu school). This relationship was fairly short - 17 months I believe as a direct student before his death in 1966. Inaba sensei started studying aikido in the early 1960s at Hombu (and with Yamaguchi sensei). Being allowed to study with Kunii sensei was not easy, but was enabled by an introduction from Ashizu sensei, a Shinto philosopher that Inaba sensei was also studying with (and indeed that association went on for nearly 30 years). The relationship with Kunii sensei was obviously very close and Inaba sensei mentioned being asked by Kunii sensei to pass what he learnt through direct transmission and also the results of his own independent study. Inaba sensei obviously feels a very strong obligation as a result.

Developing the Body

This has long been a key concern - developing our bodies as a foundation for studies in budo. This time around there were copies of Inaba sensei's book on the subject (English and Japanese in the one volume). The English title is "Researching Japanese Budo". It originated in a series of articles for a Japanese magazine in the early '90s, and I was involved in helping revise the translation - I hasten to add that this meant reviewing the resulting English and offering some suggestions for clarifying unclear points or reworking phrases which appeared a little clumsy. The real work was done by Endo san and Annika Hansen. The original translation was done by Yamada sensei and Diane Zingale, of which a number of people received photo copies nearly 10 years ago.

Anyway, pressure of time meant that it still isn't quite where it might be in the English translation, but certainly full of very useful and interesting information - perhaps will review separately. I suggest contacting Shiseikan directly if you wish to get a copy.

Energy Inherited from your Ancestors

This was a new concept for me, and something I did not find easy to get a handle on in a physical sense. Intellectually to me it makes sense to talk about what we inherit from our direct ancestors and also the energy of the place we are in, but how to do anything physical with such energy was not clear. The exercise demonstrated was basically kokyu-ho (e.g. both hands grasped and lifting your hands and thus your partner in front of you). The difference as shown between this and normal ki energy was not clear to me in any practical way, so very much one for further research.

Kenjutsu Kata

I was very interested to see demonstrated some more dynamic ways of studying kenjutsu. Inaba sensei discussed the various kata and went through the details of the Uradachi (second set of 10 techniques) in some detail. However, as well as the classic form of the kata with uke and nage, we were also shown more dynamic forms, for example where uke more or less attacked in a kendo style and the response need to be much more freeform than in the set version of the kata. I was particularly interested as I had been thinking prior to the course that my own kenjutsu risked being a little formulaic - falling into the pattern of anticipating the technique and only being able to respond in the fixed form - even getting annoyed when the attack wasn't "just right"! Some teaching of the ukemi side were also very useful for me and others who are in positions to teach. It is important to ensure that you are applying appropriate pressure to your partner to draw their best out from them.

He did mention the requirements to study each set of kata (Kihon, Ura, Aishin, Jissen, Kassen) properly before moving on to the next, but also to place them in a context of being able to react flexibly.

Inaba sensei went through the Uradachi in some detail as part of one of the instructor's courses. Particular points of note included the requirement for both sides to treat the shinai as a real sword and thus heavy contact by uke is not appropriate - you can't lean on the blade of a katana! Normally nage leads by taking the first step, although it may appear that both people start together. It is uke's responsibility to respond to nage.

Kesa Giri Footwork

I don't think anyone present will forget the outdoors practice of kesa giri - the location was at Clydey Cottages - a place with a great lawn and stunning view over the valley. Inaba sensei really demonstrated the difference between short, restricted, inhibited kesa giri and the more expansive cuts which really started to connect with the nature all around us. The difference that this made in everyone's kesa giri was very marked, partiularly the following day back inside - you could see the expansiveness still present.

This moved on also to the use of the feet. For practical reasons of the ground being perhaps unstable, or uneven, or on a slope, he emphasized the need to step with the feet fully before completing the cut. It is easy in the clean confines of the dojo to get used to cutting with the landing of the foot and the focus point of the sword cut occurring at the same point. Outdoors however, this doesn't work - as soon as you step you risk slipping and sliding, so it is important to have fairly firm footing before completing the cut.

Battojutsu (Sword drawing)

This was another memorable day outside (though slightly more rainy) focussing on the drawing of the sword. We were all using bokuto (wooden swords) for practice but he did discuss the use of katana (live blades). He does not recommend practice with iaito (metal swords with blunt blades), as these tend to give one a false sense of security. A katana will "teach" you if you let it. It requires much more focus and attention which is good for practice and concentration. In a way, you should not use a katana too much since familiarity may breed contempt and thus lack of focus.

Various exercises and cutting to different sides (front, back and both sides) were demonstrated and practiced. In practice you shouldn't just get into the habit of thinking "OK, now I will pretend someone is attacking from behind, so let's react". It should be a much more instinctual response - wait and be alert, and the right moment will arise. Remember also flexible footwork - react off either foot.

Stepping off the Line

This was another exercise completed outside. It is one I have seen and practiced several times over the years, and yet realised this time around that I really need to focus much more on it for my own practice.

You start a couple of paces away from uke, and then step in so as to "invite" an attach (standard punch to the stomach). As the punch comes in you need to step off the line so that you end up in a triangle with feet pointing at your opponents centre. If uke attacks right hand (and foot), then it is quite normal to initially half step with the right foot to "invite", and then step off the line with the left foot closest to uke. See Figure 1.

The alternative which we practised involves "inviting" with a half step forward of the left foot. The result is similar, but left and right feet are reversed.

This appears quite simple but is difficult to do well, and in particular getting the timing correct and being able to move your body appropriately is key. Watching Inaba sensei, he doesn't necessarily appear to move particularly fast, but it is at just the right moment and with appropriate speed.

Further developments of this exercise shorten the distance to one step, then half a step, then no steps (with uke's hand on your belly). Challenging stuff!

The Grading

This happened at the end of the week during the last session ranging from shodan to yondan (errrr that would be me...). Most people were from Tetsushinkan but some others from London and Greece, and everyone generally did very well. Results are on the Tetsushinkan events page.

As is typical with Inaba sensei's gradings they were relatively short. In each case various attacks were specified and the actual technique was not specified - indeed it was perfectly acceptable to perform the same technique several times in a row to the same attack. The important factors are spirit, body movement and effectiveness, rather than necessarily technical beauty. People also demonstrated some of the Kashima Shinryu kenjutsu kata according to the knowledge and experience of the person being graded (e.g. Kihondachi or Uradachi). This approach is certainly different to what is normally done within many standard aikikai  dojos. I did discuss this afterwards over a beer with a couple of people - "but how do you know they know all the appropriate techniques?" was one question. I obviously can't speak for Inaba sensei but I think that it is usually clear from a demonstration the sort of level a person has reached, and it is perfectly in tune with his general approach of studying relatively few techniques but in great depth. In addition, I think that this sort of approach works well in smaller organisations where the people being graded are known to the examiners - I think there might be more problems with larger scale organisations.

Instructor's Meeting - The Future

There were a couple of interesting discussions with vairous instructors and countries present, e.g. Bjorn Eirik Olsen and others from Norway, Pascal Durchon and Joel Roche from France, Anita Kohler from Germany and various others from UK etc. As with all good discussions, some heat and light at times was eventually turned into some good results and conclusions.

Inaba sensei was keen to get an understanding of how he could best support those present, and what sort of future courses and training was both sought and might be possible. He did mention that his preferred method is to teach a small group of people at the Shiseikan where he could give a lot of time and attention to them, and he made a generous offer for those prepared to arrange visits. Pascal and Anita in particular expressed a desire to give more people the opportunity to get to know Inaba sensei and experience his teaching. The rough plan for 2007 is for a seminar in France to be organised primarily by Pascal, followed by Norway in 2008 and Germany in 2009. Visits to Shiseikan are left to individual groups to arrange. I think the intention for Paris is to perhaps offer open sessions where anyone can attend, but also a more restricted session for those with previous experience and who have already made contact with Inaba sensei and are already studying his methods. This would allow for more advanced training, in particular for kenjutsu. I think it will be very interesting to see how this works, and certainly look forward to Paris next summer!

Conclusion

The course was excellent - lovely surroundings and a great atmosphere. Serious study and yet not in a heavy atmosphere. The number of seniors around (from Japan and elsewhere) made for some excellent training.

Inaba sensei is a fascinating and inspiring teacher on many levels, and it is worth taking any opportunity to get around him and learn from him. Some of his methods are a little extreme perhaps, and I am not clear in my own mind how easy it is to combine what he teaches with classic aikikai aikido. The differences go to quite deep levels. This is fine for those of us who primarily follow his teaching. Others who think it might be nice to spice up their traditional methods may find it distinctly more challenging. How much integration is possible and how much is it two separate approaches - I suspect it is going to take me at least quite a few more years to decide. In the meantime he is planning for the future. He is not going to go on for ever - seek him out and take advantage of him!