Robert Cowham's Weblog 17 to 21 of 43 articles Syndicate: full/short

BCS CMSG News and Events for 2006   19 Mar 06
[print permalink all comment ]

Tools Fair on 15th June - Cradle to Grave Support

As Chair of the BCS CMSG now (Shirley Lacy passed on the baton in November but fortunately for me remains in the background as vice chair), I have been feeling a little concerned about responsibility for the BCS CMSG Tools Fair - Cradle to Grave Support on 15th June. I am relieved that things are looking good now with good selection of sponsors supporting us:

Gold sponsors
  • Marval
  • Frontrange
  • Touchpaper
  • Square Mile Systems
  • MKS
  • Serena
  • Aldon
Standard Sponsors
  • Perforce
  • Telelogic
  • Axios
  • Accurev
  • Unicom
  • SpectrumSCM
     

Some newish names to me not being hugely up on the service management/ITIL side, but change, configuration and release management are a major focus in that field so look forward to meeting and hearing about them and the issues and solutions available.

A couple of big names missing on the software side which is a shame. A couple of organisations undergoing a re-org and thus no marketing budget to spend (well not now anyway). IBM/Rational aren't there because I can't find anyone to talk to that would appear to be able to make a decision. We used to have good relations with Rational but since they were acquired by IBM can't get anything out of them (I wonder what their responsiveness is if I were looking for support or trying to buy a produce?!). If anyone knows who to contact in the UK on marketing/events I'd be grateful for a heads up. The other one I have failed to get to anyone appropriate is Microsoft - would be good to find out about Team System etc, but all enquiries get passed from pillar to post and a deafening silence is the result.

Show me the money… identifying the value of Configuration Management

The other event we have on the 27th April is an evening event which is a relatively new departure for us. This will be at London South Bank University in their conference centre (which we have used before), and is in the form of a workshop lead by David Cuthbertson. This is going to focus on selling the benefits of CM and we hope will be a useful way of bringing together both new and more experienced people in the field to share experiences. Details to go up on the web site very shortly! David has spoken several times in the past and always gotten excellent reviews. He is also skilled at running workshops and getting contributions from others present. Should be a great evening. Apologies to those not close enough to London who want to attend, but we are looking at running (in particular evening) events elsewhere in the country - get in touch if you have any ideas.

Perforce Automatic Merging   16 Mar 06
[print permalink all comment ]

Updated: 2005-11-29 - see link to scripts.
Updated: 2006-03-16 - some clarifications and link to Miki Tebeka's scripts

A common branching pattern is to have mainline and then task branches where work is done and then "published" by merging to the mainline as shown in Figure 3 of the article Building for Success.

In perforce the "publish" and the "catchup" are both performed by using the integrate command, typically with a branch spec. For example, you might have a branch spec

Branch:	task/fred
View:
	//depot/main/... //depot/task/fred/...

The key thing is that Fred should "catchup" before doing the "publish". This is so that the more risky merging is done in his task branch and not in the mainline. When doing the merge Fred should be bringing all changes by other people in the project into his branch and with the publish he should just be able to copy his code into the mainline.

There are several ways to achieve this:

  • Education - tell Fred what to do and rely on him doing it - so what happens if he doesn't - can you "persuade" him not to?!
  • Don't give Fred write access to the mainline (e.g. via protections or a trigger), and instead have the integration team do it. The problem then being that the integration team may not know the code as well as Fred and are perhaps more likely to make a mistake.

The basic way of detecting in Perforce whether a catchup has been done is to do a preview integrate from main to task/fred and check that nothing needs to be done, so check for no results from:

p4 integrate -n -b task/fred

If the above produces no results then proceed to do the publish:

p4 integrate -r -b task/fred
p4 resolve -as

The key step is that the "resolve -as" (safe automatic merge), resolves as many files as it can. It looks to see if their are only "theirs" (source) diff chunks or "yours" (target) diff chunks and will select the theirs or yours file appropriately. (Note that "theirs" and "both" or "yours" and "both" are also processed in the same way). The key point is that if there are "theirs" and "yours" (or "conflict") diff chunks, then safe automatic resolve will not process that file.

Of course having done the automatic resolve and with the changed files sitting in our client workspace it is usually a good idea to do things like a build and smoke test - there's not a whole heap of trust otherwise...

Thus, in our script we can check for anything not safely resolved automatically (resolve -n shows what still needs to be resolved).

p4 integrate -r -b task/fred
p4 resolve -as
p4 resolve -n
if any results from above command then exit with error

build
if any problems then exit with error

run smoke tests
if any problems then exit with error

If no errors at this point we are ready to submit.

There are some extra wrinkles to this:

  • the "resolve -as" may validly not work if you have done a merge with edit during the catchup ("edit from" in the terminology from p4 integrated). You need to detect such situations and do a "resolve -at" to copy them over.
  • as soon as one person has done a publish then all other branches will require to do a catchup to pull it in - this means publishes are going to become serial
  • there is a window during which the publish is being performed when someone else might sneak in and do a publish thus meaning you need to do a catchup - consider a simple "locking" strategy to prevent this.

Note that the build and smoke test steps need to take an appropriate amount of time. If it takes hours to do them then the process is unlikely to work. Thus a few minutes or tens of minutes is likely to be the limit - this may mean cutting down on the number of tests that are executed - but that is usually OK. Apply judgement!

Although at the last point you might think it would be a brave person to do the submit automatically in a script! It's probably a good idea to do things like run diffs and give a visual once over before finally checking in, but it should be a pretty easy decision at this point.

In terms of automating the above, I can heartily recommend the various scripting languages with built-in calls to the Perforce API: Ruby, Python and Perl. Getting the results of a command is easy, and exceptions allow you to make error handling pretty easy as well.

Very brief examples of scripts designed to perform the above check in Python and Ruby are now available online. Note they are designed to be run from Custom Tools menu in p4v/p4win. Please note that Miki Tebeka has kindly published more production quality script implementations in python including a GUI front end. He also includes an example of an installer to automate installation of tools for p4win and p4v - check them out. Thanks Miki!

p.s. The above works for any codeline for which you have responsibility for accepting changes - it doesn't have to be a mainline - it can be a subsidiary integration line with third parties contributing, or team members "proposing" changes. You could automate the script and put it on an intranet page - let people try it out, and if successful then have changelists auto-checked in.

Gravity and Budo   23 Feb 06
[print permalink all comment ]

I have been meaning to blog on my budo (martial arts) practice (Aikido and Kashima Shinryu Kenjutsu), and thought I'd start with the topic of gravity. I am working my way to potential links between budo and subjects like software development and configuration management, but don't want to get too airy fairy and draw invalid analogies, so for now will just keep the topics separate and perhaps see what emerges.

It took me a while to realise that when he "discovered" gravity old Isaac Newton was helping out budo practitioners! Gravity is our friend in budo, and I find it particularly true of kenjutsu (sword work).

Try holding your sword out with one hand and then just dropping it. If you're on the same planet as me then it will drop to the ground pretty reliably and consistently :) After having dropped it a few times, try holding on while you "drop" it. This means that you drop your whole body and hand together with the sword, in such a way that the sword falls at the same speed as if you had just let it go. In my experience people find this surprisingly difficult at first. You have to relax your knees and hips in particular and just drop. Best practiced on a mat to protect those knees a bit. Most people can't let go of their leg muscles and get in the way of gravity - they interfere with the sword dropping. Once you are really getting the feeling for it you can start ever so gently making a cut whilst dropping the sword. This I find is very useful when working out the cut for Kihondachi number 2 (Ashibaraiukefune)..

A variation is to kneel in seiza and just hold your sword out in front of you balanced vertically upright. Let it overbalance and just fall to the mat. Try that a few times, and then do the same thing while holding on to it. Make sure you are just letting it fall. Relax and just drop the sword and your hand together.

Another interesting exercise from standing is kesa giri (diagonal cut). I find this helps to get a much smoother cut than just giving people a bokuto (wooden sword) and showing them how to cut. In that case it tends to be all arms and shoulders. With left foot forward slightly hold the bokuto in your right hand only and just swing it gently up and rest it on your left shoulder. From that position just "pull" it down in a very natural manner so that it just falls to the ground in an arc (you keep holding the sword so the tip strikes the ground - again mats help here to protect the sword). An alternative is to have a partner just hold a bokuto low down and across the path of yours so that you naturally strike it (in a way they catch your sword). Once you are comfortable with doing this in a totally natural and relaxed manner, letting gravity do all the work, you can then try holding it with both hands in the normal way and doing the same thing. This is usually substantially more difficult - people mess up the perfectly natural stroke with muscle!

Of course gravity also relates to balance, particularly of swords - finding and being in contact with the balance of your sword at all times while moving it. Another key related factor is relaxation which I will talk more about another time. Also the relationship between power, relaxation, speed and control.

Cheng Hsin. I recently got a copy of his DVD which is well worth watching. Also, read his book Effortless Power.

Karel Koskuba, who teaches Yiquan, also puts it interestingly when he discusses mobiliser muscles and stabiliser muscless. The stabilisers are the ones that allow us to stand up against gravity and are tremendously powerful. If we learn to use them more we can then access much greater strength than just through the more conscious muscles of the arms say.

So let gravity do as much as possible and really learn how to harness it, particularly in sword work. Requires quite a bit of sensitiviy and slow practice to really be aware of how your sword is moving, how it wants to move and what is happening in your body.

Fast Perforce Checkpointing   03 Feb 06
[print permalink all comment ]

There was an interesting discussion not too long back on how to do fast check-pointing for your server.

The basic procedure with check-pointing for backups is shown in the System Administrator's Guide. For larger sites it can take tens of minutes, getting up to hours sometimes, which becomes inconvenient if you have limited windows for backup due to people working in different time-zones etc.

As an aside on the -z option to zip a checkpoint while backing up - it is worth checking for your server hardware the performance of the CPU overhead of zipping vs. the writing to disk of the checkpoint. Thus in some circumstances it might be worth check-pointing and zipping as you go, and in others zipping offline.

With thanks to Chris Bartz who posted it to the Perforce User mailing list in such a well documented fashion:

Okay, here are the gory details. I can't take credit for inventing it; Perforce tech support gave me most of the details and I'm pretty sure others are doing very similar things. To bootstrap the process you need to create an offline database. This is done by:

1) use "p4 counter journal" to get journal counter value. The checkpoint name will be checkpoint.<journal counter+1>.
2) "p4 admin checkpoint" (or "p4d -jc" if you prefer)
3) Optional. Zip and backup the truncated journal file
4) Delete old offline database db.* files
5) Build offline database with "p4d -r <offlineDir> -jr <checkpoint>"
6) Zip and backup checkpoint

We do the above steps once a week so that we start each week with a fresh offline database. We currently keep all the journals between rebuilding the offline database so we could recover from a real checkpoint plus journal files if there was some problem with the offline database.
Rebuilding and keeping all the journals in between isn't really required but when I set it up I wasn't 100% confident in the whole process. If I were making other changes to the process I would probably go with once a month rebuilds and maybe not keep all the journals.

The offline checkpoint is done daily with:

1) use "p4 counter journal" to get value of journal counter
2) Truncate journal file with "p4d -r <root> -jj <journal filename>" This creates a files <journal filename>.jnl.<journal counter> and starts a new journal file
3) Read truncated journal into offline database with "p4d -r <offline root> -jr <journal filename>.jnl.<journal counter>"
4) Optional. Zip and backup journal file
5) Checkpoint offline database with "p4d -r <offline root> -jd <checkpoint>.<journal counter + 1>". The journal file + 1 is so it has the same name as perforce would give it if we checkpointed the live database.
6) Optional. Zip and backup checkpoint
7) Optional. Delete old checkpoints and journals (we keep all journals between rebuilding the offline database and 3 checkpoints).

When this is done we have a checkpoint and journal file that should be exactly the same as if we did the "p4 admin checkpoint" on the live database. There is essentially zero downtime (except the weekly rebuild).

The offline database could be on another machine and the checkpoint could be done there if disk space or processing power were an issue.

The depot files are backed up after this process is done. We do not shut perforce down for that backup. You really don't need to; what perforce does to handle this is simple and does work.

Another note on the final remark - imagine the metadata (as restored from the checkpoint + journal) has information about 9 revisions of a file, and due to the backup having happened a little time after the checkpoint (and journal being a little out of date), and yet the RCS format archive file actually contains 10 revisions. The server will carry on fine. Obviously the opposite is not true (metadata has 10 revisions and archive file only 9). In both cases, if there is some inconsistency you will potentially lose some work, but most recent activity will be stored in people's workspaces (and they may remember any changelists they have recently submitted).

Thus your disaster recovery scenario needs to include what happens when you get your server back online and what people need to do (e.g. Tech Note 2 - Working Disconnected). Sally Page of Symbian gave an excellent presentation to the UK User Group on Symbian's DR experience and lessons learnt.

Review of 59 Minute Scrum   02 Feb 06
[print permalink all comment ]

I went to the 59 Minute Scrum  session last October which sounded interesting - from the blurb:

"This session will give attendees a unique opportunity to experience agile practices first hand in a non-technical environment."

I had good expectations but was left a little under-whelmed at the actual event. Now I have to admit that I arrived late and missed the intro so maybe it's all my fault and the intro answered everything! (Discussion over a beer afterwards showed I wasn't totally alone though).

The format was to split up into teams and discuss various scenarios. These were implementation projects including marketing, programmes and anything else we fancied on topics such as: a theme park around "spam", a health club for pets and a space tourism project.

The event was fairly tightly timetabled so we had a slot to discuss our backlog, then split up further in our teams to actually come up with ideas on particular areas. Then round 2 for more implementation. Finally, all the teams (8 if I remember correctly - typically 2 per idea) presented their solutions to the whole group and touched briefly on how well they had addressed the backlog/made progress etc.

There was then an all-too-brief Q&A session at the end.

For me the format didn't really work. I was left with too many questions and faffing about in the exercise had very limited benefit. I would much rather have had a well though through presentation on Scrum with lots of time for questions. At most half an hour of some exercises and then a presentation and lessons learnt would have worked much better for me.

Maybe I am just becoming a stick-in-the-mud?! I am likely to stear clear of such sessions in the future though unless there is time for a solid hour or more afterwards to get into details and ask questions.

 

Copyright © 2008 Robert Cowham