Robert Cowham's Weblog 15 to 19 of 43 articles Syndicate: full/short

Review of Pratical Perforce (Part 1)   06 Apr 06
[print permalink all comment ]

This is a partial review of Practical Perforce by Laura Wingerd, published by O'Reilly (ISBN 0-596-10185-6). The reason it is partial is that I intend to comment in more detail in future blog articles on some parts of the book, and wanted to post this without waiting for the whole thing!

As Laura mentions in the preface, the book is not intended for complete beginners, but more for readers with experience in other SCM (software configuration management) tools who are looking to understand how Perforce works.

To quote the introduction, there are two parts to this book:

  • Part I (Chapters 1-6) is a whirlwind technical tour of Perforce commands and concepts. It's not a tutorial, nor a reference, but helpful nonetheless.
  • Part II (Chapters 7-11) describes the big picture, using Perforce in a collaborative software development environment. It is particularly strong on branching patterns, how to structure codelines and tips and tricks in this area.

The real meat of the book for most Perforce sites is thus Part II, but there are definitely some goodies in Part I.

Chapter 1 presents some fundamentals about Perforce syntax and concepts. The diagrams on pages 6 &7 explain the relationship between revisions and changelists very well.

In Chapter 2, Laura discusses client workspaces and things like view syntax. She also describes basic check outs (open for edit in Perforce command line parlance), and introduces branching when she refers to cloning of files. She includes concepts of renaming and replacing content in files, reconciling changes made offline, and even introduces a couple of bits of undocumented syntax such as "p4 files @=1452". Quite a chunk of information in this chapter.

Resolving and Merging are the subject of chapter 3 and includes some very useful diagrams showing various scenarios. If you have ever had any questions about 3-way merging in Perforce - read this! On pp68-69 she gives examples of reconciling changes you have made to a file someone else renamed using the undocumented merge3 command - interesting if a touch esoteric (also referred to in "How to undo a merge" on p80. The recommendation on p74 to sync and resolve one changelist at a time is certainly worth considering, although I think it will depend on your environment as to how necessary that is.

The basics of branching are covered in chapter 4 including initial scenarios and how to track merge requirements across branches. She makes quite a lot of use of the interchanges command (not yet exposed in the GUIs) and explains the gory details of "yours", "theirs" and "base" nicely. Her approach of using filespec integrations for the initial examples is nice and simple, but I suspect more people are likely to use branch specs in real life. On p111 she gives a useful couple of commands to show how to find which changes have been merged in (more likely to be automated in scripts for most sites I would expect). Other subjects ocvered include all the gory details of what integrate actually does, as well as some very useful details as to what the interchanges command can tell us, particular with respect to cherry picked integrations.

Chapter 5 is quite short on labels and jobs and shows all the basics. A quick note on the final section where a job is used as a reference for a changelist - as of release 2005.2 there is an undocumented "dynamic label" option where a label can have an attached revision which probably makes the job trick unnecessary.

Chapter 6 gets into the subject of remote depots and proxies and also mentions the very useful spec depot option (automatic versioning of all spec objects). There is also some good advice on using p4web in browse mode to access your repository. The section on triggers and automation is a little light, but understandable.

Part II starts with Chapter 7 "How software evolves". This chapter is perhaps the highlight of the book, and introduces concepts that are totally independent of Perforce and apply to many SCM tools. Fortunately the chapter is currently available as a free PDF document from the O'Reilly website for the book. A firm understanding of the concepts introduced here will make it much easier for you to come up with suitable branching patterns for use in your organisations, and also, perhaps more importantly, give you some incredibly useful concepts for explaining your structure to other people within the organisation. Most SCM problems are due to poor communication rather than poor tools, or poor ideas. Laura relates the problems in the real world prevent us from an overly simplistic ideal world, and yet how some simple concepts allow us to manage this real world complexity. The "flow of change" and the "tofu scale" are classic concepts which should be in everyone's SCM vocabulary.

Summary

I am going to stop this post here, and will get to further chapters and some detailed comments on them as I have time.

But I will finish with the recommendation - buy this book!!

Regular Expressions in Ruby and Python   27 Mar 06
[print permalink all comment ]

A personal foible perhaps but I do find Ruby's regular expression syntax remains in my brain much more easily than the Python equivalent.

Maybe it's the Ruby inheritance from Perl that makes the difference. For simple scripts I can just write standard regexps without any recourse to documentation and they just work! For example:

some_var = "prefix interesting_match some suffix"
if some_var =~ /prefix (\w+) some suffix/
  interesting_bit = $1
  print "Match found: ", interesting_bit, "\n"
end

In order to do the same in Python I find myself faffing around with the documentation (using ActiveState Python it's great to have a proper help file, but I would really like more links between the class reference and real examples of usage to help me out) and trying to remember if I want re.search or re.match and how do I get a match group and use it, etc. I have sundry Python scripts floating around that I open up and copy relevant examples from, but it does rather take time.

import re
some_var = "prefix interesting_match some suffix"
pat = re.compile('prefix (\w+) some suffix')
m = pat.match(some_var)
if m:
    print 'Match found: ', m.groups(0)[0]

Now I have to admit that it's not a huge deal in terms of the resulting code, but it took me 5-10 minutes just now to code and debug the Python version as opposed to the Ruby version which I typed in and which worked first time.

The net result is that I am noticeably  more productive in Ruby for those little scripts that make life easier, or when I am under strict time pressure. Now this is not to say that I don't like Python, or indeed that when I have a little more time I don't get use it and enjoy it. Having done some reasonably significant work in Python, e.g. rework P4DTI for PVCS (now Serena) Tracker I feel reasonably qualified to comment.

I also took the time to get sufficiently proficient in Python extensions to enhance and maintain P4Python. Mind you I now feel somewhat humbled by the most recent efforts at a Perforce integration (PyPerforce) - shows a depth of Python extensions knowledge before which I can only bow in admiration! (Minor aside - Ruby extensions are much easier to write than Python ones due mainly to the different garbage collection models).

Finishing up, I am definitely Perl-averse these days. There are a few Perl scripts that I maintain and can't be bothered, or can't find a convincing "business" case to rewrite, but anything new is Ruby or Python.

BCS CMSG News and Events for 2006   19 Mar 06
[print permalink all comment ]

Tools Fair on 15th June - Cradle to Grave Support

As Chair of the BCS CMSG now (Shirley Lacy passed on the baton in November but fortunately for me remains in the background as vice chair), I have been feeling a little concerned about responsibility for the BCS CMSG Tools Fair - Cradle to Grave Support on 15th June. I am relieved that things are looking good now with good selection of sponsors supporting us:

Gold sponsors
  • Marval
  • Frontrange
  • Touchpaper
  • Square Mile Systems
  • MKS
  • Serena
  • Aldon
Standard Sponsors
  • Perforce
  • Telelogic
  • Axios
  • Accurev
  • Unicom
  • SpectrumSCM
     

Some newish names to me not being hugely up on the service management/ITIL side, but change, configuration and release management are a major focus in that field so look forward to meeting and hearing about them and the issues and solutions available.

A couple of big names missing on the software side which is a shame. A couple of organisations undergoing a re-org and thus no marketing budget to spend (well not now anyway). IBM/Rational aren't there because I can't find anyone to talk to that would appear to be able to make a decision. We used to have good relations with Rational but since they were acquired by IBM can't get anything out of them (I wonder what their responsiveness is if I were looking for support or trying to buy a produce?!). If anyone knows who to contact in the UK on marketing/events I'd be grateful for a heads up. The other one I have failed to get to anyone appropriate is Microsoft - would be good to find out about Team System etc, but all enquiries get passed from pillar to post and a deafening silence is the result.

Show me the money… identifying the value of Configuration Management

The other event we have on the 27th April is an evening event which is a relatively new departure for us. This will be at London South Bank University in their conference centre (which we have used before), and is in the form of a workshop lead by David Cuthbertson. This is going to focus on selling the benefits of CM and we hope will be a useful way of bringing together both new and more experienced people in the field to share experiences. Details to go up on the web site very shortly! David has spoken several times in the past and always gotten excellent reviews. He is also skilled at running workshops and getting contributions from others present. Should be a great evening. Apologies to those not close enough to London who want to attend, but we are looking at running (in particular evening) events elsewhere in the country - get in touch if you have any ideas.

Perforce Automatic Merging   16 Mar 06
[print permalink all comment ]

Updated: 2005-11-29 - see link to scripts.
Updated: 2006-03-16 - some clarifications and link to Miki Tebeka's scripts

A common branching pattern is to have mainline and then task branches where work is done and then "published" by merging to the mainline as shown in Figure 3 of the article Building for Success.

In perforce the "publish" and the "catchup" are both performed by using the integrate command, typically with a branch spec. For example, you might have a branch spec

Branch:	task/fred
View:
	//depot/main/... //depot/task/fred/...

The key thing is that Fred should "catchup" before doing the "publish". This is so that the more risky merging is done in his task branch and not in the mainline. When doing the merge Fred should be bringing all changes by other people in the project into his branch and with the publish he should just be able to copy his code into the mainline.

There are several ways to achieve this:

  • Education - tell Fred what to do and rely on him doing it - so what happens if he doesn't - can you "persuade" him not to?!
  • Don't give Fred write access to the mainline (e.g. via protections or a trigger), and instead have the integration team do it. The problem then being that the integration team may not know the code as well as Fred and are perhaps more likely to make a mistake.

The basic way of detecting in Perforce whether a catchup has been done is to do a preview integrate from main to task/fred and check that nothing needs to be done, so check for no results from:

p4 integrate -n -b task/fred

If the above produces no results then proceed to do the publish:

p4 integrate -r -b task/fred
p4 resolve -as

The key step is that the "resolve -as" (safe automatic merge), resolves as many files as it can. It looks to see if their are only "theirs" (source) diff chunks or "yours" (target) diff chunks and will select the theirs or yours file appropriately. (Note that "theirs" and "both" or "yours" and "both" are also processed in the same way). The key point is that if there are "theirs" and "yours" (or "conflict") diff chunks, then safe automatic resolve will not process that file.

Of course having done the automatic resolve and with the changed files sitting in our client workspace it is usually a good idea to do things like a build and smoke test - there's not a whole heap of trust otherwise...

Thus, in our script we can check for anything not safely resolved automatically (resolve -n shows what still needs to be resolved).

p4 integrate -r -b task/fred
p4 resolve -as
p4 resolve -n
if any results from above command then exit with error

build
if any problems then exit with error

run smoke tests
if any problems then exit with error

If no errors at this point we are ready to submit.

There are some extra wrinkles to this:

  • the "resolve -as" may validly not work if you have done a merge with edit during the catchup ("edit from" in the terminology from p4 integrated). You need to detect such situations and do a "resolve -at" to copy them over.
  • as soon as one person has done a publish then all other branches will require to do a catchup to pull it in - this means publishes are going to become serial
  • there is a window during which the publish is being performed when someone else might sneak in and do a publish thus meaning you need to do a catchup - consider a simple "locking" strategy to prevent this.

Note that the build and smoke test steps need to take an appropriate amount of time. If it takes hours to do them then the process is unlikely to work. Thus a few minutes or tens of minutes is likely to be the limit - this may mean cutting down on the number of tests that are executed - but that is usually OK. Apply judgement!

Although at the last point you might think it would be a brave person to do the submit automatically in a script! It's probably a good idea to do things like run diffs and give a visual once over before finally checking in, but it should be a pretty easy decision at this point.

In terms of automating the above, I can heartily recommend the various scripting languages with built-in calls to the Perforce API: Ruby, Python and Perl. Getting the results of a command is easy, and exceptions allow you to make error handling pretty easy as well.

Very brief examples of scripts designed to perform the above check in Python and Ruby are now available online. Note they are designed to be run from Custom Tools menu in p4v/p4win. Please note that Miki Tebeka has kindly published more production quality script implementations in python including a GUI front end. He also includes an example of an installer to automate installation of tools for p4win and p4v - check them out. Thanks Miki!

p.s. The above works for any codeline for which you have responsibility for accepting changes - it doesn't have to be a mainline - it can be a subsidiary integration line with third parties contributing, or team members "proposing" changes. You could automate the script and put it on an intranet page - let people try it out, and if successful then have changelists auto-checked in.

Gravity and Budo   23 Feb 06
[print permalink all comment ]

I have been meaning to blog on my budo (martial arts) practice (Aikido and Kashima Shinryu Kenjutsu), and thought I'd start with the topic of gravity. I am working my way to potential links between budo and subjects like software development and configuration management, but don't want to get too airy fairy and draw invalid analogies, so for now will just keep the topics separate and perhaps see what emerges.

It took me a while to realise that when he "discovered" gravity old Isaac Newton was helping out budo practitioners! Gravity is our friend in budo, and I find it particularly true of kenjutsu (sword work).

Try holding your sword out with one hand and then just dropping it. If you're on the same planet as me then it will drop to the ground pretty reliably and consistently :) After having dropped it a few times, try holding on while you "drop" it. This means that you drop your whole body and hand together with the sword, in such a way that the sword falls at the same speed as if you had just let it go. In my experience people find this surprisingly difficult at first. You have to relax your knees and hips in particular and just drop. Best practiced on a mat to protect those knees a bit. Most people can't let go of their leg muscles and get in the way of gravity - they interfere with the sword dropping. Once you are really getting the feeling for it you can start ever so gently making a cut whilst dropping the sword. This I find is very useful when working out the cut for Kihondachi number 2 (Ashibaraiukefune)..

A variation is to kneel in seiza and just hold your sword out in front of you balanced vertically upright. Let it overbalance and just fall to the mat. Try that a few times, and then do the same thing while holding on to it. Make sure you are just letting it fall. Relax and just drop the sword and your hand together.

Another interesting exercise from standing is kesa giri (diagonal cut). I find this helps to get a much smoother cut than just giving people a bokuto (wooden sword) and showing them how to cut. In that case it tends to be all arms and shoulders. With left foot forward slightly hold the bokuto in your right hand only and just swing it gently up and rest it on your left shoulder. From that position just "pull" it down in a very natural manner so that it just falls to the ground in an arc (you keep holding the sword so the tip strikes the ground - again mats help here to protect the sword). An alternative is to have a partner just hold a bokuto low down and across the path of yours so that you naturally strike it (in a way they catch your sword). Once you are comfortable with doing this in a totally natural and relaxed manner, letting gravity do all the work, you can then try holding it with both hands in the normal way and doing the same thing. This is usually substantially more difficult - people mess up the perfectly natural stroke with muscle!

Of course gravity also relates to balance, particularly of swords - finding and being in contact with the balance of your sword at all times while moving it. Another key related factor is relaxation which I will talk more about another time. Also the relationship between power, relaxation, speed and control.

Cheng Hsin. I recently got a copy of his DVD which is well worth watching. Also, read his book Effortless Power.

Karel Koskuba, who teaches Yiquan, also puts it interestingly when he discusses mobiliser muscles and stabiliser muscless. The stabilisers are the ones that allow us to stand up against gravity and are tremendously powerful. If we learn to use them more we can then access much greater strength than just through the more conscious muscles of the arms say.

So let gravity do as much as possible and really learn how to harness it, particularly in sword work. Requires quite a bit of sensitiviy and slow practice to really be aware of how your sword is moving, how it wants to move and what is happening in your body.

 

Copyright © 2008 Robert Cowham