Writing good Perforce triggers, and, more importantly, debugging them in live use, turns out to be one of those things that seems simple but has lots of tricky issues that can lead to lots of time being wasted.

In spite of thinking that I understood lots of the issues, I still spent a couple of hours recently debugging a problem that turned out to be a combination of environment and password issues. This was particularly annoying as I had rather though I knew about this stuff (and indeed have advised people over the years about it!), and yet was blindsided and caught out by some issues I had forgotten about or not thought through deeply enough.

I reserve the right to revisit this subject more than once in the future with further insights and news...

Assume Nothing About The Environment!

The classic approach to triggers is to write a nice script (Python or Ruby for me these days - no Perl, though just occasionally I miss it!) and debug it by running with the appropriate parameters from the command line (e.g. create a pending changelist and pass in the pending changelist number). This does indeed tend to turn up a number of issues, but the good thing is you can usually debug them with the appropriate command (<rant> why does python require you to execute pdb.py which isn't by default put in the path on Windows machines, and why does Ruby not learn from Perl and for example use -d as a parameter to debug things instead of "-rdebug" - very unobvious!</rant>).

The major problem turns out to be the fact that the trigger is executed by the Perforce server process and may have a very different environment to what you might think as you run a "login" session. One sort of expects this on Unix, but on Windows it can be particularly surprising how little is in the environment due to the username that the Perforce process is running in when it is running as a service (default installation on Windows).

Thus the first rule of trigger writing is "assume nothing about the environment!".

It is very easy to forget this and assume very simple things, like:

• P4PORT is always defined
• P4USER is always defined
• failures of individual p4 commands within the trigger will be obvious

Thus immediate recommendations are:

• Give full pathnames to executables. For example, "/usr/bin/ruby" or "C:\ruby\bin ruby.exe" as the initial parameter for the ruby script, rather than assuming that "ruby" or "python" or whatever will always be in the PATH of the user executing the command.
• When in doubt (I'm generally always in doubt) give full pathnames to scripts too.
• Pass in as parameters the p4port and any other parameters to be used rather than expecting them to be already present in the environment.
• Within the script, explicitly add any extra directories to the search path for commands such as "import p4" in Python or "require 'P4' " in Ruby or any equivalent import-type statement, unless you are absolutely sure that the imported libraries are globally installed on the machine your are working with. Don't assume the same directory as the trigger script itself is in is in the path unless you can prove it.
• Trap and print to stdout (or stderr which goes to the p4d server log file) any errors/stack traces including exceptions from your p4 interface to aid hunting out problems. This is much easier to say than to do!

Passwords Cause Problems

In the good old days, before "p4 login" was even a twinkle in Christopher's eye, you could write your trigger assuming super user privileges (says in Yorkshire accent "we had it tough - could only dream of admin privileges in those days") and everything would work.

Life became substantially more complicated with security level 3 and login being required. Commands failed due to not being logged in, and this turned out to be a bit of a bugger ('scuse my French) to work out (why it had failed that is).

Received wisdom is "run your triggers as a special trigger/admin user, put that user in a special group with timeout of some very large number, log them in manually and all will be sweetness and light".

The interesting thing about this approach is that it often works, but as I discovered recently, can flatter to deceive. The problem I had was that the super user was indeed in a special "long timeout" group, and logged in on the same box (generating a suitable ticket). However, as I discovered only after some hair was torn out, the P4PORT that the user was logged in under was different to that used by trigger and thus the P4TICKET file entry was also different and the existing "login" had had no effect and my trigger was unfortunately failing silently.

Thus P4PORT=localhost:1666 where localhost=some_server.some_company.com will not work if the superuser is logged in using P4PORT=some_server.some_company.com:1666, since the latter is what will be in P4TICKET and the former will not be found and thus commands will fail. Be warned and expect/check for this!

When in doubt print out the environment within your script (via some sort of debug parameter).

Belt and Braces

My current intentions on this front are to produce a trigger framework that helps detect the above problems, and helps both avoid them and, when necessary, debug them in a (relatively) painless manner. This, at the moment of writing, is a work in progress, but I hope to be able to share it with the wider Perforce community as it emerges into the glare of publicity. I do reserve the right to retain the right of surprise to add some slight spice to my upcoming presentation at the European Perforce User Conference on the 19th September (in London).

Update: hopefully will be able to share a rework/expansion of Tony Smith's P4Trigger.rb framework which addresses some of the above issues fairly shortly - seems to be working at a client - time will tell - but fairly quickly.

Future topics will include ideas on test frameworks etc.

This is a partial review of Practical Perforce by Laura Wingerd, published by O'Reilly (ISBN 0-596-10185-6). The reason it is partial is that I intend to comment in more detail in future blog articles on some parts of the book, and wanted to post this without waiting for the whole thing!

As Laura mentions in the preface, the book is not intended for complete beginners, but more for readers with experience in other SCM (software configuration management) tools who are looking to understand how Perforce works.

To quote the introduction, there are two parts to this book:

• Part I (Chapters 1-6) is a whirlwind technical tour of Perforce commands and concepts. It's not a tutorial, nor a reference, but helpful nonetheless.
• Part II (Chapters 7-11) describes the big picture, using Perforce in a collaborative software development environment. It is particularly strong on branching patterns, how to structure codelines and tips and tricks in this area.

The real meat of the book for most Perforce sites is thus Part II, but there are definitely some goodies in Part I.

Chapter 1 presents some fundamentals about Perforce syntax and concepts. The diagrams on pages 6 &7 explain the relationship between revisions and changelists very well.

In Chapter 2, Laura discusses client workspaces and things like view syntax. She also describes basic check outs (open for edit in Perforce command line parlance), and introduces branching when she refers to cloning of files. She includes concepts of renaming and replacing content in files, reconciling changes made offline, and even introduces a couple of bits of undocumented syntax such as "p4 files @=1452". Quite a chunk of information in this chapter.

Resolving and Merging are the subject of chapter 3 and includes some very useful diagrams showing various scenarios. If you have ever had any questions about 3-way merging in Perforce - read this! On pp68-69 she gives examples of reconciling changes you have made to a file someone else renamed using the undocumented merge3 command - interesting if a touch esoteric (also referred to in "How to undo a merge" on p80. The recommendation on p74 to sync and resolve one changelist at a time is certainly worth considering, although I think it will depend on your environment as to how necessary that is.

The basics of branching are covered in chapter 4 including initial scenarios and how to track merge requirements across branches. She makes quite a lot of use of the interchanges command (not yet exposed in the GUIs) and explains the gory details of "yours", "theirs" and "base" nicely. Her approach of using filespec integrations for the initial examples is nice and simple, but I suspect more people are likely to use branch specs in real life. On p111 she gives a useful couple of commands to show how to find which changes have been merged in (more likely to be automated in scripts for most sites I would expect). Other subjects ocvered include all the gory details of what integrate actually does, as well as some very useful details as to what the interchanges command can tell us, particular with respect to cherry picked integrations.

Chapter 5 is quite short on labels and jobs and shows all the basics. A quick note on the final section where a job is used as a reference for a changelist - as of release 2005.2 there is an undocumented "dynamic label" option where a label can have an attached revision which probably makes the job trick unnecessary.

Chapter 6 gets into the subject of remote depots and proxies and also mentions the very useful spec depot option (automatic versioning of all spec objects). There is also some good advice on using p4web in browse mode to access your repository. The section on triggers and automation is a little light, but understandable.

Part II starts with Chapter 7 "How software evolves". This chapter is perhaps the highlight of the book, and introduces concepts that are totally independent of Perforce and apply to many SCM tools. Fortunately the chapter is currently available as a free PDF document from the O'Reilly website for the book. A firm understanding of the concepts introduced here will make it much easier for you to come up with suitable branching patterns for use in your organisations, and also, perhaps more importantly, give you some incredibly useful concepts for explaining your structure to other people within the organisation. Most SCM problems are due to poor communication rather than poor tools, or poor ideas. Laura relates the problems in the real world prevent us from an overly simplistic ideal world, and yet how some simple concepts allow us to manage this real world complexity. The "flow of change" and the "tofu scale" are classic concepts which should be in everyone's SCM vocabulary.

Summary

I am going to stop this post here, and will get to further chapters and some detailed comments on them as I have time.

But I will finish with the recommendation - buy this book!!

A personal foible perhaps but I do find Ruby's regular expression syntax remains in my brain much more easily than the Python equivalent.

Maybe it's the Ruby inheritance from Perl that makes the difference. For simple scripts I can just write standard regexps without any recourse to documentation and they just work! For example:

some_var = "prefix interesting_match some suffix"
if some_var =~ /prefix (\w+) some suffix/
  interesting_bit = $1
  print "Match found: ", interesting_bit, "\n"
end

In order to do the same in Python I find myself faffing around with the documentation (using ActiveState Python it's great to have a proper help file, but I would really like more links between the class reference and real examples of usage to help me out) and trying to remember if I want re.search or re.match and how do I get a match group and use it, etc. I have sundry Python scripts floating around that I open up and copy relevant examples from, but it does rather take time.

import re
some_var = "prefix interesting_match some suffix"
pat = re.compile('prefix (\w+) some suffix')
m = pat.match(some_var)
if m:
    print 'Match found: ', m.groups(0)[0]

Now I have to admit that it's not a huge deal in terms of the resulting code, but it took me 5-10 minutes just now to code and debug the Python version as opposed to the Ruby version which I typed in and which worked first time.

The net result is that I am noticeably  more productive in Ruby for those little scripts that make life easier, or when I am under strict time pressure. Now this is not to say that I don't like Python, or indeed that when I have a little more time I don't get use it and enjoy it. Having done some reasonably significant work in Python, e.g. rework P4DTI for PVCS (now Serena) Tracker I feel reasonably qualified to comment.

I also took the time to get sufficiently proficient in Python extensions to enhance and maintain P4Python. Mind you I now feel somewhat humbled by the most recent efforts at a Perforce integration (PyPerforce) - shows a depth of Python extensions knowledge before which I can only bow in admiration! (Minor aside - Ruby extensions are much easier to write than Python ones due mainly to the different garbage collection models).

Finishing up, I am definitely Perl-averse these days. There are a few Perl scripts that I maintain and can't be bothered, or can't find a convincing "business" case to rewrite, but anything new is Ruby or Python.

Tools Fair on 15th June - Cradle to Grave Support

As Chair of the BCS CMSG now (Shirley Lacy passed on the baton in November but fortunately for me remains in the background as vice chair), I have been feeling a little concerned about responsibility for the BCS CMSG Tools Fair - Cradle to Grave Support on 15th June. I am relieved that things are looking good now with good selection of sponsors supporting us:

Some newish names to me not being hugely up on the service management/ITIL side, but change, configuration and release management are a major focus in that field so look forward to meeting and hearing about them and the issues and solutions available.

A couple of big names missing on the software side which is a shame. A couple of organisations undergoing a re-org and thus no marketing budget to spend (well not now anyway). IBM/Rational aren't there because I can't find anyone to talk to that would appear to be able to make a decision. We used to have good relations with Rational but since they were acquired by IBM can't get anything out of them (I wonder what their responsiveness is if I were looking for support or trying to buy a produce?!). If anyone knows who to contact in the UK on marketing/events I'd be grateful for a heads up. The other one I have failed to get to anyone appropriate is Microsoft - would be good to find out about Team System etc, but all enquiries get passed from pillar to post and a deafening silence is the result.

Show me the money… identifying the value of Configuration Management

The other event we have on the 27th April is an evening event which is a relatively new departure for us. This will be at London South Bank University in their conference centre (which we have used before), and is in the form of a workshop lead by David Cuthbertson. This is going to focus on selling the benefits of CM and we hope will be a useful way of bringing together both new and more experienced people in the field to share experiences. Details to go up on the web site very shortly! David has spoken several times in the past and always gotten excellent reviews. He is also skilled at running workshops and getting contributions from others present. Should be a great evening. Apologies to those not close enough to London who want to attend, but we are looking at running (in particular evening) events elsewhere in the country - get in touch if you have any ideas.

Updated: 2005-11-29 - see link to scripts.
Updated: 2006-03-16 - some clarifications and link to Miki Tebeka's scripts

A common branching pattern is to have mainline and then task branches where work is done and then "published" by merging to the mainline as shown in Figure 3 of the article Building for Success.

In perforce the "publish" and the "catchup" are both performed by using the integrate command, typically with a branch spec. For example, you might have a branch spec

Branch:	task/fred
View:
	//depot/main/... //depot/task/fred/...

The key thing is that Fred should "catchup" before doing the "publish". This is so that the more risky merging is done in his task branch and not in the mainline. When doing the merge Fred should be bringing all changes by other people in the project into his branch and with the publish he should just be able to copy his code into the mainline.

There are several ways to achieve this:

• Education - tell Fred what to do and rely on him doing it - so what happens if he doesn't - can you "persuade" him not to?!
• Don't give Fred write access to the mainline (e.g. via protections or a trigger), and instead have the integration team do it. The problem then being that the integration team may not know the code as well as Fred and are perhaps more likely to make a mistake.

The basic way of detecting in Perforce whether a catchup has been done is to do a preview integrate from main to task/fred and check that nothing needs to be done, so check for no results from:

p4 integrate -n -b task/fred

If the above produces no results then proceed to do the publish:

p4 integrate -r -b task/fred
p4 resolve -as

The key step is that the "resolve -as" (safe automatic merge), resolves as many files as it can. It looks to see if their are only "theirs" (source) diff chunks or "yours" (target) diff chunks and will select the theirs or yours file appropriately. (Note that "theirs" and "both" or "yours" and "both" are also processed in the same way). The key point is that if there are "theirs" and "yours" (or "conflict") diff chunks, then safe automatic resolve will not process that file.

Of course having done the automatic resolve and with the changed files sitting in our client workspace it is usually a good idea to do things like a build and smoke test - there's not a whole heap of trust otherwise...

Thus, in our script we can check for anything not safely resolved automatically (resolve -n shows what still needs to be resolved).

p4 integrate -r -b task/fred
p4 resolve -as
p4 resolve -n
if any results from above command then exit with error

build
if any problems then exit with error

run smoke tests
if any problems then exit with error

If no errors at this point we are ready to submit.

There are some extra wrinkles to this:

• the "resolve -as" may validly not work if you have done a merge with edit during the catchup ("edit from" in the terminology from p4 integrated). You need to detect such situations and do a "resolve -at" to copy them over.
• as soon as one person has done a publish then all other branches will require to do a catchup to pull it in - this means publishes are going to become serial
• there is a window during which the publish is being performed when someone else might sneak in and do a publish thus meaning you need to do a catchup - consider a simple "locking" strategy to prevent this.

Note that the build and smoke test steps need to take an appropriate amount of time. If it takes hours to do them then the process is unlikely to work. Thus a few minutes or tens of minutes is likely to be the limit - this may mean cutting down on the number of tests that are executed - but that is usually OK. Apply judgement!

Although at the last point you might think it would be a brave person to do the submit automatically in a script! It's probably a good idea to do things like run diffs and give a visual once over before finally checking in, but it should be a pretty easy decision at this point.

In terms of automating the above, I can heartily recommend the various scripting languages with built-in calls to the Perforce API: Ruby, Python and Perl. Getting the results of a command is easy, and exceptions allow you to make error handling pretty easy as well.

Very brief examples of scripts designed to perform the above check in Python and Ruby are now available online. Note they are designed to be run from Custom Tools menu in p4v/p4win. Please note that Miki Tebeka has kindly published more production quality script implementations in python including a GUI front end. He also includes an example of an installer to automate installation of tools for p4win and p4v - check them out. Thanks Miki!

p.s. The above works for any codeline for which you have responsibility for accepting changes - it doesn't have to be a mainline - it can be a subsidiary integration line with third parties contributing, or team members "proposing" changes. You could automate the script and put it on an intranet page - let people try it out, and if successful then have changelists auto-checked in.