Have been meaning to do some work on P4Python recently, and the first thing I realised I should do is to update the test harness.

This is based around the unittest module so is fairly standard Python. Does some fairly standard things with test suites etc, and provides a reasonably good example of how to use the code itself, so acting as some level of documentation of P4Python.

The old version assumed a pre-existing Perforce server installed and running with some known content. This was fine for my own personal testing but had a couple of problems:

• it assumed the training repository which is fine for Perforce Consulting Partners but which isn't available to ordinary mortals (so they couldn't run the test harness locally)
• was a snapshot with existing users and client workspaces so required a license to use - not good for everyone
• it required too much manual setup before running

Obviously a candidate for tidying. So recent work done:

• Change to use the new Perforce Sample Repository which anyone can download and install (works without a license too)
• Change to automatically create and run a new server instance on a fresh install from the sample repository download (automatically unzip etc)

The end effect is fairly nice and automated, and provides a much better ongoing resource to anyone wishing to do work on P4Python. Please note that it currently assumes Windows environment, but will insert a few checks to make platform independent shortly.

To have a look you can either:

• download the complete P4Python sources
• browse the test module
• just browse the Perforce Public Depot using your favourite method: //guest/robert_cowham/perforce/API/python

Note that credit for various aspects should go to Ravenbrook from their work on P4DTI - I stole various techniques from their implementation of a test harness.

It also serves as a reasonable example of a test harness for Perforce scripting, and am very happy to receive comments and improvements (there are no doubt some Python gurus out there who can suggest some improvements at points in the code). I use something fairly similar as a test harness for the VSStoP4 scripts which are actually written in Perl (too horrible to write such a thing in Perl was my feeling)!

Would like to encourage people to take their own copies and give back some improvements in the general framework at least...

This is a personal review of the week in Wales with Inaba sensei and other teachers organised by Tetsushinkan dojo. It is not a blow by blow account of everything that took place, but a personal summary with various highlights of things that were important for me.

First a quick overview:

• An advanced guard of the Japanese contingent arrived on Thursday (not including Inaba sensei)
• Udegawa san taught the normal Friday evening class at Tetsushinkan
• Saturday everyone decamped the 4-6 hours to Wales (this included Inaba sensei and Endo san who only arrived that day from Poland where they had been teaching the previous week). Various other overseas visitors flew in that day and also drove down.
• Sunday was the official opening and first practice
• Standard schedule during the week
  • Morning practice 9.30 - 12.30
  • Afternoon activities in groups ranging from canoeing to horse riding to cliff walks
  • A couple of Instructor's sessions 6-7.30pm for invited people (others free to watch)
  • A couple of meetings with instructors to review links with Shiseikan, including future courses
• On Tuesday evening we greeted Toyama guji (the Chief Priest of the Meji Jingu shrine who was "on tour" in Europe) with his entourage to Wales. He came to watch practice on Wednesday and kindly hosted a meal on the Wednesday evening for an invited group.
• Final practice on Friday, with an excellent party that evening in a restaurant in Newport
• Return to London on Saturday
• Japanese all left Sunday

Summary

It was a truly excellent week. I came away with masses of food for thought and further reflection regarding both life and budo studies. It was a great pleasure to renew old friendships and to make some new friends. I also feel a renewed personal challenge to do more with these insights!

The teaching started with a formal opening ceremony conducted by Mori san, a priest from the Meiji Jungu shrine and included prayers and harai tachi - cleansing of the dojo and the spirits of those present. Inaba sensei talked about his wish to demonstrate the importance of this aspect as part of his teaching. Shinto is a relatively recent name for practices which go back thousands of years. The gods and spirits come and sit in the branches of the himorogi, part of the small shrine set up during practice. He does not view this as a religious practice and certainly has no wish to convert everyone to Shintoism! However, he wished to demonstrate the importance of setting the tone for the week - in a way a sort of misogi for all those present.

Those of you who have seen Inaba sensei will know that talking and lecturing form quite a part of his teaching. This can be a bit frustrating at first for those used to other styles of teaching where much more emphasis is placed on practice. Many of Inaba sensei's messages keep recurring over the years, but as often happens, various things made more sense to me this time - maybe I have changed in the meantime and so now hear them differently!

Why Are you Practicing?

As with the initial ceremony, this was something Inaba sensei returned to at several points. We should understand why we are practising - what do we wish to protect or defend? What are we prepared to do in the face of attack, for example on our loved ones? Answers to this question inform all the rest of our studies, and thus really are fundamental. Study and thought will be well repaid.

Relationship with Kunii Sensei

Inaba sensei gave some more details on his relationship with Kunii Sensei (the 18th generation Headmaster of the Kashima Shinryu school). This relationship was fairly short - 17 months I believe as a direct student before his death in 1966. Inaba sensei started studying aikido in the early 1960s at Hombu (and with Yamaguchi sensei). Being allowed to study with Kunii sensei was not easy, but was enabled by an introduction from Ashizu sensei, a Shinto philosopher that Inaba sensei was also studying with (and indeed that association went on for nearly 30 years). The relationship with Kunii sensei was obviously very close and Inaba sensei mentioned being asked by Kunii sensei to pass what he learnt through direct transmission and also the results of his own independent study. Inaba sensei obviously feels a very strong obligation as a result.

Developing the Body

This has long been a key concern - developing our bodies as a foundation for studies in budo. This time around there were copies of Inaba sensei's book on the subject (English and Japanese in the one volume). The English title is "Researching Japanese Budo". It originated in a series of articles for a Japanese magazine in the early '90s, and I was involved in helping revise the translation - I hasten to add that this meant reviewing the resulting English and offering some suggestions for clarifying unclear points or reworking phrases which appeared a little clumsy. The real work was done by Endo san and Annika Hansen. The original translation was done by Yamada sensei and Diane Zingale, of which a number of people received photo copies nearly 10 years ago.

Anyway, pressure of time meant that it still isn't quite where it might be in the English translation, but certainly full of very useful and interesting information - perhaps will review separately. I suggest contacting Shiseikan directly if you wish to get a copy.

Energy Inherited from your Ancestors

This was a new concept for me, and something I did not find easy to get a handle on in a physical sense. Intellectually to me it makes sense to talk about what we inherit from our direct ancestors and also the energy of the place we are in, but how to do anything physical with such energy was not clear. The exercise demonstrated was basically kokyu-ho (e.g. both hands grasped and lifting your hands and thus your partner in front of you). The difference as shown between this and normal ki energy was not clear to me in any practical way, so very much one for further research.

Kenjutsu Kata

I was very interested to see demonstrated some more dynamic ways of studying kenjutsu. Inaba sensei discussed the various kata and went through the details of the Uradachi (second set of 10 techniques) in some detail. However, as well as the classic form of the kata with uke and nage, we were also shown more dynamic forms, for example where uke more or less attacked in a kendo style and the response need to be much more freeform than in the set version of the kata. I was particularly interested as I had been thinking prior to the course that my own kenjutsu risked being a little formulaic - falling into the pattern of anticipating the technique and only being able to respond in the fixed form - even getting annoyed when the attack wasn't "just right"! Some teaching of the ukemi side were also very useful for me and others who are in positions to teach. It is important to ensure that you are applying appropriate pressure to your partner to draw their best out from them.

He did mention the requirements to study each set of kata (Kihon, Ura, Aishin, Jissen, Kassen) properly before moving on to the next, but also to place them in a context of being able to react flexibly.

Inaba sensei went through the Uradachi in some detail as part of one of the instructor's courses. Particular points of note included the requirement for both sides to treat the shinai as a real sword and thus heavy contact by uke is not appropriate - you can't lean on the blade of a katana! Normally nage leads by taking the first step, although it may appear that both people start together. It is uke's responsibility to respond to nage.

Kesa Giri Footwork

I don't think anyone present will forget the outdoors practice of kesa giri - the location was at Clydey Cottages - a place with a great lawn and stunning view over the valley. Inaba sensei really demonstrated the difference between short, restricted, inhibited kesa giri and the more expansive cuts which really started to connect with the nature all around us. The difference that this made in everyone's kesa giri was very marked, partiularly the following day back inside - you could see the expansiveness still present.

This moved on also to the use of the feet. For practical reasons of the ground being perhaps unstable, or uneven, or on a slope, he emphasized the need to step with the feet fully before completing the cut. It is easy in the clean confines of the dojo to get used to cutting with the landing of the foot and the focus point of the sword cut occurring at the same point. Outdoors however, this doesn't work - as soon as you step you risk slipping and sliding, so it is important to have fairly firm footing before completing the cut.

Battojutsu (Sword drawing)

This was another memorable day outside (though slightly more rainy) focussing on the drawing of the sword. We were all using bokuto (wooden swords) for practice but he did discuss the use of katana (live blades). He does not recommend practice with iaito (metal swords with blunt blades), as these tend to give one a false sense of security. A katana will "teach" you if you let it. It requires much more focus and attention which is good for practice and concentration. In a way, you should not use a katana too much since familiarity may breed contempt and thus lack of focus.

Various exercises and cutting to different sides (front, back and both sides) were demonstrated and practiced. In practice you shouldn't just get into the habit of thinking "OK, now I will pretend someone is attacking from behind, so let's react". It should be a much more instinctual response - wait and be alert, and the right moment will arise. Remember also flexible footwork - react off either foot.

Stepping off the Line

This was another exercise completed outside. It is one I have seen and practiced several times over the years, and yet realised this time around that I really need to focus much more on it for my own practice.

You start a couple of paces away from uke, and then step in so as to "invite" an attach (standard punch to the stomach). As the punch comes in you need to step off the line so that you end up in a triangle with feet pointing at your opponents centre. If uke attacks right hand (and foot), then it is quite normal to initially half step with the right foot to "invite", and then step off the line with the left foot closest to uke. See Figure 1.

The alternative which we practised involves "inviting" with a half step forward of the left foot. The result is similar, but left and right feet are reversed.

This appears quite simple but is difficult to do well, and in particular getting the timing correct and being able to move your body appropriately is key. Watching Inaba sensei, he doesn't necessarily appear to move particularly fast, but it is at just the right moment and with appropriate speed.

Further developments of this exercise shorten the distance to one step, then half a step, then no steps (with uke's hand on your belly). Challenging stuff!

The Grading

This happened at the end of the week during the last session ranging from shodan to yondan (errrr that would be me...). Most people were from Tetsushinkan but some others from London and Greece, and everyone generally did very well. Results are on the Tetsushinkan events page.

As is typical with Inaba sensei's gradings they were relatively short. In each case various attacks were specified and the actual technique was not specified - indeed it was perfectly acceptable to perform the same technique several times in a row to the same attack. The important factors are spirit, body movement and effectiveness, rather than necessarily technical beauty. People also demonstrated some of the Kashima Shinryu kenjutsu kata according to the knowledge and experience of the person being graded (e.g. Kihondachi or Uradachi). This approach is certainly different to what is normally done within many standard aikikai  dojos. I did discuss this afterwards over a beer with a couple of people - "but how do you know they know all the appropriate techniques?" was one question. I obviously can't speak for Inaba sensei but I think that it is usually clear from a demonstration the sort of level a person has reached, and it is perfectly in tune with his general approach of studying relatively few techniques but in great depth. In addition, I think that this sort of approach works well in smaller organisations where the people being graded are known to the examiners - I think there might be more problems with larger scale organisations.

Instructor's Meeting - The Future

There were a couple of interesting discussions with vairous instructors and countries present, e.g. Bjorn Eirik Olsen and others from Norway, Pascal Durchon and Joel Roche from France, Anita Kohler from Germany and various others from UK etc. As with all good discussions, some heat and light at times was eventually turned into some good results and conclusions.

Inaba sensei was keen to get an understanding of how he could best support those present, and what sort of future courses and training was both sought and might be possible. He did mention that his preferred method is to teach a small group of people at the Shiseikan where he could give a lot of time and attention to them, and he made a generous offer for those prepared to arrange visits. Pascal and Anita in particular expressed a desire to give more people the opportunity to get to know Inaba sensei and experience his teaching. The rough plan for 2007 is for a seminar in France to be organised primarily by Pascal, followed by Norway in 2008 and Germany in 2009. Visits to Shiseikan are left to individual groups to arrange. I think the intention for Paris is to perhaps offer open sessions where anyone can attend, but also a more restricted session for those with previous experience and who have already made contact with Inaba sensei and are already studying his methods. This would allow for more advanced training, in particular for kenjutsu. I think it will be very interesting to see how this works, and certainly look forward to Paris next summer!

Conclusion

The course was excellent - lovely surroundings and a great atmosphere. Serious study and yet not in a heavy atmosphere. The number of seniors around (from Japan and elsewhere) made for some excellent training.

Inaba sensei is a fascinating and inspiring teacher on many levels, and it is worth taking any opportunity to get around him and learn from him. Some of his methods are a little extreme perhaps, and I am not clear in my own mind how easy it is to combine what he teaches with classic aikikai aikido. The differences go to quite deep levels. This is fine for those of us who primarily follow his teaching. Others who think it might be nice to spice up their traditional methods may find it distinctly more challenging. How much integration is possible and how much is it two separate approaches - I suspect it is going to take me at least quite a few more years to decide. In the meantime he is planning for the future. He is not going to go on for ever - seek him out and take advantage of him!

Life has been just a touch busy recently having been flat out on various client projects pretty much over the whole summer (managed a week away but only just!). All grist to the mill for future blogging, so hopefully a variety of articles to come!

Meanwhile one of the things I was doing was preparing and then giving a presentation for the (first) Perforce European Conference on 19th September in central London.

I think the papers will be out pretty shortly on the Perforce site, but meanwhile a few highlights and personal notes. There were some big names present and it was good to hear about various practices and principles in operation.

Keynote

Christopher Seiwald did a variation on his slightly "aw shucks" style keynote. Some key points:

• Perforce doing fine: 200,000 users and 4,000 companies
• Company motto: "Aim low and hit!" (do one thing well, remain best of breed and wait for the analyst pendulum to swing back to best of breed rather than suite integration, which it seems to do on a regular basis)
• Working on a variety of things for future world domination, but don't want to pre-announce as usual
• Very pleased with the way things are happening in Europe, and obviously at the response to this event.
• Next US Conference 9 - 11 May 2007, Las Vegas.
• Sydney office now opened to give global timezone support coverage!

Symbian

Deepak Modgill did a nice presentation on the challanges faced by Symbian for their offshoring. Another in the Symbian series of how their business and vairous configuration management practices have evolved. Not deeply technical but interesting never-the-less.

SAP

Obviously a flagship site for Perforce. Thomas Kroll and Claudia Loff did a good presentation. Interesting how much process and tools they had wrapped around Perforce. A few key stats:

• 4,800 users
• 80+ Perforce servers (but all on same cluster hardware)
• Fujitsu Siemens clusters with 32Gb RAM running SunOS 9
• SAN (mirrored) for main data

They use a very structured process (repository structure and branching scheme) and a parallel (P4SAP) system with its own database to record things like changes and migrations (they call them transports) of releases between different servers. There is also a layer P4MS (Management System) to handle users etc.

Quite impressive.

Process Automation

Obviously my talk was wonderful! I was thought fairly pleased with how it went down and got some good comments afterwards. For anyone interested, the Ruby triggers framework and a couple of utilities are in my area of the Perforce Public Depot.

I will no doubt be blogging on various related aspects (that I haven't already touched on).

Bank of America

Good talk by Sean Cody and Kevin Breidenbach about different approaches with the bank. They have been replacing ClearCase with Perforce in various groups, mainly due to the performance for shared development between US, UK and India. Experience of Multisite sometimes taking hours to "sync up", vs. 10-20 minutes max in Perforce.

Another feature of the talk was the power of continuous integration.

Google

Dan Bloch discussed Google's use of Perforce and in particular how they manage issues around Perforce database locking and identifying and bumping off rogue commands.

Some more stats:

• 3,000+ users
• Single Perforce Repository
• HP DL585 4-way Opeteron with 128Gb RAM
• Linux 2.4 and NetApp filer

Sounds like it wins the contest for largest number of users against a single server!

The details of the lock identification was very interesting and Dan said he would be releasing the lock.pl script and some docs on the Public Depot real soon now!

Perforce 2006.1 Update

A very interesting and technical talk by Michael Shields regarding a variety of performance optimisations made between 2005.2 and 2006.1.

Summary: 2006.1 is quite a bit faster!

Read the slides for more details.

Laura Wingerd

Laura did another fairly technical talk on what has happened to the branching/merging algorithm, and more particularly common ancestor detection algorithm used in various releases of 2006.1. In her usual inimitable style she came up with some very useful ways of explaining things like convergence and divergence of branches over time. Things got decidedly more technical with discussions on common ancestors and I was left knowing I have to go through some of this in detail in a quiet moment just to make sure I really do understand it! The changes with 2006.1 look good, but I did get the impression some edge cases could give some slightly surprising results if you don't know what's going on behind the covers (and indeed the driving intentions behind the algorithm).

Summary

Venue worked very well for location. Networking with both Perforce people and various other delegates was as ever a highlight.

Unfortunately the room booked was not huge which meant the event sold out well ahead of time - a shame a more flexible venue wasn't chosen, but that was only quibble. Organisation well run.

An excellent day!

There are times when you receive a third party code drop which you wish to import. The classic method is documented in Tech Note 15 and its reference to working disconnected (Tech Note 2). The techniques mentioned work very well to find new files, deleted files and changed files.

There is sometimes a fly in the ointment to do with keyword expansion. This is things like a CVS code drop containing expanded keywords:

$Id: //depot/robertcowham.com/main/blog/data/scm/p4_handling_keywords.html#1 $

The Perforce equivalent of this might be:

$Id: //depot/robertcowham.com/main/blog/data/scm/p4_handling_keywords.html#1 $

The simple command to find differences is "p4 diff -se". If your local version has Perforce keyword expansion turned on then you will get a load of files spuriously identified as having changed where the only real change is in the keywords.

Thus we want a simple script to run through the diffs and exclude any diffs where only keywords are found (note that this includes where the keyword is embedded, such as in a static variable assignment).

The following simple script is a good base for this. It does the job, and performs pretty well, handling thousands of files in a few minutes. It makes use of unified diff format where changed lines have a prefix in the first character of the output.

# Script to import a set of changed files with existing keywords already expanded
# (either Perforce or CVS).
# Does "diff -se" and processes the output

# Args: current directory to check
  
require 'P4'

p4 = P4.new
p4.tagged
p4.connect

def process_file(p4, f)
  diffs = p4.run_diff("-f", "-du", f)
  real_diffs = Array.new
  diffs.each { |line|
    case line
    when /^====/
    when /^\@\@/ 
    when /^ /
    else
      if line !~ /\$Id|\$DateTime|\$Revision|\$Date|\$Author|\$Name|\$RCSfile|\$Source/
        real_diffs << line
        # puts f, line
      end
    end
  }
  if real_diffs.size > 0
    print "Editing #{f}\n"
    p4.run_edit(f)
  end
end

all_files = p4.run_diff("-se", ARGV[0])
print "Processing #{all_files.size}\n"
i = 0
all_files.each{|f|
  i += 1
  # print"Processing #{i}\r" if i % 10 == 0
  # print"Processing #{f['depotFile']}\n"
  process_file(p4, f)
}

It is pretty easy to run, e.g.:

diff_se.rb ...

The net result will be a list of files checked out (p4 edit) in the default changelist.

Note that one of the big advantages of Perforce branching and merging is that it handles merges neatly when keyword expansion is used between branches (and thus you don't get spurious conflicts).

CVS Imports

If you use the cvs2p4 scripts to import a CVS repository you can end up with a slight problem since the conversion copies the CVS archive files (in RCS format) and Perforce uses them unchanged. The problem comes about because CVS stores the keywords already expanded in the RCS archive. Perforce stores its RCS files with the keywords not expanded, which makes it easier for it to do the merging between branches (without keyword conflicts). While Perforce can handle a CVS archive with the the keywords "pre-expanded", it does lead to spurious merge conflicts. Note that this problem is only really present during the early merges after the CVS import. It will no longer be present as soon as the base file for any merge is fully in Perforce format (i.e. after at least one merge has been done).

Writing good Perforce triggers, and, more importantly, debugging them in live use, turns out to be one of those things that seems simple but has lots of tricky issues that can lead to lots of time being wasted.

In spite of thinking that I understood lots of the issues, I still spent a couple of hours recently debugging a problem that turned out to be a combination of environment and password issues. This was particularly annoying as I had rather though I knew about this stuff (and indeed have advised people over the years about it!), and yet was blindsided and caught out by some issues I had forgotten about or not thought through deeply enough.

I reserve the right to revisit this subject more than once in the future with further insights and news...

Assume Nothing About The Environment!

The classic approach to triggers is to write a nice script (Python or Ruby for me these days - no Perl, though just occasionally I miss it!) and debug it by running with the appropriate parameters from the command line (e.g. create a pending changelist and pass in the pending changelist number). This does indeed tend to turn up a number of issues, but the good thing is you can usually debug them with the appropriate command (<rant> why does python require you to execute pdb.py which isn't by default put in the path on Windows machines, and why does Ruby not learn from Perl and for example use -d as a parameter to debug things instead of "-rdebug" - very unobvious!</rant>).

The major problem turns out to be the fact that the trigger is executed by the Perforce server process and may have a very different environment to what you might think as you run a "login" session. One sort of expects this on Unix, but on Windows it can be particularly surprising how little is in the environment due to the username that the Perforce process is running in when it is running as a service (default installation on Windows).

Thus the first rule of trigger writing is "assume nothing about the environment!".

It is very easy to forget this and assume very simple things, like:

• P4PORT is always defined
• P4USER is always defined
• failures of individual p4 commands within the trigger will be obvious

Thus immediate recommendations are:

• Give full pathnames to executables. For example, "/usr/bin/ruby" or "C:\ruby\bin ruby.exe" as the initial parameter for the ruby script, rather than assuming that "ruby" or "python" or whatever will always be in the PATH of the user executing the command.
• When in doubt (I'm generally always in doubt) give full pathnames to scripts too.
• Pass in as parameters the p4port and any other parameters to be used rather than expecting them to be already present in the environment.
• Within the script, explicitly add any extra directories to the search path for commands such as "import p4" in Python or "require 'P4' " in Ruby or any equivalent import-type statement, unless you are absolutely sure that the imported libraries are globally installed on the machine your are working with. Don't assume the same directory as the trigger script itself is in is in the path unless you can prove it.
• Trap and print to stdout (or stderr which goes to the p4d server log file) any errors/stack traces including exceptions from your p4 interface to aid hunting out problems. This is much easier to say than to do!

Passwords Cause Problems

In the good old days, before "p4 login" was even a twinkle in Christopher's eye, you could write your trigger assuming super user privileges (says in Yorkshire accent "we had it tough - could only dream of admin privileges in those days") and everything would work.

Life became substantially more complicated with security level 3 and login being required. Commands failed due to not being logged in, and this turned out to be a bit of a bugger ('scuse my French) to work out (why it had failed that is).

Received wisdom is "run your triggers as a special trigger/admin user, put that user in a special group with timeout of some very large number, log them in manually and all will be sweetness and light".

The interesting thing about this approach is that it often works, but as I discovered recently, can flatter to deceive. The problem I had was that the super user was indeed in a special "long timeout" group, and logged in on the same box (generating a suitable ticket). However, as I discovered only after some hair was torn out, the P4PORT that the user was logged in under was different to that used by trigger and thus the P4TICKET file entry was also different and the existing "login" had had no effect and my trigger was unfortunately failing silently.

Thus P4PORT=localhost:1666 where localhost=some_server.some_company.com will not work if the superuser is logged in using P4PORT=some_server.some_company.com:1666, since the latter is what will be in P4TICKET and the former will not be found and thus commands will fail. Be warned and expect/check for this!

When in doubt print out the environment within your script (via some sort of debug parameter).

Belt and Braces

My current intentions on this front are to produce a trigger framework that helps detect the above problems, and helps both avoid them and, when necessary, debug them in a (relatively) painless manner. This, at the moment of writing, is a work in progress, but I hope to be able to share it with the wider Perforce community as it emerges into the glare of publicity. I do reserve the right to retain the right of surprise to add some slight spice to my upcoming presentation at the European Perforce User Conference on the 19th September (in London).

Update: hopefully will be able to share a rework/expansion of Tony Smith's P4Trigger.rb framework which addresses some of the above issues fairly shortly - seems to be working at a client - time will tell - but fairly quickly.

Future topics will include ideas on test frameworks etc.