SAP and the 25 zones
Recently we've been supporting the SAP team by getting things ready for the upgrade they are doing to SAP in the coming months. Part of this means migrating to new servers. We're using a cluster of fairly beefy Sun (actually Fujitsu) SPARC VII processors which are each quad-core and use SMT to present two threads per core. In essence the operating system believes it has 32 CPUs to play with:
==================================== CPUs ==================================== CPU CPU Run L2$ CPU CPU LSB Chip ID MHz MB Impl. Mask --- ---- ---------------------------------------- ---- --- ----- ---- 00 0 0, 1, 2, 3, 4, 5, 6, 7 2530 5.5 7 161 00 1 8, 9, 10, 11, 12, 13, 14, 15 2530 5.5 7 161 01 0 32, 33, 34, 35, 36, 37, 38, 39 2530 5.5 7 161 01 1 40, 41, 42, 43, 44, 45, 46, 47 2530 5.5 7 161
The machines also have 256GB of RAM in them. The SAP software is very server-hungry. Our new configuration has 25 separate SAP instances doing various different functions. Without zones (or Logical Domains), we would have to have 25 separate machines to run the software. Some other institutions actually do that! We have some in-house scripts that were written by the very talented Mark White in our team which meant I could create the server zones and add them to the cluster in record time. Once I had set up the file to create the zones, I left it running over night. Each zone takes about fifteen minutes to create but as I don't need to sit and watch it, it only takes about 10 minutes of my time to get things up and running.
The Enterprise Incident
Today I was in an Incident Review Meeting, going over a problem we had a couple of weeks ago. The gist of the problem was that we had a brief hiccup on the network whilst some planned maintenance was taking place. Unfortunately this then caused a number of knock-on problems relating to our storage systems and the high-availability cluster systems (that we depend on to provide failure in the event of a failure!).
Our main storage is a pair of NetApp clusters which can provide access to a large amount of storage using a variety of protocols, including iSCSI, NFS and CIFS. Generally within the Unix team we tend to use either iSCSI or NFS as CIFS has its roots in Microsoft technologies that often don't play well in a Unix environment (although there is the excellent Samba implementation of CIFS). All the storage network traffic runs over the same core network as the rest of network data, so when the network outage occured, we instantly lost connection to the storage on the systems.
The first thing our systems noticed was this;
Apr 28 07:27:15 m in.mpathd[311]: [ID 594170 daemon.error] NIC failure detected on bge852002 of group ipmp2 Apr 28 07:27:15 m in.mpathd[311]: [ID 594170 daemon.error] NIC failure detected on bge251002 of group ipmp1 Apr 28 07:27:16 m in.mpathd[311]: [ID 594170 daemon.error] NIC failure detected on bge852001 of group ipmp2 Apr 28 07:27:16 m in.mpathd[311]: [ID 832587 daemon.error] Successfully failed over from NIC bge852001 to NIC bge852000 Apr 28 07:27:21 m in.mpathd[311]: [ID 594170 daemon.error] NIC failure detected on bge251001 of group ipmp1 Apr 28 07:27:21 m in.mpathd[311]: [ID 832587 daemon.error] Successfully failed over from NIC bge251001 to NIC bge251000 Apr 28 07:27:27 m in.mpathd[311]: [ID 168056 daemon.error] All Interfaces in group ipmp2 have failed Apr 28 07:27:27 m Cluster.PNM: [ID 890413 daemon.notice] ipmp2: state transition from OK to DOWN.
What this basically tells us is that some of the network interfaces on "m" (one of the nodes in our production cluster) lost some of their network connections. We use a technology called IPMP (IP Multi-Pathing) to distribute data across a number of network connections. The bad news really hits where it says "ipmp2: state transition from OK to DOWN". That means that it lost all the interfaces on a particular IPMP interface and it couldn't pass traffic any more and the cluster software noticed.
From that point it gets rapidly worse;
Apr 28 07:30:18 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - pussygalore:e1000g268000 being drained Apr 28 07:30:29 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - solitaire:e1000g268002 being drained Apr 28 07:31:08 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268002 - solitaire:e1000g268002 online Apr 28 07:31:09 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268002 - pussygalore:e1000g268000 online Apr 28 07:32:18 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - solitaire:e1000g268002 being drained Apr 28 07:32:26 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - pussygalore:e1000g268000 being drained Apr 28 07:32:32 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - oddjob:e1000g268002 being drained Apr 28 07:32:33 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268001 - oddjob:e1000g268001 being drained Apr 28 07:32:33 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268000 - oddjob:e1000g268000 being drained Apr 28 07:32:41 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268000 - solitaire:e1000g268000 being drained Apr 28 07:32:42 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268001 - solitaire:e1000g268001 being drained Apr 28 07:32:44 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268000 - jaws:nxge268000 being drained Apr 28 07:32:44 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268001 - jaws:nxge268001 being drained Apr 28 07:32:44 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - jaws:nxge268002 being drained Apr 28 07:32:46 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268002 - goldfinger:e1000g268000 being drained Apr 28 07:32:46 m cl_runtime: [ID 489438 kern.notice] NOTICE: clcomm: Path m:bge268001 - goldfinger:bge268001 being drained
This is the cluster software noticing that interconnects between "m" and the other nodes has disappeared. Eventually, it says;
Apr 28 07:33:17 m cl_dlpitrans: [ID 624622 kern.notice] Notifying cluster that this node is panicking Apr 28 07:33:17 m unix: [ID 265925 kern.notice] CMM: Cluster lost operational quorum; aborting.
At this point, "m" has realised that the cluster no longer has "operational quorum". This basically means that it can no longer talk to a majority of the cluster and it considers itself to be on the losing team and waves the white flag! Most Solaris administrators will recognise the following;
Apr 28 07:33:19 m genunix: [ID 672855 kern.notice] syncing file systems... Apr 28 07:33:19 m genunix: [ID 904073 kern.notice] done Apr 28 07:33:20 m genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel Apr 28 07:35:39 m genunix: [ID 409368 kern.notice] ^M100% done: 258777 pages dumped, compression ratio 3.36, Apr 28 07:35:39 m genunix: [ID 851671 kern.notice] dump succeeded Apr 28 07:36:54 m genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_142900-03 64-bit Apr 28 07:36:54 m genunix: [ID 943908 kern.notice] Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
At this point "m" has panicked (forced a system shutdown) and started to boot. When "m" comes back up, it starts the cluster software, which is aware of the other hosts in the cluster;
Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node goldfinger (nodeid = 1) with votecount = 1 added. Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node pussygalore (nodeid = 2) with votecount = 1 added. Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node oddjob (nodeid = 3) with votecount = 1 added. Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node solitaire (nodeid = 4) with votecount = 1 added. Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node m (nodeid = 5) with votecount = 1 added. Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node blofeld (nodeid = 6) with votecount = 1 added. Apr 28 07:37:53 m cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node jaws (nodeid = 7) with votecount = 1 added.
"m" happens to have no zones to activate. We always keep one low-spec machine in a cluster for us to do administration work on and in this cluster "m" is that node.
In the time it took "m" to restart, the network problem had sorted itself out. The other nodes in the cluster have just gone through exactly the same panicking process, but as they have more CPUs and RAM, they take a little bit longer to start up, so at first "m" can't see them. You can see this in the logs;
Apr 28 07:38:53 m cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path m:bge268000 - jaws:nxge268000 errors during initiation Apr 28 07:38:53 m cl_runtime: [ID 618107 kern.warning] WARNING: Path m:bge268000 - jaws:nxge268000 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Apr 28 07:38:53 m cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path m:bge268000 - blofeld:nxge268000 errors during initiation Apr 28 07:38:53 m cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path m:bge268001 - jaws:nxge268001 errors during initiation Apr 28 07:38:53 m cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path m:bge268002 - jaws:nxge268002 errors during initiation Apr 28 07:38:53 m cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path m:bge268002 - blofeld:nxge268002 errors during initiation Apr 28 07:38:53 m cl_runtime: [ID 604153 kern.notice] NOTICE: clcomm: Path m:bge268001 - blofeld:nxge268001 errors during initiation Apr 28 07:38:53 m cl_runtime: [ID 618107 kern.warning] WARNING: Path m:bge268000 - blofeld:nxge268000 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Apr 28 07:38:53 m cl_runtime: [ID 618107 kern.warning] WARNING: Path m:bge268001 - jaws:nxge268001 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Apr 28 07:38:53 m cl_runtime: [ID 618107 kern.warning] WARNING: Path m:bge268002 - jaws:nxge268002 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Apr 28 07:38:53 m cl_runtime: [ID 618107 kern.warning] WARNING: Path m:bge268002 - blofeld:nxge268002 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Apr 28 07:38:53 m cl_runtime: [ID 618107 kern.warning] WARNING: Path m:bge268001 - blofeld:nxge268001 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Apr 28 07:38:54 m cl_runtime: [ID 980942 kern.notice] NOTICE: CMM: Cluster doesn't have operational quorum yet; waiting for quorum.
Note that at this point, we still don't have operational quorum, but as the cluster is just starting, it waits. After a couple of minutes the other nodes in the cluster start up.
Apr 28 07:44:28 m cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node goldfinger (nodeid: 1, incarnation #: 1272437064) has become reachable. Apr 28 07:44:28 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268001 - goldfinger:bge268001 online Apr 28 07:44:28 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268000 - goldfinger:bge268000 online Apr 28 07:44:28 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268002 - goldfinger:e1000g268000 online Apr 28 07:44:49 m cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node pussygalore (nodeid: 2, incarnation #: 1272437087) has become reachable. Apr 28 07:44:49 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268001 - pussygalore:bge268001 online Apr 28 07:44:49 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268000 - pussygalore:bge268000 online Apr 28 07:44:49 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268002 - pussygalore:e1000g268000 online Apr 28 07:45:03 m cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node oddjob (nodeid: 3, incarnation #: 1272437101) has become reachable. Apr 28 07:45:03 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268002 - oddjob:e1000g268002 online Apr 28 07:45:03 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268001 - oddjob:e1000g268001 online Apr 28 07:45:03 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268000 - oddjob:e1000g268000 online Apr 28 07:45:36 m cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node blofeld (nodeid: 6, incarnation #: 1272437135) has become reachable. Apr 28 07:45:36 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268002 - blofeld:nxge268002 online Apr 28 07:45:36 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268001 - blofeld:nxge268001 online Apr 28 07:45:37 m cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path m:bge268000 - blofeld:nxge268000 online Apr 28 07:46:37 m cl_runtime: [ID 525628 kern.notice] NOTICE: CMM: Cluster has reached quorum.
As the nodes come back up, the cluster software notices the links to them come back. Eventually, when we can see more than 50% of the cluster we reach operational quorum and the cluster is ready to go! At this point we then start up the zones running on the nodes and all the services come back up.
It's interesting to note that throughout this outage, the systems didn't notice their iSCSI attached storage failing. The reason for this is that the Solaris Cluster software noticed quicker than the storage layer of the operating system and panicked the machine. In the time it took to reboot, the network had come back so the iSCSI devices were available again.
Honey oats and worrying news from a supplier
I've been learning a bit about Solaris Cluster this week. This is the software that we use to try (try) and maintain availability of some of our systems. At the moment we generally use it to start up containers on a new machine in the event of hardware failure, but it can do an awful lot more.
This week I've also heard some news about Sun Microsystems, now Oracle, which has left me concerned about the ongoing viability of the SPARC servers that we use. Most of our corporate systems (Finance, HR, CIS, etc) use SPARC based servers. Fortunately, alternatives are available but migrating these systems to a different platform would be a huge exercise which would most likely take us years to complete. We've already had some (very) informal discussions about an exit strategy to allow us to move to other architectures. As these systems are all fairly critical to the operation of the University, we'll need to think long and hard about how we approach any potential change.
Lunch in Weston Park and getting down to some work!
Today I managed to get out for a walk with Chris and Dan for some lunch. We popped into the Weston Park Museum for something to eat and then went for a quick wander and took a couple of pictures. It's great to be back in Sheffield, especially when the weather is so good and I can get an hour out at lunch to have a walk about!
One of the first things I have been asked to do is to try and break into the machines. I am in the unusual position of having quite a bit of knowledge about the internal systems from my previous employment at the University but as yet I haven't been given access to much. My task has been to try and see what I can break into. We manage user access to most of our systems centrally but administrator access is managed using a much less formal approach, mainly because it is not a common event to add or remove systems administrators, so the effort involved in trying to formalise it isn't very worthwhile.
So far, the systems are holding up pretty well; I managed to log in to an old server related to the MUSE portal and I managed to gain administrator access to the iceberg HPC cluster without too much trouble. In both cases I managed to get in using old credentials from when I worked here 3 years ago. These don't present an immediate threat to our systems security but it does show an area where we should look to improve the way we manage access.
I've also been getting up to speed with the way our Solaris systems work now. When I left the University we used a fairly traditional approach to running systems. A server would be set up to run specific applications. Now we use features like Solaris Cluster and Solaris Zones to create virtualised servers that allow us consolidate many different applications onto fewer larger servers. This gives many benefits in terms of the way we manage our servers but also lowers the total number of physical machines we have. That has quite a few cost saving benefits for us, partly in terms of saving electricity but also lowering our total maintenance spend, which is a significant part of the total cost of running a server. Over time we are aiming to move more of our services onto Zones. One of the major project we in our team have is to move the servers that run the SAP software to a new cluster of 3 machines. They currently run on 13 power-hungry servers, which at some point we hope to be able to flog back to our supplier! Good for the planet and good for the University!
Back
After a 3 year hiatus, I have the pleasure of coming to work back at Sheffield. I'm back in the Unix Admin team in CiCS, the same team I left although I'm now in a different office.
The first things I am looking at are security related. Then I've got to go and buy some coffee from the Union as at the moment I am "borrowing" some from Dominic.
It's good to be back!
Google Nexus One Review
Now, I should preface this with an apology. I’m no journalist. I don’t normally write reviews of things, so please forgive me if it seems like I don’t know what I’m talking about. I don’t. Given that my previous phone was an iPhone, I’ll draw a lot of comparisons between the iPhone and the Nexus One and lastly, I’m a techie person and I expect that most of the people reading this will be reasonably techie sorts too, so if I lose you, tough.
With that said, let’s crack on.
So, about 20 minutes after the Google Nexus One announcement last Tuesday, I placed my order. I’ve been thinking about replacing my iPhone 3GS for a while but was waiting for the right device to win me over. I was pretty sure it was going to be an Android handset, mainly because of the availability of Apps, wanting to get away from the crack-habit that is iPhone and my hatred of all things Blackberry. The Nexus One looked like the right phone.
The phone comes supplied in a presentation box, much as the iPhone does and came with the usual accessories – headset, 4GB memory card, charger, USB cable and a little sock to keep the phone clean and scratch free whilst it’s in your pocket (why Apple can’t include one of those instead of the stupid screen cleaning cloth I’ll never know).
The metal and glass design give it a robust build feel and it looks pretty good too. It is roughly the same dimensions as the iPhone 3G and 3GS and weighs a fraction less than the 3GS at 130g. It has a replaceable battery, which apparently lots of people consider to be a major failing of the iPhone. I’ve never had a need for a spare battery, although I do tend to keep a USB cable on me just in case. Data Storage is via a 4GB MicroSD card which was supplied with the phone. It’s not as big as the 32GB my 3GS had, but I can buy a bigger card, and when 64GB cards come out, I’ll probably get one of those.
Upon powering on, the first thing of note that the phone asked me to do was to sync up to a Google account. I use Google Apps for all my e-mail, calendaring and contacts information, so in this respect everything was ready to go. I just put in my login details and all my contacts, calendars and e-mail appeared seconds later. Easy. It also supports the usual Exchange and IMAP protocols too although I haven’t tested those out.
Call quality is fantastic, both for me and for the person at the other end. I used it in a noisy environment and the other person could here me fine. This is probably due to the active-noise cancellation that is a fairly unique feature on the Nexus One.
Likewise, the Display is fantastic. It’s an AMOLED device, which has been used in some phones before. I’ve heard reports of it being difficult to see in sunlight although I’ve not experienced this myself, possibly because I haven’t seen much sunlight for 3 weeks due to the weather here.
The first gripe I have is that the UI just isn’t as responsive as the iPhone. It’s by no means slow and is perfectly usable, but when you perform some actions, like a swipe between panes in the Home screen, it doesn’t feel as slick. One of the things I loved about the really old Nokia Series 30 phones of 10 years ago was their responsiveness. When you pressed a button, the phone appeared to be doing something instantly. The fact it took 10 seconds to send an SMS was a secondary concern… at least it felt quick.
The selection of apps available for Android OS that the Nexus One uses is good. Apple restrict what app developers are allowed to do with their applications for the iPhone, even down to preventing them from doing mundane things like changing the screen brightness. Android has a more liberal philosophy where app developers are essentially allowed to do what they like. This really demonstrates itself through the way apps are able to integrate with each other. There are a few nice touches in applications, such as the Facebook app, which integrates with your contacts to provides profile photos and contact information where it is available. Apps can also run in the background, meaning that my twitter app can automatically notify me when someone @replies to me.
Google Maps works really well and I like the Latitude feature. The GPS reception is incredible. It worked on a bus and indoors. It didn’t work inside a data centre i was in, but there were two floors of metal and concrete above me, so that doesn’t really surprise me. Sadly, it seems that Google haven’t enabled the Navigation feature for the UK yet, so I haven’t been able to test that out. Hopefully it’s just a matter of time.
Battery life seems OK. I tend to keep my phone plugged into power whilst I am at work and always charge it at night, so it rarely gets empty (unless I forget). It’s certainly no worse than my 3GS. One superb feature is the Battery Usage display. There is also a really nice paid app called Power Manager which can automatically turn on and off features like WiFi, 3G and GPS when your battery runs low to conserve power.
The built in camera is pretty standard stuff. A 5MP with LED flash and a real focusing mechanism. It’s fine for taking snaps, seems to work OK in low light, but it’s no comparison to a real camera. I took a photo in fairly terrible conditions (very bright light and snow) and it came out fairly well and other people have taken nice pictures. A little pink maybe, but as good as any camera phone I’ve used before. It’s not going to have me selling my Canon 5D mk II yet, although it does fit in my pocket a lot better!
I’ll confess I haven’t used the music app much. It seems to work although it’s not as well developed as the iPod interface on the iPhone. I’ve got a Spotify account which I use for music and that works brilliantly. The supplied headset has the usual mediocre sound quality you would expect from a free headset bundled with a phone. It’s similar to the iPhone headset, except with a more useful remote control on the cord with controls to skip tracks and pause.
One feature I have found really nice is Voice Input. Every text input field on the Nexus One is voice enabled. You can use this to speak an SMS or e-mail in to your phone and send it. It works quite well too although my colleague did say something mildly offensive into it and it suggested “Your mom”. Maybe it is so advanced that it understands “Your mom” jokes. Either way, it made us laugh a lot. The voice recognition is handled server side, so performance depends on your signal. It works fine on 3G or wi-fi connections, but can be slow to respond on older GPRS or EDGE networks. However, it does mean that as Google make improvements to the voice recognition technology, the Nexus One will be able to take advantage of it automatically.
The development platform from Android is nice. I was able to download the whole lot, including the Eclipse IDE and all the Android SDKs then write my first “Hello World” app and have it running on my phone within about 90 minutes, and all without paying anyone anything for the pleasure (apart from the cost of the phone of course!) – you can see the product of my endeavors here.
So, in summary. The iPhone still has a slight edge in terms of slickness of presentation, but in terms of what you can do with the phone, the Nexus One is way ahead. The hardware is fantastic. The software has a few little niggles, but there’s nothing where I think it’s particularly bad and I’m sure some of them will be sorted out with software updates (similar to the iPhone, iPhone 3G and iPhone 3GS).
If there is anything that anyone wants to ask me about the phone, then please feel free to comment.
Update: Flash support
For those asking if it has Adobe Flash support, it doesn’t yet but Adobe have announced that they will support it soon.
Update: Multi touch
There’s no multi touch on this device. However, the hardware is capable of it and applications are able to take advantage of this if they want. Dolphin Browser is one example of an app that does this.
From a personal point of view, I’m not that keen on multitouch. It requires two hands to make it work properly and it just seems awkward. I prefer double-click to zoom or zoom buttons as used on the Nexus One. However, I know that some people like it, so wouldn’t it be great if it supported both? If only Apple and Google can sort out their Patent problems.
Hearing
Created by Train Horns
The girls in rank order
Smashie and Nicey
Ok, prompted by Leo and Owen, the girls in rank order:
In reverse order, Smashie and Nicey/Alan Freeman style…
5. Annie – she felt nauseous when we stopped.
4. Laura Q – reasonable, but too much faffing about!
3. Edie – Plenty of experience, but not the best… maybe it was the flip-flops.
2. Caroline FH – very good, she’s had lots of experience!
1. Eleanor – a natural, with a vice-like grip between her thighs that inspires confidence.
If only you can work out what it’s a rank order of 
Getting VMware Workstation 6.5 to work on AMD64 Ubuntu 9.04
If you get a segfault when trying to start vmware on Ubuntu 9.04 (Jaunty Jackalope) you can fix the problem by moving /usr/lib/vmware/modules/binary to somewhere else (binary.old) will do.
The included 64 bit binaries cause vmware to segfault when starting up.
Good luck!
Afuse FTW
Today I started to try and get autofs and sshfs to play nicely with each other. Sadly, because of the way both autofs and ssh works, it’s not easy to do.
Then along came afuse, which solved the exact problem I had. Now I can do;
cd ssh/some.other.host
and I get to the root directory of another machine. Magic!
