Tech Life Index your brightness

Capturing Context

Each organization in a company has Their Favorite Application. It’s not truly their favorite application; it’s just the application they must use in their particular capacity in the organization. Stand up right now and walk into an unfamiliar part of your building and stalk your co-workers. If someone stops and asks you what you’re doing, tell them, “Rands sent me” and vigorously nod your head.That always works.

As you walk the hallways of this strange new part of your organization, look at their screens. What’s the consistent application sitting on their monitors? Is it Excel? Well, you’re probably in some area of Operations. Are you seeing a lot of Word? Maybe Legal possibly Tech Pubs. Is the cube empty? That’s Sales.

The most common application in Engineering is an editor. Whether it’s a terminal window or the world’s fanciest integrated development environment, Their Favorite Application is a code editor, but it’s not their secret weapon… that’s version control.

The concept behind version control is simple. It’s a central network repository for all the files of which a project is comprised. If I want to edit a file, I run a tool (which works alongside my editor) that makes a local read/write copy of that file on my system. I make my changes and then, using the same tool, I check in my file to the network. So, what’s the big deal?

Usually, there is no deal. You merrily check in and check out your files with no fuss. The deal occurs when you realize that software projects are often massive collections of files which are edited by teams of people. A version control system solves the problem that occurs when two engineers have checked out and have changed the same file at the same time. Whoever checks in their changed file first has no deal. When the second one checks in, that engineer receives a message that warns, “Hey, this file has changed while you had it checked out. Whaddayawannado?” The user then gets to figure out how to merge the two files into a consistent working whole.

That’s version control as a traffic cop and that’s cool because it prevents folks from bonking each other on the heads, but I haven’t gotten to the major cool and that’s the other thing I do when I check in. I don’t just check the file in; I also include information about what I changed:

“Rands added a new blingleforth function. It rocks.”

The version control server then copies my new version of the file up, tags the new version with my name and my comments, and increments the version number associated with the file.

Let’s ignore the useful fact that every single version of the file is stored in this system and focus on the comment I included with this change. This is the big deal. This is the secret weapon in Engineering. We not only save every version of our work, we also capture the context of the change. Version control stores the thoughts that made our ideas bright.

If you’re thinking, “My, what a quaint nerd custom”, if you’re not having a Holy Shit moment, think about two products: del.icio.us and Flickr. Both have built their feature sets around capturing context, and by context, I do mean tags. Each time someone adds a new link or photo to these services, they can add whatever tags they like. No rules. Just start typing words regarding what makes the link or photo relevant. That’s context.

When you start stumbling around Flickr and del.icio.us, you realize the value that is created when people choose to capture and share the context of their content. At SXSW, I was in awe of the folks who were taking the time not only to capture and upload their photos to Flickr, but also taking the time to carefully tag all their content. Thirty minutes after a presentation, there were dozens of tagged pictures sitting in Flickr for the presentation I just watched.

So what?

Think of the big project you’re working on right now. For me, it’s this article. I’m merrily typing away and hitting the Save button every twelve seconds because I’m a twitchy saver. Comes from years of flaky Windows applications that liked to crash. If you save a lot, you’re not screwed.

When should I capture context on this project? When should I stop and capture the thoughts about what I just wrote? Whenever I’ve created significance. I’ve been keeping track of these moments while I’ve been writing and so far, they are:

  1. New intro down, borrowed from version control article I can never seem to finish.
  2. Removed Wikipedia from example technology — Flick’n’del.icio.us are enough… don’t confuse them with wikis.
  3. Keep moving disclosure paragraph around… haven’t found a home, yet. Might be wrongly in love with this paragraph.

Do these comments matter to you? No. Do they matter to me? Yes. Do I want my favorite editor to prompt me every time I hit the Save key for context? No. I want another verb, let’s call it Wow, and let’s have it mean, “I’ve done something significant to my project and I want to capture the context of that change”.

This is not an obvious activity for most people. In fact, huge passive aggressive battles have been fought within my engineering teams over these change comments. It’s a fight between those who are lazy and just want to check in their files and those who know that, while having the code safely in version control is good, understanding what is happening to the project on a day-to-day basis is even better. It’s called a status report.

That’s right; I finally found my technology angle on killing status reports. We need our tools to allow us to capture context at the moment we’re being bright not Friday at 4pm when we’re trying to get the hell out of work. How much easier would your status report process be if all you had to do on Friday afternoon was ask Your Favorite App, “Show me all the Wow for the last week”? That report alone is enough incentive for me try to remember to record my Wow amongst all my twitchy saving.

Nerd Disclosure

I’m serious a version control nerd. At Borland, I was the junior engineer, which means I was saddled with build duties. This means if the product did not build, someone yelled at me. This gave me a strong incentive to build an application that forced each engineer to make a comment, no matter how small, each time they checked into the project. IT’S NOT ME. IT’S HIM. At Netscape, I watched in awe as CVS was merged with bug tracking and build systems via primal web applications. I still drool over Tinderbox. At my start-up, I was the guy who took Microsoft SourceSafe out behind the building and kicked the shit out of it.

I live and breathe version control because I see the value. Each year, I learn more about more Favorite Applications. I learn how executives live and breathe presentation software. I learn about the magic that those folks in Operations can coax out of Microsoft Excel. Everyone is hard at work creating stuff, and, some minor tweaks to Our Favorite Apps, we can wrestle version control from the nerds and help everyone index their brightness.

26 Responses

  1. Rands, I hope that when you were Junior developer at Borland you were not in change of Team Source… because if you were we’re gonna need to talk 🙂

  2. The day I figure out how to design with version control that’s as low impact as coding with version control, a sacred circle somewhere will be complete. I can’t count the number of times I’ve wished I could go back a few revisions on a design and work from there. I’ve started saving incremental “snapshots” of designs, but I still only do that when the dust settles, not when I’m trying 10 different ideas at once.

  3. As a response to Wilson Miner above, I’m a huge fan of Subversion which will also does binary files well (not like CVS where I cringed every time I checked in a binary wondering if I’d ever be able to see it again)

    Merging is still a problem though.

    http://en.wikipedia.org/wiki/Subversion_%28software%29

    Great article Rands, I’m all about the required commit notes.

  4. JohnO 11 years ago

    I’ve used CVS and MS Sourcesafe before, and come to really like Arch (coupled with the web ready ArchZoom), and Baz. They work wonders together.

  5. Thanks for the vindication. Since I started using CVS about a year ago, my whole life has been sucked in. And the voice in the back of the head always telling me “What if…?” is now only a whisper.

  6. Wilson, there is a somewhere where the sacred circle is complete–Smalltalk. The commercial Smalltalks allow nearly effortless version control with tagging, branching etc at all levels, from individual methods all the way up to entier packages of classes. Completely integrated with the code browser. Squeak and Montichello aren’t quite there yet, but it will come.

    And since in that place the design is the code plus (automated) unit tests, life is good. If only you could get there from here…

  7. Alex Montgomery 11 years ago

    I’ve been thinking for a while that (on the Mac, at least) simple single file revision control would fit nicely into the now-unused (but unlimited) file forks, using extended attributes for the tags.

    Since I don’t write any major apps, and this feature would be mostly ignored as a stand alone ‘manager’ app, I haven’t tried implementing it. But if it were just a standard feature of all Document based apps, I bet it’d see a lot of use, and people would wonder how on earth they got by without it.

  8. Am I the only person who wishes they could make contextual notes on folders and files directly in the operating system? Two weeks ago, I was sort of developing a system in my mind to put a readme.txt file into each important directory on my system. I was then going to write a Ruby script to keep a history of all of them somewhere. I guess I could check my entire My Documents into version control, but that seems somewhat overkill, no?

  9. Joshua Volz: I keep My Documents under version control, it’s great. Disk space is cheap, and I have complete confidence that I can find any version of any document I’ve produced over the past year or so. See the article at http://www.onlamp.com/pub/a/onlamp/2005/01/06/svn_homedir.html.

    Rands: any comments on which software you prefer for VC these days? And how about the distributed versus the centralized distinction?

  10. Joshua: Add me to the people saying “No, it doesn’t seem like overkill at all.” All my personal documents are in a version control system. At least, all the ones that I actually edit. I don’t do it for digital photos and music and other things that are quite large and are effectively write-once, but it would be trivial to do it for a readme.txt file in the photo directory nonetheless.

    One of the other cool things about version control that Rands didn’t mention is this: it’s like rsync, but better. Any time I stop using a computer, I check in my changes. Any time I’m about to use a computer, I run an update. I have my documents scattered over a home computer, a laptop, and an office computer, yet keeping everything up-to-date on all of them takes no thought at all.

    And, related to that: Last year, my laptop crashed and wouldn’t boot. Half my doctoral dissertation was on it, and this looked like a failed hard drive. My reaction was, “Drat, that will be a bit annoying to fix.” I didn’t worry about lost data at all — everything on it was backed up and up to date on the version control server.

  11. Tyler 11 years ago

    I have no idea what any of this really is, as far as version control, and I wish I did. If there was a draft for people who wished they could have a serious case of N.A.D.D. then sign me up. Yes. Im a wannabe. But the ironic thing is I have no idea what IT is!!

  12. As you use VC for more and more of your “stuff”, you’ll also find that file formats matter. For instance, while a diff of two versions of a Visio document is worthless; a diff of two versions of a (uncompressed) Dia document lets you see what you changed.

  13. I’m on a fairly huge project (we’re in year 5 with no end in sight) for the federal government, and we have just cut over from CVS to ClearCase. I have transitioned from application architect to librarian, because I’m a sucker. I will just state that I am in love with ClearCase. It is awesome.

    One key thing that we’ve implemented that you reference is you *cannot* check in without making a comment. Your commit will fail. Developers are free to make meaningless comments, but they are all contractors. The government project managers LOVE reports and yelling at people, so seeing a developer putting in comments like ‘Matt is teh suX0r for making us put in comments’ will tend to get them noticed. Problem solves itself.

    I hadn’t thought of using it as a status report, but I’m going to start right now.

  14. It says a lot about me that I assumed the “Favorite Application” you were going to cite would be a certain bug tracking database. 🙂

  15. Alex: RCS is perfect for that. It’s a standard unix utility, it might already be on your mac. I use it for small stuff like config files and short text documents.

    Version control is something a lot of people don’t learn in school, and they should. I’ve heard a few hiring managers bitch loudly about getting green CS majors who don’t even know CVS.

  16. Oh. Also. CVSweb is the best application ever. Side by side color diffs in a web browser -and- a great annotated blame listing. Makes sifting through giant piles of spaghetti code worked on by other people much more tolerable.

  17. Amen to using Version Control. I feel aboslutely naked without it. I tend to use Perforce because that’s who I work with, but when licensing/money is an issue am happy with Subversion for the basics.

    Re document control, I wrote the Perforce plugin for MS OFfice to scratch a personal itch – the ability to easily check stuff out/in from within the app. For Word, it uses the built in diffing capability to diff the current doc with checked in one, or any combination of 2 from history of the file. The functionality of the plugin isn’t huge, but I do find it makes a tremendous difference to producitivity (e.g. diffing without it is sufficiently painful that I almost never did it – now it’s easy).

    The thing that worries me is the number of applications out there e.g. for blogging which have no concept of version control. Also, I have rejected a number of tools which store text in a database because I want the security of having versions of all my articles.

    Re the comments, Perforce requires *something* there. Interestingly I maintain the scripts to import from VSS and have used them at many clients, and many an imported repository has swathes of checkins with no comments in VSS!

    Robert

  18. John Whitlock 11 years ago

    I like your point about separating the “Wow” from the object. I finished tagging all my MP3 files a while back, but I did it using MusicMatch, which makes tags that don’t play nicely with the rest of the world. Artist, Album, Art work well, but Preferences don’t. It also looks like no one with programming skills has had this particular itch, so I’ll have to roll my own solution (Python + libid3).

    My point it, there is stuff that belongs in the ID3 tag (the artist, album, title), and stuff that probably shouldn’t (play count, preferences, maybe the album art). I really don’t want 3 bits changing in a 10 MB file every time I play an MP3, triggering an incremental backup. That stuff should be in a separate file, or better yet, a database. That stuff is the WOW for MP3s, or, to use another term, play count and preferences are orthogonal to the music. When I want one, I usually don’t want the other, and they should be separate.

    Maybe this is why MS Office’s internal versioning is so annoying to me. We use track changes all the time, but we end up copying the whole file (XXX_Proposal_v03_JW.doc) and emailing that around. Maybe the new XML format will be something that will look attractive in CVS.

    BTW, I hate revision history in code files. Please leave that stuff in the CVS system, where it belongs!

  19. Hooray for kicking the shit out of SourceSafe. I just regret that you didn’t completely kill it.

  20. Joe Pellerin 11 years ago

    i am no microsoft apologist – and certainly not a superengineer like the rest of ya’ll… so, please, someone tell me: other than the fact that it is a microsoft product (sufficient for suckhood in some circles), *why precisely* does vss suck so much?

    from another friend, i’ve heard that it can corrupt very large builds (vlb’s)… i, personally, make no vlb’s – mostly vsb’s. it seems to check code in, check it out, keep context and version history, and check differences. these are all the things that the amazing rands listed as being awesome about version control…

    so, please – halp! can you point me to a whitepaper that explains? “why msvss sucks goat dingles” or something? (yes, i googled the equivalent to no avail)

  21. Aaron Digulla 11 years ago

    Joe,

    MSVSS sucks because it gets in the way when more than one person works on the code.

    It can’t merge, it can’t share (once someone has the code checked out, it’s as if he’s the only person in the world who has the code).

    And tool integration is … well … User interface … automatic builds …

    A tool will, can and should never know better what I want to do than I.

    I’m a huge fan of CVS but lately, I’ve switched to Subversion because it fixes almost all of the CVS bugs (svn rename dir1 dir2, yay!)

    And once you’ve seen the Eclipse Synchronize perspective, version control will never be the same again.

  22. J Osborne 11 years ago

    Joe, MSVSS also gets heaps of derision because it is worse then the free competitors (CVS for around a decade, SVN aka Subversion for a year or 3 depending on your comfort with using new software for such an important bit of infrastructure).

  23. Floid 11 years ago

    JFS (and probably NTFS and who knows what else) support Extended Attributes, which, like HFS forks, theoretically allow for arbitrary tagging and so on.

    The bitch is that sometimes you want to preserve metadata, sometimes you don’t, and it’s rarely obvious which operations will/won’t. (For instance, Mail.app appears to happily attach resource forks to messages, and various semi-automatic packing formats have been around for ages, while back in OS/2 or *NIX and-probably-NT but-I-don’t-touch-that land, you’d have to think a bit to accomplish the same.

    Of course, if versioning data is stored with the file, you’re also screwed when you delete that file. Unless you’re using some log-structured-FS that doesn’t exist and would raise more ‘So, uh, exactly how far can I roll back? Oh crap shouldn’t have filled the slice with hentai DivXes ^H^H^H^H^H^H^H^H^H^H^H^H^H^H a giant temporary file or core dumps or something last night’ issues if it did.

    This is all horribly obvious but I’d like to see someone figure out how to do consistent interaction design for it because flat files are getting kind of boring.

  24. Floid 11 years ago

    Okay, here’s an interesting, halfassed, postmodern idea tangentially related to the problem and the design of editors in general.

    Imagine the following:

    -An editor that chews on input and spits out (gasp) XML atomically, relying as anything does on the FS syncer for on-disk integrity, with no attempt at garbage-collection or cleanup. This creates something equivalent to the mess of a .DOC file and basically keeps a versioned log for the session.

    -The editor has a Wow-key, if you’d like, to shove in a comment or annotation. It automatically logs time/date and maybe spits an anchor tag into the text stream to handle divining cursor location.

    -By default, whacking save forces a sync on the working XML, walks said XML mess, and spits out a clean output file in whatever export/interchange format is appropriate for the app.

    -The XML DTD used for this ridiculous idea is something ridiculously simple and standardized that mostly consists of tags for data and tags for metadata.

    -*If* the filesystem supports a more efficient way of packing metadata, an external daemon shipped with the filesystem handles the task of walking closed files, checking for the appropriate type declaration, and doing the conversion in the background.

    -Somewhere in the operating system’s VFS layer or whatever lives a function that calls on the filesystem to return a handle to a virtual? or temporary? file that’s a dump of filesystem_proprietary_metadata_packing back into the standard XML representation. Possibly/probably the FS is required to handle this as a true flat file somewhere for data-integrity purposes until it’s closed and the asynchronous packing daemon can catch it again.

    This maybe adds minimal complexity to everything, with most of the work falling on the authors of filesystems that bother to support EAs or Resource Forks or whatever, who in tradeoff would get to see applications actually finally take advantage of them.

    There may or may not be a convenient way to skip the flat file step, but give me a break, I just came up with this while sucking down a cigarette.

    (See references to the ‘Grande Paihia’ browser — http://www.paihia.co.uk/ though there’s nothing there — which is another ‘hey XML might be good for something’ project, translating documents of various types into some kind of conformant representation to keep the renderer simple.)

  25. Floid 11 years ago

    Okay, I left out the part about the editor — or if not the editor, at least some utility — being smart enough to let you walk-for and dump a chronological list of annotations instead of just doing the floating-sticky-note bullshit.

    Annotations/’Wow-notes’ without anchors in the current version of the text would be preserved and listed with broken or missing links, instead of lost forever when the associated paragraph object or whatever the hell is deleted.

  26. Giles 9 years ago

    Completely with you on the use of SCM systems outside the tech sphere. I used File Hamster to protect the office files (spreadsheets, documents, etc) on my last project – seems to work quite well. The nicest feature is the balloon popup that appears bottom-right, which allows you to enter comments on a recent save. This seems to tie in nicely with the wow feature.

    BTW The reason I’m 18 months off the pace on this post is that I just read about this in the book. Good job, Rands!