What's git, and why do you use it?

At Freelock, we're always trying to figure out ways to do things better. Recently I started digging into a developer tool that's making, as Bryan over at the Linux Action Show would say, my head explode.

For a long time, we've managed our custom code projects and business documents in a central repository, called Subversion (also known as svn). Subversion is relatively easy to understand--it's like having a library of files you can check a copy out of, do some work on it, and then check it back in. Subversion is the librarian that tracks who has copies of what, and when they checked it out. So if Erik checks in changes to a brochure, and then Jill goes to submit changes to the same document, Subversion will say "hey wait a minute, that document has already been changed--you need to make sure you put Erik's changes in your document before I'll let you put in your document."

This is great for managing conflicts between people working on a single team, or for code that is being developed in relative isolation from the rest of the world.

The problem is, we're doing more than that--we're taking code from various open source projects and either customizing it or building new applications on top of it. And so when the outside projects get updated, we need to manually update anything we've written that depends on that code. There is no longer a single repository where we control our code--there is our code library, plus another one for every project we use.

This makes managing add-ons for projects like Joomla or ZenCart quite challenging, because our add-ons get scattered throughout the filesystem to be able to hook into the right place. And if we have to touch a core file, we're going to end up needing to re-implement our change with any update to that core file.

There are other issues we run into, managing our code and hosting, all of which take fairly time-consuming, manual intervention. Here's the list:

• Since we host and provide security updates for Joomla, Word Press, Zen Cart, Drupal, and others, we need to upgrade dozens of installations any time there's a new release that has a fix for a security vulnerability. With Joomla this has happened quite a lot, and every Joomla installation needs to be upgraded individually--and tested. And since each installation is slightly different, we can't manage them easily within a single repository, while updating the underlying code.
• Templates, modules, components, blocks, themes, plugins, and whatever. Developing these types of add-ons are our bread-and-butter. But code for these often get scattered across an installation, making it quite difficult to manage just our add-ons while we develop them, or roll back to earlier versions if there's a problem.
• The Dojo Toolkit, and builds. We're doing a lot of development with Dojo right now, to add desktop-like functionality such as trees, sortable tables, right-click menus, animations, and lots of other really cool things. However, if you don't "build" the code after you write it, it's painfully slow in a web browser. And due to the nature of how Subversion works, you can't easily store a built Dojo tree if you ever want to change it again. Which means you'd need to build it every place you deploy it. And on some computers, it can take a long time to build--on our demo server, one of our projects currently takes 8 minutes.
• As we get more directly involved with open source projects like LedgerSMB, we're finding the need to change core files while we hack away at some particular feature. To do this, you create a branch of the code, work on your feature, and then merge your changes back into the "trunk." If you don't have access to save directly to the project repository, doing this gets a lot more complicated.

Git to the rescue. Git solves all of these issues. Read on for a technical discussion of how.

Managing lots of installations with Git
We haven't yet started doing this, but for projects like Joomla and ZenCart, this is going to be really effective. It's not quite as helpful for Drupal, which allows for a bunch of sites to be centrally managed out of the box--though even there it will help us recover quickly from a failed upgrade.

Here's the basic approach:

1. Create a master git repository, either by pulling down the Subversion repository using git svn, or just unpacking tarballs of a release and importing into our git repository for the project.
2. Create a new git clone for each installation, and drop into each installation directory. Create a current branch for that repository, and rebase it to the current installed version.
3. Commit local modifications to the repository. Make sure the .git directory is getting backed up, and is not accessible through the web server.
4. When a new release is available, use git pull to update each local installation. Use git reset HEAD^ --hard to undo if there's a problem.

Managing submodules, customizations
This we get pretty much for free by having dedicated repositories for particular problems. Managing add-ons we'd like to use elsewhere can still be challenging, but we can pull customizations from other repositories to assemble custom packages--it's at least easier to manage these across installations.

The main thing to do is to make sure when you're developing on a particular add-on, you're always doing it in a branch dedicated to those changes. Then you can merge them into other projects more easily. It's also possible to "cherry-pick" commits involving particular files, and pull those changes into a new branch if development ended up getting mixed in with different features.

Dojo Toolkit builds
We're huge fans of the Dojo Toolkit. It makes developing really sophisticated browser-based applications much easier, not just providing a large set of active widgets but also a data abstraction layer and an encapsulation system for your own code.

Its biggest downside is that every feature you want to use means adding another file that the browser needs to load. And we use a lot of features... the base package loads about 20 files, and in our applications there are well over a hundred. This takes a long time to load, and isn't very scalable.

The solution? Use the Dojo build process. A Java program can build all the features you want to use into a single compressed, stripped, javascript file. The difference in speed is incredible--applications suddenly load right up. And this build system can easily build your Javascript as well.

The problem is, this build system is not compatible with Subversion, because Subversion scatters its working copy data inside each directory, but the build system deletes the directories every time you build. So you can't easily manage built code inside Subversion.

Git stores all of its repository information in the top level directory, so you can easily build and re-build Dojo and manage it without any issues.

Do local development on open source projects
Now here's the traditional reason to use git: to allow you to manage changes to open source projects without needing write access to their code repositories. We simply set up git repositories to track their Subversion repository, and then we can use all the features of git to manage our customizations.

When the original project has new releases, we just pull them into our repository and git handles most of the merging for us. If we want to contribute back to the project, we can copy them into our Subversion tracking branch and commit them straight back to Subversion--or send a patch.

We're starting to set up public git repositories for projects we work with extensively. You can browse them here: http://git.freelock.com. You can clone and pull updates from here: git://git.freelock.com/git/<projectname>.git. You can read more about how to manage non-standard Subversion repositories in git here.