Managing development projects with Dojo and Git

At Freelock, we're big fans of the Dojo Toolkit. It's a Javascript library for providing data-backed widgets in web applications, on-the-fly graphing, animations, and much more. As we move more of our web applications to the browser, we keep pushing the edge of what's available, and sometimes hit bugs in Dojo itself. So as I mentioned in a previous post, I often find myself teetering between the most recent stable releases and the tip of Dojo development. And I think we have the best way of managing these changes of anyone.

To complicate things, Dojo uses a "build" process to optimize files, packing a bunch of individual widgets into a single compressed script. It's a great way to strike a balance between having a rich toolkit with a huge number of widgets, combined with a system that only loads exactly what you need. The build system can also pack your own, non-Dojo javascript, making your applications load and render much quicker. But this build system carries its own set of management challenges, especially if you want to manage production systems with the same source-code management tools as your production system.

We used to use Subversion to manage the code for our projects. Dojo still does. But the complexity of managing Dojo code, Dojo builds, and our own code proved to be far too cumbersome, so I learned how to use a distributed version control system called git. And it's fantastic.

So here's a little primer on how we manage projects that use Dojo and builds, with git.

Project Setup

We put Dojo into a git "submodule". Without going too deep into how to use git, we basically create a clone of our main Dojo git repository as a submodule of our main project. This allows it to be omitted in production deployments entirely, as well as making it easy to switch Dojo versions independently of the rest of your project.

We maintain our own git clone of the entire Dojo project at git://git.freelock.com/git/dojo.git. Feel free to use it in your projects, or make your own clone. Unlike the new official Dojo git repositories on github, we have all of the main projects in our git repository: dojo, dijit, dojox, and util. It's updated from Dojo's svn repository every 6 hours. You can browse it via gitweb at http://git.freelock.com. For more about how we created it, see here.

So getting Dojo into your project is simple:

cd <project_root>
git submodule add git://git.freelock.com/git/dojo.git public_html/dojo

... and then wait for the Dojo repository to get cloned into your tree. (Note that it's currently 314MB -- it can take a while to download!)

You now have the entire dojo tree, with trunk checked out, in the public_html/dojo directory.

Dojo's a great base for your own Javascript, with its class, module, and AJAX functions. I recommend creating another directory with the name of your application inside public_html, at the same level as the dojo directory. For example, my Auriga project has these directories:

public_html/auriga
public_html/dojo
public_html/dojo/dijit
public_html/dojo/dojo
...

Then in whatever template or file bootstraps your application, add a dojo.moduleUrl path to your code.

Notes:

• You must be in the top directory of your main project when using git submodule
• git submodule creates a file called .gitmodules where it stores the path and remote repository for the submodule. The check-out commit id is stored in the corresponding path, inside the parent repository.
• Many of our custom applications assume the project root contains a public_html that is the web root for the site. This doesn't have to be the case. If your .git repository is under the web root, I highly recommend making sure the web server denies access to all .git* files for security reasons.

Switching between trunk and a tag

So far so good. We have the basics of an application set up. We've been developing with the main Dojo trunk. But what if you want to use a stable Dojo release?

Simple. Just check it out:

cd public_html/dojo
git tag     # shows you a list of available tags
git checkout svn/release-1.3.2   # checks out Dojo version 1.3.2

... Refresh your web browser, and you'll get that version of Dojo. Note that when you're under public_html/dojo, any git commands work on your local clone of the Dojo tree. You'll need to cd up to your main project to commit changes to your code. When you're in your main project, git status will show you that public_html/dojo has changed (since it's now on a different commit). To make sure the Dojo submodule gets correlated with your commit, be sure to:

git add public_html/dojo

... and commit with your changes. Also note that when you check out a tag, you don't necessarily get a tracking branch, so don't make any changes to the Dojo core files without creating a local branch first.

To switch back to trunk is even easier:

cd public_html/dojo
git checkout master

Cloning your project, updating submodules

So now you want to let another developer work on other features. Being git, this will be distributed. However you get it there, create a clone for the other developer to use.

What you'll notice when you clone, is that there is nothing in public_html/dojo. How do you get it there? If you try to git submodule add it, git won't be happy. Instead, you use some other submodule commands (remember to cd to the top directory in your project first):

git submodule init    # Prepares git for updating the submodule
git submodule update  # Copies down the submodule repository
      # from the original source, checks out the branch specified
      # by commit id (again, prepare to wait!)

Now your clone has the same Dojo code checked out.

git submodule update is also the command that checks out a particular commit from the Dojo repository. If you create branches in your main project, or check out an earlier version, and the Dojo version associated with that commit is different, git checkout does not update it--you need to run git submodule update. (And unfortunately, git submodule update will go out to the remote repository and download any newer commits, just like a git fetch--which can take a while).

Creating a build

So far so good. You can see how it's a bit easier to manage associating particular commits of your main project with specific versions of Dojo, by using git. But when you get to builds, you see how git becomes vastly superior to Subversion, mainly because of its central .git repository instead of scattered .svn directories.

To create a Dojo build, you usually first create a "Layer" file, and drop it in utils/buildscripts/profiles. Instead of putting it there, we put our profile in the main project, in a subdirectory we call "configs." So setting up and creating a build looks like this:

cd public_html/dojo/util/buildscripts/profiles
ln -s ../../../../../configs/auriga.profile.js

I never check this symlink into the Dojo git config--I just manage it in the main project. But regardless of what tag or branch of the Dojo repository I check out, the profile remains listed as untracked. Making it easy to do a build anytime I want. And I can easily re-create the symlink if something happens to it. For convenience, I put the build command into a comment at the top of the profile file. So now, an actual build:

cd public_html/dojo/util/buildscripts
./build.sh profile=auriga action=release optimize=shrinksafe.keepLines \
releaseName=build
cd ../../../   # now I'm in public_html
rm -Rf build   # delete the old build
mv dojo/build . # copy the build out of the dojo repo, into main project

Now here's the cool part: My Dojo build is now in the main project, not in the Dojo repository. I can add it to git, see what has changed from the prior build, commit it with the rest of the code. And very easily deploy it to production.

Deploy to production

If I don't plan to develop on a particular clone of my project, I don't need the Dojo files, at all. All I need is the build. So after committing a new build to the main repository, I just pull it to production and it's ready to go.

One trick I do is build my templates with a switch to make it so I can easily switch between built Dojo and non-built Dojo. With a build, you need script tags to load your build layers, and, obviously, they need to point to the path of your build. For development, you should be loading the dojo.js file from the submodule path. I set a constant in a config file that I keep outside version control (though I keep a model config file in the project root to track any changes to the config).

So on the development instance, you can toggle a constant to change between a build and the raw Dojo files. On production, you can just leave it with the built version. And you never need to use "git submodule init/git submodule update" on your production sites, at least not for Dojo.

Updating Dojo

Back to development. Now you want to get the newest changes from trunk. If you already have the Dojo master branch checked out, you can do either of the following:

cd <project root> ; git submodule update  # or
cd public_html/dojo; git pull

Git submodules are complete clones of the entire Dojo tree, and git fetch, git pull, or git submodule update (in the parent project) will all download fresh commits from the source repository.

However, there's one thing you don't get with these: fresh tags. So when the next Dojo stable release comes out, to see the tag, you need this to see it:

git fetch -t   # -t fetches tags in addition to commits

Developing on a branch

One final tip, related to my previous post. In all of the above, I assume you're not making any changes to the Dojo tree, you're just checking out tagged releases or specific commits. What if you need to backport a patch to a stable release, or create your own code changes?

Just remember that your Dojo submodule is a full-fledged git repository. You can create your own branches, merge, cherry-pick, format-patch, and all that other glorious git goodness. Just remember that as soon as you commit to it, other developers on your project won't have those commits. You do risk breaking their development tree if you make a commit that only exists in your instance of the submodule. You'll have to work out your own way of pushing/pulling commits across your various Dojo repositories. I leave that for another article...

I hope these tips help, and feel free to contact us if we can help with your project, or leave a comment below if you have another great tip!