Better Development Environments

Jon Jensen

End Point Corporation

What is this about?

  • I work primarily with web applications in ecommerce, but many of the lessons I’ll talk about apply to software development in general, not just web development.
  • The only true small-scale development is a single developer, working on a single project, that starts and stays small, and progresses fairly linearly. Anything more than this — two developers, one developer with a few simultaneous projects on the same codebase, a project that becomes important to its users, a project that grows beyond a weekend hack — needs the same infrastructure as any large-scale project. But usually doesn’t get it.
  • I’m focusing on camps, which is a particular development system we created at End Point, but many lessons can be applied piecewise in any development environment.
  • I think camps are an example of the way open source best works: solving a real problem, finding that the solution applies in another place, then another, then generalizing it and releasing it for others to use. Camps are used extensively by End Point and our clients, and are fairly mature, but the camps open source project is still young.


  1. Set up your new server or hosting account
  2. Start writing code
  3. Enjoy positive user feedback
  4. Step on other developers’ toes but figure you’ll deal with version control and so on “when we have time”
  5. Finish development and go live
  6. Hack bugfixes and new features directly in production
  7. Profit!?1!@1 Suffer!

Developers need a place to work. The easiest place to do this is by editing code on the production site. When you’re working on a new site, there is no production site yet, so what you’re working in is clearly the development environment. When it’s time for that to go live, everyone’s in a hurry, things are all set up, so you switch it live. From this moment on, where do you develop? It’s of course very dangerous to keep developing right there, but it solves the immediate problems, everyone’s busy, and you make comrpomises.

Frameworks that build in support for development and testing, such as Ruby on Rails, are a huge step forward and greatly reduce this problem if their conventions are followed, but they also introduce some new problems of their own which I’ll cover in a few minutes.

An exaggeration?

Perhaps. But there’s still plenty of that around.

Modern web development frameworks provide some good conventions and defaults, and make it hard to do as badly as the HOWNOTTO describes.

However, while it may seem that the problem of development environments is solved in some modern frameworks, it’s not entirely.

What are camps?

Camps: A system and set of conventions to make it easy to manage many parallel development and staging environments, and keep them synchronized with the production environment.

“Camps” is just a word, not an acronym.

Camps leverage popular open source tools and build upon them (for example, version control systems).

A camps demo: Spree (Rails)

Spree is a new open source Rails shopping cart.

I’m using it here as an example of a simple application running in camps.

See End Point Spree Development/ for more details about End Point's work on Spree.

About the demo server

Turkey Store camps

  • Show the camp-index.
  • ssh to the server as user jon.
  • Create a new camp.
  • Scroll back in screen to review what mkcamp did.
  • Reload camp-index to show the new camp in the list.
  • The new camp site should look the same as camp0.
  • ls -lFa camp$x and look at httpd, pgsql, rails.
  • Show in "ps uxww" the different services per camp.
  • netstat -nl | grep $camp_number
  • Run psql_camp, "show port", show that it’s a separate database cluster.
  • create role whatever;
  • Switch to camp$y, psql_camp, show that the new role from camp$x is not there.

“Different makes a difference”

If you’re not using a current near-identical copy of your production database, you’re developing against a toy.

Say you develop with Rails locally. You’ve probably got some Apache redirects, rewrites, subdomains set up, and they’re not reflected in the WEBrick toy webserver in development. You can’t test Apache changes there.

Are you using the same hardware (32- vs. 64-bit), operating system, libraries, web server, database, as the production system and as the other developers? Who really wants to deal with the Ubuntu/RHEL/Debian/Mac OS X differences, WEBrick vs. Apache, Postgres 8.1 vs. 8.3 version differences, setup & keeping database fresh. What about development environment accessibility, automation, backups?

Camps to the rescue

Most simple sites don’t stay simple. If they’re at all successful, there are more projects, more people, more time pressures. It’s better to plan for success.

Camps require some initial investment, force some discipline, and require a little occasional maintenance. The value comes out when you’re dealing with a system that actually matters. And the payoff usually comes in saved time and sanity, more quickly than you’d expect.

Camps mean:

Also helpful:

  1. Use https in development just like in production
  2. Use subdomains in development so cookies are separate
  3. Scripted destruction of old camps
  4. Outgoing email intercept
  5. Unit tests
  1. Except for the weird bugs that only seem to crop up after code’s already been promoted into production. For example, you never bothered setting up SSL for development, as that’s obviously not needed. But browser security warnings dealing with http vs. https appear in production and you realize you need SSL in development to catch them.
  2. And there are pesky cookie bugs that are hard to reproduce but that keep coming back. It turns out that some of your development environments are stepping on each other’s cookies, and some development cookies are even messing up production. That “shouldn’t” matter because it just affects internal people, but those internal people take and process orders, do data entry, test your code and report bugs. Setting up a separate subdomain for each development environment solves the problem, but what a pain!
  3. It needs to be just as easy to destroy a no-longer-needed camp as it was to create it. But that involves a little bit more than just deleting the files, so we’ll script it. The database, app server, and web server need to be shut down, the files removed, and the camp removed from the camps index.
  4. Outgoing email is an important part of many web apps, and where the app server supports it, we have camps automatically configure it to intercept all outgoing email and reroute it to the developer. This makes testing easy, even with various recipients. If the app server doesn’t support it, there’s a daemon called "fakemail" that can intercept it server-wide.
  5. Unit tests are much talked about, not often enough done. Use them, and benefit. Camps don’t do anything about them specifically, but every developer has a place to run them.


Camps are language-agnostic, and have been used so far with:

Application frameworks

Camps are also framework-agnostic, and have been used so far with:


  • Change colors and title of the Spree header.
  • Show the change.
  • git status; git diff
  • git commit -a; git push
  • View camp0; nothing changed yet.
  • Switch to staging user; cd camp0 && git pull
  • Refresh camp0 display, now change should be there.

Glance at a real camps system

They have a lot going on:

  • Many projects
  • 55 camps
  • 18 developers
  • 3 different storefronts (in every camp)
  • 3 staging camps

The actual camps are firewalled so as to keep search engines from indexing them and causing trouble for their production system.

Let’s look at a snapshot of the camp index from one company’s actual camp system.

Note the comments have links to RT ticket numbers in their private ticket tracking system.

A few of the 18 “developers” are actually business people or graphic designers. They’re less likely to do the version control work themselves, but they edit files in their camps and have a developer to the committing for them.

This camp index shows running camps in bold as a convenience. As we see, it’s typical for not all camps to be running at the same time, to conserve server resources when a project is back-burnered for a while.

Deployment challenges

  • Office firewalls blocking “weird” ports
  • Migrating into camps and discovering even more application spaghetti and goo than expected

You should expect the unexpected when it comes to firewall. A surprisingly tight egress firewall on the camps server may be only the beginning of woes. The business’s various offices may have tight egress firewalls per location that don’t allow access to your camps server at all, or to ports 9000-9199, and it will take time to get their network administrator to open this up. Some network administrators have the funny idea that “opening up 200 ports” is a very dangerous thing to do. We’ve always had it work out, but it may take some discussion and time. In the worst case you could set up a VPN between their office and the camps server.

All sorts of weird and little-known integration with services on other servers, long-forgotten scripts, cron jobs, and apps on the production and developent servers, etc., will come out as you install an existing app into camps. There’s nothing you can do about it except roll with the punches. It’s not because of camps; it’s because you’re trying to make order out of disorder.

Resource challenges

On busy servers with many active camps, we’ve at various times run into limits of:

  • RAM
  • semaphores (Apache)
  • shared buffers (Postgres)
  • disk space
  • CPU and I/O
  • 100 camps (arbitrary, but real)

In short, hardware is cheaper than developers. Camps optimize for developers and business users over hardware. For camps you also will want to invest in the server, while the client machines aren’t a big concern (as long as they run an ssh client and a web browser).

Because camps have their own running daemons for each part of the application stack, they use more RAM than a system that cuts corners. RAM’s cheap enough that you should just buy more. But it may be different for the business people to think that you’ll want more RAM in your development box than in production, or the same amount at least.

Each Apache daemon uses semaphores, and the default Linux kernel needs to be configured to have more. It’s easy to do.

Each Postgres daemon uses some shared buffers, and the kernel needs to be configured to have more. It’s also easy to do.

Get lots of cheap disk. You’re going to have a complete copy of the whole application stack: docroot, app, database. To save some disk space you can use symbolic links for large docroot segments such as images or downloads, if developers don’t routinely need to change those in camps. You can use writeable LVM atomic filesystem snapshots of a single database to make database copies take less space and refresh more quickly.

CPU and I/O may become a limiting factor during camp rebuilds (lots of file copying and database imports), or during performance testing of a development or staging site. Having developers update camp databases or create new camps after hours can help. Or just getting better hardware usually solves the problem.

Guiding principles

Starting with the most important:

  1. Accessible
  2. Separate
  3. Automated
  4. Current
  5. Versioned
  6. Identical
  7. Easy
  8. Fast

I can summarize the many details of the previous slides with these guiding principles.

  1. Keeping camps accessible is actually the most important thing. If the business people can’t get to them, development happens in the shadows, the wrong problems get solved, other developers have no visibility, and working together is difficult.
  2. Camps need to be separate from production, and from each other. We compromise for usability’s sake and keep one developer’s camps all in the same home directory so it’s easy to get at the files, but unless the developer chooses to intermingle them, they are all completely independent.
  3. Automating camps was crucial. Anything that must be done manually is hard to make time for, and very annoying when it’s finally done. It puts burdens on those who know how things work, and if those who don’t know as well end up doing a task, they may break things for others. It doesn’t make sense to automate a one-off, but pretty much everything about camps is going to be repeated.
  4. Having current code, data, and configurations is essential. Many people think it’s fine to do development with stale data, or a subset of the data, but this always leads to trouble. Displays don’t look the same with different data. Queries don’t perform the same with less data, or very different data.
  5. Version control is wonderful. It doesn’t solve all problems, but it solves the problems it’s meant to solve very well. I don’t list it first only because in practice the higher-ranked items surprisingly turned out to be even more important -- but not sufficient!
  6. Making the various environments as nearly identical as possible is one of the main lessons here. Most compromises for time, disk space, memory, or anything else turn out to be a mistake. When in doubt, make it identical. As said in Practices of an Agile Developer: “Different makes a difference.” (#21)
  7. If it’s not easy, people will usually not do it, or will do it incorrectly. People are busy. Things need to be done under unexpected time pressure. Plan for the unexpected by automating things, and making them easy. If you feel you’re spoiling yourself or your fellow developers, you’re probably on the right track, and will soon wonder how you lived without it.
  8. If something takes too long, people will avoid doing it. Don’t live with slowness for too long without looking for ways to improve it.

Coordinating people

  1. Version control commit notifications by email, with inline diffs
  2. Use a ticket tracker
  3. Use a wiki
  4. Offer a web repository browser
  5. screen -x
  6. Phone conferences

A quick aside: These things aren’t part of camps per se, but I wouldn’t want to do development without them, so I want to mention them here.

  1. As the project has grown, developers have been unaware that someone else already committed a change for the same problem they were working on. You set up an automated commit email notification in the version control system, so everyone finds out every time a commit is made. Adding an inline diff makes it trivial for developers to review each other’s code, hit reply to comment or ask questions, and be aware of changes that are being made.
  2. It was hard to keep track of the open projects, bugs, feature enhancement requests, etc. until you set up a ticket tracker. The hardest thing was deciding which one to use! But it really didn’t matter, compared to not having one at all.
  3. Setting up a wiki was a big improvement as well. Now project documentation, people’s contact information, emergency procedures, etc. are in one location everyone knows about, and everyone can easily contribute to.
  4. Offering developers a web interface to browse the version control repository makes the previously hidden visible. Less technical people can just as easily browse the change history and see what changed when, and whether a particular change has been committed yet or not. When using distributed version control, this isn’t quite as important since each cloned repository can be browsed independently, but it’s still helpful.
  5. GNU screen is a powerful tool in general, but especially the shared screen feature is excellent for pair programming, code review, and troubleshooting.
  6. Cheap and easy phone conferencing can make a world of difference in working together with other developers. Make sure you have a VoIP or land line with a hands-free headset, that doesn’t use batteries, so you can work together comfortably for hours if needed, as if in the same room. Combined with shared screens and these other tools, phone calls can make for some very efficient collaboration.

Current camps work

  • Convert internal wiki documentation to public website tutorials and command-line help
  • Create RPM and Debian packages for easy camps system install on a fresh server
  • Unify commands into a single “camp” command à la git, svn, and similar (details on next slide)

Command unification

The current set of commands is too inconsistent:

current upcoming
(new) camp help
mkcamp camp make
rmcamp camp destroy
re --start camp start
re --stop camp stop
re camp re[start]
psql_camp camp psql
mysql_camp camp mysql
refresh-camp camp update
camp-info camp info
(web only) camp list
with-camp camp [command]
camp-lint camp lint
(new) camp test

Future directions for camps

  • Refactor core code to accommodate any number of arbitrary services, instead of hardcoding the types of webserver, application server, and database
  • Integrate one-off LVM snapshot support to current camp system
  • Support virtual machine components
  • Remove limit of 100 camps per system
  • Support Mercurial, Bazaar, and (again) CVS
  • Adapt system service SELinux policies (e.g. for Apache and PostgreSQL) for those services in camps, to provide the same constraints in development

Further information

(If viewing these slides online, note that printing and print preview will show more detailed notes alongside what’s in the slides.)