Let’s look at a snapshot of the camp index from one company’s actual camp system.

Note the comments have links to RT ticket numbers in their private ticket tracking system.

A few of the 18 “developers” are actually business people or graphic designers. They’re less likely to do the version control work themselves, but they edit files in their camps and have a developer to the committing for them.

This camp index shows running camps in bold as a convenience. As we see, it’s typical for not all camps to be running at the same time, to conserve server resources when a project is back-burnered for a while.

You should expect the unexpected when it comes to firewall. A surprisingly tight egress firewall on the camps server may be only the beginning of woes. The business’s various offices may have tight egress firewalls per location that don’t allow access to your camps server at all, or to ports 9000-9199, and it will take time to get their network administrator to open this up. Some network administrators have the funny idea that “opening up 200 ports” is a very dangerous thing to do. We’ve always had it work out, but it may take some discussion and time. In the worst case you could set up a VPN between their office and the camps server.

All sorts of weird and little-known integration with services on other servers, long-forgotten scripts, cron jobs, and apps on the production and developent servers, etc., will come out as you install an existing app into camps. There’s nothing you can do about it except roll with the punches. It’s not because of camps; it’s because you’re trying to make order out of disorder.

In short, hardware is cheaper than developers. Camps optimize for developers and business users over hardware. For camps you also will want to invest in the server, while the client machines aren’t a big concern (as long as they run an ssh client and a web browser).

Because camps have their own running daemons for each part of the application stack, they use more RAM than a system that cuts corners. RAM’s cheap enough that you should just buy more. But it may be different for the business people to think that you’ll want more RAM in your development box than in production, or the same amount at least.

Each Apache daemon uses semaphores, and the default Linux kernel needs to be configured to have more. It’s easy to do.

Each Postgres daemon uses some shared buffers, and the kernel needs to be configured to have more. It’s also easy to do.

Get lots of cheap disk. You’re going to have a complete copy of the whole application stack: docroot, app, database. To save some disk space you can use symbolic links for large docroot segments such as images or downloads, if developers don’t routinely need to change those in camps. You can use writeable LVM atomic filesystem snapshots of a single database to make database copies take less space and refresh more quickly.

CPU and I/O may become a limiting factor during camp rebuilds (lots of file copying and database imports), or during performance testing of a development or staging site. Having developers update camp databases or create new camps after hours can help. Or just getting better hardware usually solves the problem.

I can summarize the many details of the previous slides with these guiding principles.

1. Keeping camps accessible is actually the most important thing. If the business people can’t get to them, development happens in the shadows, the wrong problems get solved, other developers have no visibility, and working together is difficult.
2. Camps need to be separate from production, and from each other. We compromise for usability’s sake and keep one developer’s camps all in the same home directory so it’s easy to get at the files, but unless the developer chooses to intermingle them, they are all completely independent.
3. Automating camps was crucial. Anything that must be done manually is hard to make time for, and very annoying when it’s finally done. It puts burdens on those who know how things work, and if those who don’t know as well end up doing a task, they may break things for others. It doesn’t make sense to automate a one-off, but pretty much everything about camps is going to be repeated.
4. Having current code, data, and configurations is essential. Many people think it’s fine to do development with stale data, or a subset of the data, but this always leads to trouble. Displays don’t look the same with different data. Queries don’t perform the same with less data, or very different data.
5. Version control is wonderful. It doesn’t solve all problems, but it solves the problems it’s meant to solve very well. I don’t list it first only because in practice the higher-ranked items surprisingly turned out to be even more important -- but not sufficient!
6. Making the various environments as nearly identical as possible is one of the main lessons here. Most compromises for time, disk space, memory, or anything else turn out to be a mistake. When in doubt, make it identical. As said in Practices of an Agile Developer: “Different makes a difference.” (#21)
7. If it’s not easy, people will usually not do it, or will do it incorrectly. People are busy. Things need to be done under unexpected time pressure. Plan for the unexpected by automating things, and making them easy. If you feel you’re spoiling yourself or your fellow developers, you’re probably on the right track, and will soon wonder how you lived without it.
8. If something takes too long, people will avoid doing it. Don’t live with slowness for too long without looking for ways to improve it.

A quick aside: These things aren’t part of camps per se, but I wouldn’t want to do development without them, so I want to mention them here.

1. As the project has grown, developers have been unaware that someone else already committed a change for the same problem they were working on. You set up an automated commit email notification in the version control system, so everyone finds out every time a commit is made. Adding an inline diff makes it trivial for developers to review each other’s code, hit reply to comment or ask questions, and be aware of changes that are being made.
2. It was hard to keep track of the open projects, bugs, feature enhancement requests, etc. until you set up a ticket tracker. The hardest thing was deciding which one to use! But it really didn’t matter, compared to not having one at all.
3. Setting up a wiki was a big improvement as well. Now project documentation, people’s contact information, emergency procedures, etc. are in one location everyone knows about, and everyone can easily contribute to.
4. Offering developers a web interface to browse the version control repository makes the previously hidden visible. Less technical people can just as easily browse the change history and see what changed when, and whether a particular change has been committed yet or not. When using distributed version control, this isn’t quite as important since each cloned repository can be browsed independently, but it’s still helpful.
5. GNU screen is a powerful tool in general, but especially the shared screen feature is excellent for pair programming, code review, and troubleshooting.
6. Cheap and easy phone conferencing can make a world of difference in working together with other developers. Make sure you have a VoIP or land line with a hands-free headset, that doesn’t use batteries, so you can work together comfortably for hours if needed, as if in the same room. Combined with shared screens and these other tools, phone calls can make for some very efficient collaboration.