Developers need a place to work. The easiest place to do this is by editing code on the production site. When you’re working on a new site, there is no production site yet, so what you’re working in is clearly the development environment. When it’s time for that to go live, everyone’s in a hurry, things are all set up, so you switch it live. From this moment on, where do you develop? It’s of course very dangerous to keep developing right there, but it solves the immediate problems, everyone’s busy, and you make comrpomises.
Frameworks that build in support for development and testing, such as Ruby on Rails, are a huge step forward and greatly reduce this problem if their conventions are followed, but they also introduce some new problems of their own which I’ll cover in a few minutes.
Perhaps. But there’s still plenty of that around.
Modern web development frameworks provide some good conventions and defaults, and make it hard to do as badly as the HOWNOTTO describes.
However, while it may seem that the problem of development environments is solved in some modern frameworks, it’s not entirely.
Camps: A system and set of conventions to make it easy to manage many parallel development and staging environments, and keep them synchronized with the production environment.
“Camps” is just a word, not an acronym.
Camps leverage popular open source tools and build upon them (for example, version control systems).
Spree is a new open source Rails shopping cart.
I’m using it here as an example of a simple application running in camps.
See End Point Spree Development/ for more details about End Point's work on Spree.
If you’re not using a current near-identical copy of your production database, you’re developing against a toy.
Say you develop with Rails locally. You’ve probably got some Apache redirects, rewrites, subdomains set up, and they’re not reflected in the WEBrick toy webserver in development. You can’t test Apache changes there.
Are you using the same hardware (32- vs. 64-bit), operating system, libraries, web server, database, as the production system and as the other developers? Who really wants to deal with the Ubuntu/RHEL/Debian/Mac OS X differences, WEBrick vs. Apache, Postgres 8.1 vs. 8.3 version differences, setup & keeping database fresh. What about development environment accessibility, automation, backups?
Most simple sites don’t stay simple. If they’re at all successful, there are more projects, more people, more time pressures. It’s better to plan for success.
Camps require some initial investment, force some discipline, and require a little occasional maintenance. The value comes out when you’re dealing with a system that actually matters. And the payoff usually comes in saved time and sanity, more quickly than you’d expect.
It’s surprisingly important to have a “home page” for all camps, so that anyone can easily see at a glance which developers are working where, and on what projects, and can easily click through to test things out.
There’s nothing about the Ruby on Rails conventions for development that is inherently opposed to a central development server, but that’s not how people do it by default. We added support for Rails to the camps system and all the expected benefits were there. Code that was developed in private, not accessible to the business people, was now always visible. Other developers could log into an account and see code in progress, even before it’s committed to version control.
Though it isn’t supposed to ever happen, files do get changed directly in production, either by the business folks or by developers. Since you’re still pushing code from development to production with rsync, those changes simply get overwritten, leading to confusion and frustration. So you start using the version control system to pull the changes into production, and the version control system alerts you to any conflicts and allows you to commit changes made there. And helps you to educate the people who made such changes so that most occurrences of this go away.
Camps are language-agnostic, and have been used so far with:
Camps are also framework-agnostic, and have been used so far with:
They have a lot going on:
The actual camps are firewalled so as to keep search engines from indexing them and causing trouble for their production system.
Let’s look at a snapshot of the camp index from one company’s actual camp system.
Note the comments have links to RT ticket numbers in their private ticket tracking system.
A few of the 18 “developers” are actually business people or graphic designers. They’re less likely to do the version control work themselves, but they edit files in their camps and have a developer to the committing for them.
This camp index shows running camps in bold as a convenience. As we see, it’s typical for not all camps to be running at the same time, to conserve server resources when a project is back-burnered for a while.
You should expect the unexpected when it comes to firewall. A surprisingly tight egress firewall on the camps server may be only the beginning of woes. The business’s various offices may have tight egress firewalls per location that don’t allow access to your camps server at all, or to ports 9000-9199, and it will take time to get their network administrator to open this up. Some network administrators have the funny idea that “opening up 200 ports” is a very dangerous thing to do. We’ve always had it work out, but it may take some discussion and time. In the worst case you could set up a VPN between their office and the camps server.
All sorts of weird and little-known integration with services on other servers, long-forgotten scripts, cron jobs, and apps on the production and developent servers, etc., will come out as you install an existing app into camps. There’s nothing you can do about it except roll with the punches. It’s not because of camps; it’s because you’re trying to make order out of disorder.
On busy servers with many active camps, we’ve at various times run into limits of:
In short, hardware is cheaper than developers. Camps optimize for developers and business users over hardware. For camps you also will want to invest in the server, while the client machines aren’t a big concern (as long as they run an ssh client and a web browser).
Because camps have their own running daemons for each part of the application stack, they use more RAM than a system that cuts corners. RAM’s cheap enough that you should just buy more. But it may be different for the business people to think that you’ll want more RAM in your development box than in production, or the same amount at least.
Each Apache daemon uses semaphores, and the default Linux kernel needs to be configured to have more. It’s easy to do.
Each Postgres daemon uses some shared buffers, and the kernel needs to be configured to have more. It’s also easy to do.
Get lots of cheap disk. You’re going to have a complete copy of the whole application stack: docroot, app, database. To save some disk space you can use symbolic links for large docroot segments such as images or downloads, if developers don’t routinely need to change those in camps. You can use writeable LVM atomic filesystem snapshots of a single database to make database copies take less space and refresh more quickly.
CPU and I/O may become a limiting factor during camp rebuilds (lots of file copying and database imports), or during performance testing of a development or staging site. Having developers update camp databases or create new camps after hours can help. Or just getting better hardware usually solves the problem.
Starting with the most important:
I can summarize the many details of the previous slides with these guiding principles.
A quick aside: These things aren’t part of camps per se, but I wouldn’t want to do development without them, so I want to mention them here.
The current set of commands is too inconsistent:
|re --start||camp start|
|re --stop||camp stop|
|(web only)||camp list|