DevCamps Documentation: Master

NAME

Camp::Master - library routines for management of camps

VERSION

3.06

CAMP MASTER DATABASE CONFIGURATION

The camp system relies on a single master database to hold the known camp users, camp types, and existing camps. All camps within the system are registered within this database, along with the type of camp, the version control system used for that camp, the owner, etc. The camp system will not function without this database.

Connectivity info for this database must be provided to the camp system. The camp master database configuration file, camp-db-config, serves this purpose.

It is expected to live at $camp_base/camp-db-config.

Entries consist of the form $key:$value, one per line.

The following key/value pairs are expected to be provided by this file:

dsn

The DBI DSN for the master camp database. See the DBI docs for DBI well-formedness; it depends in no small part on the database driver used (currently, only Pg and mysql drivers are supported).

user

The username to use when connecting to the master camp database.

password

The password to use when connecting to the master camp database. If your database is configured to allow access without a password, then you may leave this option out; however, that is typically not recommended, since the connection attempts will happen from different UNIX user accounts.

Therefore, a typical camp-db-config file might look like:

 dsn:dbi:Pg:dbname=camps
 user:camp
 password:fuggedabowwditt
COPY PATHS CONFIGURATION

Your camp type may specify arbitrary paths that should be copied/symlinked into camp instances via the copy-paths.yml file. As the extension suggests, this is expected to be a YAML file.

The structure of the file is as follows:

  • Each path to copy (file or directory, it matters not) gets a "document" within the YAML structure. This essentially means that you start each path's entry with the standard "---" YAML document header. The result of this is that Camp::Master sees the path entries as an array.
  • Each path entry is a hash. Each path hash must provide the following key/value pairs:
    source

    The source path to copy from. This can be relative to the camp type path, or absolute.

    target

    The target path to copy to. This is always relative to the camp instance base.

    In addition, the following optional items may be provided:

    default_link

    If provided with a Perly true value (non-blank, non-zero), the target will be created as a symlink to source rather than a full copy. This is sensible for large docroots, image directories, etc.

    If left unspecified for a given path, this effectively defaults to false.

    always

    If provided with a Perly true value (non-blank, non-zero), the default behavior determined by default_link is always used and the camp creator isn't given a choice in the matter.

    exclude

    May be provided with a single scalar entry or an array of such entries; each should correspond to an rsync/tar-style "--exclude" pattern, which will be provided to the rsync call that performs the copy. This naturally only comes into play if copying the source rather than symlinking.

    parse_source

    If provided with a Perly true value, the source path will be parsed for camp configuration tokens (see CAMP CONFIGURATION VARIABLES). If not specified, parsing does not occur.

    parse_target

    If provided with a Perly true value, the target path will be parsed for camp configuration tokens (see CAMP CONFIGURATION VARIABLES). If not specified, parsing does not occur.

    allow_errors

    If provided with a Perly true value, rsync errors will be ignored. Otherwise, rsync errors are fatal and abort the mkcamp process.

    remote_source

    If provided will assume that the source location is on a remote server. Value should be the protocol used to connect to the remote location, 'ssh' is the only valid value at the moment.

    Because the YAML stream is converted to an array, the paths are processed in order of specification within the stream. Therefore, you can have dependencies between paths and they'll work out as long as you arrange them in the proper order in the file.

    When a new camp instance is created, each path is processed in order. Per path, the camp creator is offered the opportunity to specify whether the camp should receive a full copy or a symlink of the source path in question; the default response will be noted based on the default_link setting for that path. However, if the always option is set for the path, the user is not given the choice and the default simple happens automatically.

    Here's an example of a file:

     # The /var/www/html docroot will be symlinked to $camp/htdocs by default,
     # but the user is given the choice to override with a copy.
     ---
     source: /var/www/html
     target: htdocs
     default_link: 1
     # The /var/cgi-bin/foo.com script directory will by copied (since no default_link
     # is specified) by default to $camp/cgi-bin.  The user is given the choice to
     # symlink instead.
     ---
     source: /var/cgi-bin/foo.com
     target: cgi-bin
     # The /var/www/image_repository is symlinked to $camp/image_repository; the user
     # is not given a choice about this, because always is set.
     ---
     source: /var/www/image_repository
     target: image_repository
     default_link: 1
     always: 1
     ---
     source: remote@endpoint.com:/var/www/image_repository
     target: image_repository
     default_link: 1
     always: 1
     allow_errors: 1
     remote_source: ssh
CAMP CONFIGURATION VARIABLES

Camp configuration occurs at two levels:

Base camp level

Configuration information provided at the $camp_base trickles down to all camp types underneath that base.

Camp type level

Each camp type has a subdirectory under the $camp_base, and configuration information provided at this level applies only to the camp type in question, overriding any conflicting settings coming in from the base camp level.

The configuration information is parsed at the base camp level first, then the type-specific level second.

In each level, the local-config file is processed. This is expected to consist of key/value pairs of the form:

 some_key:some_value

Empty lines and lines starting with the pound character are ignored, meaning that you can embed comments in your configuration files.

All values specified in these files undergo token substitution, such that any configuration variable can be substituted into a value by using the token:

 __CAMP_VARIABLENAME__

For instance, if the local-config file has a line:

 my_variable:some_useless_value

then a subsequent line:

 other_variable:__CAMP_MY_VARIABLE__foo

will result in the "my_variable" key having value "some_useless_valuefoo".

These variables are used both by Camp::Master itself, and in the various templates that Camp::Master renders at camp creation time; the token substitution logic is the same in each case. This allows your template files to undergo rendering and have camp-specific information substituted in, localing the file to the camp being created (which is the entire point of the camp system).

Prior to parsing the base level and camp level configuration files, a number of configuration entries are determined automatically by Camp::Master.

base_path

The $camp_base (defaults to /home/camp)

type_path

The path to the camp type's directory under $camp_base.

type

The type of camp.

root

The camp owner's home directory path.

path

The local path of the camp being operated upon (for instance, /home/some_user/camp16)

number

The camp number

name

The camp's name (for instance, "camp23")

docroot

The Apache docroot for the camp.

http_port

The custom port the camp will use for HTTP traffic.

https_port

The custom port the camp will use for HTTPS traffic.

httpd_path

The camp-specific directory where Apache configuration and log files will go.

httpd_lib_path

The path to the Apache modules that must be referenced when launching camp Apache.

Defaults to /usr/lib/httpd.

httpd_cmd_path

The path to the Apache executable for controlling the Apache server.

Defaults to /usr/sbin/httpd.

httpd_define

A space-separated list of parameter names that should be defined for Apache at startup time (via -D on the command line). These can be tested for in Apache configuration with the IfDefine parameter. See http://httpd.apache.org/docs/2.2/mod/core.html#ifdefine for details.

httpd_specify_conf

Specify location of configuration file under "httpd_path" and include that in command strings.

For example, "conf/apache2.conf", used when executable has been compiled with a specific full path. To detect need for this option see httpd -V output and check for "SERVER_CONFIG_FILE" with a value similar to "/etc/apache2/apache2.conf" (any path beginning with "/").

skip_apache

A boolean (0/1) setting that determines whether to skip Apache-specific httpd server setup, for example when using nginx.

httpd_start

When setting skip_apache, set this to the command to start your webserver. An example with nginx:

 /usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
httpd_stop

When setting skip_apache, set this to the command to stop your webserver. An example with nginx:

 pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill $pid
httpd_restart

pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill -HUP $pid || /usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf

skip_ssl_cert_gen

A boolean (0/1) setting that determines whether to skip generation of a self-signed SSL certificate for the HTTP server.

Default is 0 (false).

icroot (Interchange only)

The main interchange directory within the camp.

cgidir (Interchange only)

The path where the interchange linker programs should reside; defaults to cgi-bin under the camp's path.

catroot (Interchange only)

The main path for your Interchange catalog; requires that the catalog variable be provided in your configuration file.

Defaults to path + '/catalogs/' + catalog.

ic_socket (Interchange only)

The path where the Interchange socket file will reside

Defaults to icroot + '/var/run/socket'

railsdir (Rails only)

The path where rails will live within the camp; defaults to "rails" under the camp's path.

mongrel_base_port (Rails only)

The lowest port to be used by the Mongrel cluster for balancing Rails children

proxy_name (Rails only)

The ProxyBalancer name used for the Mongrel listeners within the camp Apache.

proxy_balance_members (Rails only)

Rendered Apache ProxyBalancer member configuration directives suitable for placement directly within a virtualhost's <Proxy balancer://...>...</Proxy> container. Defaults to using three mongrel listeners with ports incremented by one starting at the mongrel_base_port.

app_init

When using an application server other than those natively supported in camps, set this to the command to initialize your application server. This will run only once during camp creation. For example (using a custom init script):

 __CAMP_PATH__/bin/init-magento
app_start

When using an application server other than those natively supported in camps, set this to the command to start your application server. For example (using a custom startup script):

 __CAMP_PATH__/bin/start-unicorn
app_stop

Command to stop your application server. An example for Ruby's Unicorn:

 pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid
app_restart

Command to restart your application server. An example for Ruby's Unicorn:

 pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill -HUP $pid || __CAMP_PATH__/bin/start-unicorn
post_mkcamp_command

Command to be run after the mkcamp process is completed.

db_type

The type of database server used for the camp; will be either 'pg' for Postgres or 'mysql' for MySQL.

The database type is determined by the subdirectories within the camp type path; the camp type path must have either a pgsql directory or a mysql directory (for pg or mysql, respectively).

db_path

The main database path for the camp's database server; dependent on the database type, will be either 'pgsql' or 'mysql' for Postgres and MySQL respectively.

db_host

The database hostname; defaults to 'localhost'.

db_port

The custom port to use for your camp's database server. Defaults to 8900 + camp number.

db_encoding

The encoding to use when initializing the database cluster. Defaults to UTF-8.

db_data

The directory where the camp's database is expected to store binary data. Defaults to db_path + '/data'.

db_tmpdir

A temporary directory for varying, ephemeral data like logs, pids, etc., specific for the camp's database server. Defaults to db_path + '/tmp'.

db_log

The base database server logfile path; for Postgres, this is in fact the logfile to be used; MySQL uses multiple logging paths ordinarily, so it can be used as a prefix for a full path with MySQL (it probably could be used for all logging too).

Defaults:

Pg

db_tmpdir + 'postgresql.log'

MySQL

db_tmpdir + 'mysql.log'

db_conf

The database server configuration file within the camp.

Defaults:

Pg

db_data + 'postgresql.conf'

MySQL

db_path + 'my.cnf'

db_socket (MySQL only)

Path for the camp's database server UNIX socket. Defaults to db_tmpdir + 'mysql.$camp_number.sock'

All of the above variables are calculated for you. This calculation is one of the first things performed by Camp::Master; therefore, they can be safely overridden in your base or type configuration files as appropriate for your camp deployment.

A few variables must be provided by your configuration files in order for the camp system to function properly:

db_source_scripts

Space-separated list of paths to SQL files that should be run upon preparing a camp's database server. You may specify as many as you need, but at least one should be provided.

The paths may be absolute or relative; if relative, they are expected to be relative to the camp type's directory.

Each script specified in the space-separated list is run within its own db client shell; they are executed in order of specification. No assumptions are made for transaction handling; if you want transactions, put them in the scripts themselves.

It's important to know the context in which these run, which differs between database server types due to fundamental structural differences between those servers and how they organize things:

Postgres

In Postgres, each script is run via a connection to the "postgres" user's "postgres" database. Therefore, your scripts need to provide \connect information in order to make changes against a different database.

MySQL

In MySQL, the scripts connect using the db_default_database and db_default_user variables, which effectively makes these variables required for MySQL deployments.

db_source_scripts_parse

Scripts listed in db_source_scripts are normally processed verbatim. List again here any scripts here that you would like to first have preprocessed with variable token substitution. This is useful if, for example, you would like to reference the camp number or camp owner's name in the script.

The paths should be specified identically to how they're listed in db_source_scripts or they will not be tokenized.

db_mysql_scripts (MySQL only)

Space-separated list of paths to SQL files that should be run upon preparing a camp's MySQL database server, prior to processing the roles for that database; this gives an opportunity to initialize the database to a known set of users/permissions, for instance, or initalize things in other ways needed prior to role creation.

db_sleep_time

Allows specification of a number of seconds to sleep between starting the database server at camp creation time and processing the roles and db_source_scripts; necessary for UNIX socket connections in which the server needs a few seconds to start up and initialize its socket file.

Sane values appear to be in the 5-15 second range. Defaults to 5 seconds; set to 0 or blank to turn off.

db_dbnames

Space-separated list of database names that are expected to live within your camp's database server.

This was formerly used to set up Postgres password access to each database in your user account's .pgpass file, but now each database user's password is applied to all databases ("*") there. It is still up to you to configure Postgres to allow appropriate access.

camp_subdirectories

A space-separated list of directories that are expected to reside under your camp (naturally, these paths are expected to be relative to the camp directory itself). These directories will be cleared out if you rebuild an existing camp (with the mkcamp script), which is the main reason they need to be provided to Camp::Master.

catalog_linker_filenames (Interchange only)

Space-separated list of filenames that need to be put in place within your camp as Apache/Interchange linker programs. Specify at least one for your main catalog.

catalog (Interchange only)

The name of the catalog for your camp's Interchange deployment. Specify one, even if you have multiple catalogs; the catroot variable is set based on this. For the majority of End Point Interchange deployments, multiple catalogs are a non-issue.

Some optional variables may also be provided to affect the functioning of your camp deployment:

db_locale (postgres only)

The locale to use when initializing the database cluster. There is no default, meaning that the cluster would by default initialize to whatever locale Postgres itself uses as the default.

db_default_database

The default database to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the database used for db_source_scripts import on MySQL.

db_default_user

The default user to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the user used for db_source_scripts import on MySQL.

db_pg_ssl_active (postgres only)

Boolean used to indicate that Postgres instance has SSL activated and therefore a certificate should be generated. Default: off.

db_pg_ssl_C (postgres only)

The country to use for your camp's Postgres SSL certificate (defaults to value of ssl_C).

db_pg_ssl_ST (postgres only)

The statename to use for your camp's Postgres SSL certificate (defaults to value of ssl_ST).

db_pg_ssl_L (postgres only)

The locality to use for your camp's Postgres SSL certificate (defaults to value of ssl_L).

db_pg_ssl_O (postgres only)

The organization name to use for your camp's Postgres SSL certificate (defaults to value of ssl_O).

ssl_C

The country to use for your camp's SSL certificate (defaults to US).

ssl_ST

The statename to use for your camp's SSL certificate (defaults to 'New York').

ssl_L

The locality to use for your camp's SSL certificate (defaults to 'New York').

ssl_O

The organization name to use for your camp's SSL certificate (defaults to 'End Point Corporation').

vcs_default_type

Specifies the default version control system to use for the camp type. This is optional, but is certainly highly convenient for camp users and makes things easier to manage. The main reason to use a different version control system would be for using git-svn in a camp against a Subversion repository.

If this is set for a camp type, then it is not necessary for a user to specify their VCS type when creating an instance of that camp. The value of vcs_default_type has no bearing on existing camps; each extant camp's original VCS type is recorded in the camp database and will not change without heroic efforts on the part of the camp's owner.

repo_path

Use this to specify the SCM repository path. This is not specific to VCS type and has been preserved for legacy purposes. SVN and Git repository paths can be specified in type-specific manner using repo_path_svn and repo_path_git, respectively; when those are used, they override the repo_path value.

There is no default value for repo_path, so don't count on it being present in your code unless you are quite sure of what you're doing.

repo_path_svn

Use this to specify the main project Subversion repository path; is is assumed that it will be accessed via the "file://" protocol, meaning that you need only provide the path to the repository (and any subdirectories within that repository).

When determining the Subversion repository location, Camp::Master consults, in this order:

  • repo_path_svn
  • repo_path
  • Default: type_path . '/svnrepo/trunk'

There is no default value for repo_path_svn, however, so do not count on this being available for token substitution unless you explicitly set it in the relevant local-config. The default repository location described above is enforced through logic, but not within the master configuration hash.

repo_path_git

Use this to specify the main project Git repository path; no assumptions about the path/URL are made, so you are free to specify a file path (which is most likely as of this writing), a git://* URL, an rsync://* URL, etc.

Similar to determination of Subversion repository location, Camp::Master consults the following items in order when finding a camp type's Git repository location:

  • repo_path_git
  • repo_path
  • Default: type_path . '/repo.git'

There is no default value for repo_path_git, so you cannot count on its presence unless you know what you're doing and are working with a camp type that definitely specifies a value for it.

git_clone_options

Options to pass literally to the "git clone ..." call when building a new camp and cloning from the source Git repo. See the "git-clone" man pages for valid options. There is no default, meaning that the effective default behavior is invoking git-clone without options.

Git is efficient in its use of disk space, but by using the various options like local, shared, or reference can potentially be even more efficient and speedy.

git_branch

Specifies a branch to track and checkout after cloning. Use this in the case that your Git repository's active branch differs from the branch the camp type needs to work with. This would likely be the case if you're using a single master Git repository to track several lines of development; one camp type would work with the default branch "master", while other camp types would need to specify alternate branches.

This assumes that you're using a remote name of "origin". Since git_clone_options allows you to specify a different name for your clone, git_branch introduces some extra semantics: if the value of git_branch contains a forward slash ("/"), the word characters preceding the first slash are treated as the remote name.

If you use git_clone_options "-o foo" to specify a remote name of "foo" rather than "origin", you can have your camp type check out branch "bar" via git_branch of "foo/bar". However, if you're using the default clone remote name of "origin", then git_branch "bar" would suffice to checkout branch "bar".

This also means that if the desired branch has forward slashes in the name, you'll need to include the remote name as the first portion of your git_branch value. So, if you want a branch named "releases/1.1", you would need a git_branch value of "origin/releases/1.1".

TEMPLATE FILE OPERATIONS

Rendering and installing arbitrary files into a camp from templates is a critical aspect of the camp system. Configuration files for Apache, Interchange, Rails, etc. should be reduced to templates, with things like hostnames, file paths, port numbers, etc. replaced with camp-system tokens of the form described in the section regarding CONFIGURATION VARIABLES. At camp creation time, Camp::Master will parse each template, performing token substitution, and install each parsed template into the appropriate location within the new camp.

The files to parse are specified within the camp type subdirectory's camp-config-files file. Specify one path per line; blanks and lines starting with the pound character are ignored.

All the files specified will undergo the described token substitution and installation into the new camp. The paths specified in camp-config-files are expected to be relative to the camp type directory's 'etc/' subdirectory, and also reflect the target path of the file when copied into the new camp itself. Thus, a file at /home/camp/some_type/etc/blah/foo.conf would be registered in camp-config-files with a path relative to etc/, or "blah/foo.conf", and would be installed at /home/some_user/campNN/blah/foo.conf after parsing.

Also, some assumptions are made about what files will be included if your camp uses Interchange versus Rails:

  • If your camp-config-files file specifies interchange/bin/interchange, then it is assumed to use Interchange.
  • If your camp-config-files file specifies rails/.../config/mongrel_cluster.yml, then it is assumed to use Rails.

Note that these are not mutually exclusive. It is theoretically acceptable for a single camp to employ both app server types. If you are bursting with curiosity and sadly deficient in sanity, by all means try this out.

While this can be expected to vary between deployments, a number of files are obvious candidates for inclusion within this templating scheme:

  • rails/.../config/database.yml
  • interchange/interchange_local.cfg
  • catalogs/.../catalog_local.cfg
  • httpd/conf/sites/some_site.conf

Anything that relies on file paths, ports, domain names, etc. should be abstracted out into such templates. A common pattern (implied by interchange_local.cfg and catalog_local.cfg above) is to encapsulate such details within "local" configuration files that are included from master configuration files; the master configuration files can stay in version control and have no need to vary between camps, production, etc., while the "local" configuration files are small, containing only what is absolutely necessary, and vary in a controlled way between environments by virtue of being managed by the camp system.

VERSION CONTROL SYSTEM OPTIONS

The camp system provides three different version control choices:

  • Subversion ('svn')
  • Git ('git')

Typically speaking, use of Git for a particular camp type implies non-use of Subversion, and the opposite also holds. However, there is no reason that all types cannot be made available for use within the same camp type; the Git software suite comes with '''git-svn''', which can be used to track a Subversion repository via a Git repository. Therefore, all options can be available for a camp type provided the administrator takes the trouble to arrange it. That arrangement takes place largely outside of the camp system itself, and would instead involve post-commit hooks within one or both repositories to ensure that a change in one repository is pushed to the other.

Camp::Master ultimately makes no assumptions about your camp type's VCS offerings; if a camp user attempts to make a new camp using some version control system, Camp::Master will attempt to do it, and if the requested version control system hasn't been configured for the camp type, an error will result. Configuration options to tell Camp::Master about your VCS setup are fairly straightforward; see the configuration variable information for:

  • default_vcs_type
  • repo_path
  • repo_path_git
  • repo_path_svn

While no assumptions are made, the strong recommendation is to use Git. Subversion is slow, inefficient in its use of disk space, relatively inflexible, bad at branching, etc.

GIT USE

When creating a new camp using Git as the VCS type, Camp::Master assumes an SVN-like flow of operations; the camp type specifies where the Git repository lives, and the new camp will result in a new repository that tracks that central repository. Consequently, the operations used by Camp::Master tend to be clone, pull, and push operations, for remote repository use. When you're working within your Git-based camp, you of course need to keep this in mind, knowing that a checkout or a commit are only relative to the repository of that camp, and that you ultimately need a push to commit your stuff to the central camp repository.

At this time, Camp::Master doesn't provide any tools for creating a Git camp using an alternate repository as the repository to track; this is something that can be done using git directly, after creating a new camp. Create the camp with Git as the VCS choice, then use git directly within that camp to point the camp at an alternate remote repository.

BUGS AND LIMITATIONS

This module has tests. It also probably has bugs. It is, after all, software.

AUTHOR

Ethan Rowe and other contributors

LICENSE AND COPYRIGHT

Copyright (C) 2006-2016 End Point Corporation, https://www.endpoint.com/

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see: http://www.gnu.org/licenses/

Commands:

Configuration, API, and Internals:

© 2006–2017 End Point Corporation