Camp::Master - library routines for management of camps
VERSION3.06
CAMP MASTER DATABASE CONFIGURATIONThe camp system relies on a single master database to hold the known camp users, camp types, and existing camps. All camps within the system are registered within this database, along with the type of camp, the version control system used for that camp, the owner, etc. The camp system will not function without this database.
Connectivity info for this database must be provided to the camp system. The camp master database configuration file, camp-db-config, serves this purpose.
It is expected to live at $camp_base/camp-db-config.
Entries consist of the form $key:$value, one per line.
The following key/value pairs are expected to be provided by this file:
The DBI DSN for the master camp database. See the DBI docs for DBI well-formedness; it depends in no small part on the database driver used (currently, only Pg and mysql drivers are supported).
The username to use when connecting to the master camp database.
The password to use when connecting to the master camp database. If your database is configured to allow access without a password, then you may leave this option out; however, that is typically not recommended, since the connection attempts will happen from different UNIX user accounts.
Therefore, a typical camp-db-config file might look like:
dsn:dbi:Pg:dbname=camps user:camp password:fuggedabowwdittCOPY PATHS CONFIGURATION
Your camp type may specify arbitrary paths that should be copied/symlinked into camp instances via the copy-paths.yml file. As the extension suggests, this is expected to be a YAML file.
The structure of the file is as follows:
The source path to copy from. This can be relative to the camp type path, or absolute.
The target path to copy to. This is always relative to the camp instance base.
In addition, the following optional items may be provided:
If provided with a Perly true value (non-blank, non-zero), the target will be created as a symlink to source rather than a full copy. This is sensible for large docroots, image directories, etc.
If left unspecified for a given path, this effectively defaults to false.
If provided with a Perly true value (non-blank, non-zero), the default behavior determined by default_link is always used and the camp creator isn't given a choice in the matter.
May be provided with a single scalar entry or an array of such entries; each should correspond to an rsync/tar-style "--exclude" pattern, which will be provided to the rsync call that performs the copy. This naturally only comes into play if copying the source rather than symlinking.
If provided with a Perly true value, the source path will be parsed for camp configuration tokens (see CAMP CONFIGURATION VARIABLES). If not specified, parsing does not occur.
If provided with a Perly true value, the target path will be parsed for camp configuration tokens (see CAMP CONFIGURATION VARIABLES). If not specified, parsing does not occur.
If provided with a Perly true value, rsync errors will be ignored. Otherwise, rsync errors are fatal and abort the mkcamp process.
If provided will assume that the source location is on a remote server. Value should be the protocol used to connect to the remote location, 'ssh' is the only valid value at the moment.
Because the YAML stream is converted to an array, the paths are processed in order of specification within the stream. Therefore, you can have dependencies between paths and they'll work out as long as you arrange them in the proper order in the file.
When a new camp instance is created, each path is processed in order. Per path, the camp creator is offered the opportunity to specify whether the camp should receive a full copy or a symlink of the source path in question; the default response will be noted based on the default_link setting for that path. However, if the always option is set for the path, the user is not given the choice and the default simple happens automatically.
Here's an example of a file:
# The /var/www/html docroot will be symlinked to $camp/htdocs by default, # but the user is given the choice to override with a copy. --- source: /var/www/html target: htdocs default_link: 1 # The /var/cgi-bin/foo.com script directory will by copied (since no default_link # is specified) by default to $camp/cgi-bin. The user is given the choice to # symlink instead. --- source: /var/cgi-bin/foo.com target: cgi-bin # The /var/www/image_repository is symlinked to $camp/image_repository; the user # is not given a choice about this, because always is set. --- source: /var/www/image_repository target: image_repository default_link: 1 always: 1 --- source: remote@endpoint.com:/var/www/image_repository target: image_repository default_link: 1 always: 1 allow_errors: 1 remote_source: ssh
Camp configuration occurs at two levels:
Configuration information provided at the $camp_base trickles down to all camp types underneath that base.
Each camp type has a subdirectory under the $camp_base, and configuration information provided at this level applies only to the camp type in question, overriding any conflicting settings coming in from the base camp level.
The configuration information is parsed at the base camp level first, then the type-specific level second.
In each level, the local-config file is processed. This is expected to consist of key/value pairs of the form:
some_key:some_value
Empty lines and lines starting with the pound character are ignored, meaning that you can embed comments in your configuration files.
All values specified in these files undergo token substitution, such that any configuration variable can be substituted into a value by using the token:
__CAMP_VARIABLENAME__
For instance, if the local-config file has a line:
my_variable:some_useless_value
then a subsequent line:
other_variable:__CAMP_MY_VARIABLE__foo
will result in the "my_variable" key having value "some_useless_valuefoo".
These variables are used both by Camp::Master itself, and in the various templates that Camp::Master renders at camp creation time; the token substitution logic is the same in each case. This allows your template files to undergo rendering and have camp-specific information substituted in, localing the file to the camp being created (which is the entire point of the camp system).
Prior to parsing the base level and camp level configuration files, a number of configuration entries are determined automatically by Camp::Master.
The $camp_base (defaults to /home/camp)
The path to the camp type's directory under $camp_base.
The type of camp.
The camp owner's home directory path.
The local path of the camp being operated upon (for instance, /home/some_user/camp16)
The camp number
The camp's name (for instance, "camp23")
The Apache docroot for the camp.
The custom port the camp will use for HTTP traffic.
The custom port the camp will use for HTTPS traffic.
The camp-specific directory where Apache configuration and log files will go.
The path to the Apache modules that must be referenced when launching camp Apache.
Defaults to /usr/lib/httpd.
The path to the Apache executable for controlling the Apache server.
Defaults to /usr/sbin/httpd.
A space-separated list of parameter names that should be defined for Apache at startup time (via -D on the command line). These can be tested for in Apache configuration with the IfDefine parameter. See http://httpd.apache.org/docs/2.2/mod/core.html#ifdefine for details.
Specify location of configuration file under "httpd_path" and include that in command strings.
For example, "conf/apache2.conf", used when executable has been compiled with a specific full path. To detect need for this option see httpd -V output and check for "SERVER_CONFIG_FILE" with a value similar to "/etc/apache2/apache2.conf" (any path beginning with "/").
A boolean (0/1) setting that determines whether to skip Apache-specific httpd server setup, for example when using nginx.
When setting skip_apache, set this to the command to start your webserver. An example with nginx:
/usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
When setting skip_apache, set this to the command to stop your webserver. An example with nginx:
pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill $pid
pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill -HUP $pid || /usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
A boolean (0/1) setting that determines whether to skip generation of a self-signed SSL certificate for the HTTP server.
Default is 0 (false).
The main interchange directory within the camp.
The path where the interchange linker programs should reside; defaults to cgi-bin under the camp's path.
The main path for your Interchange catalog; requires that the catalog variable be provided in your configuration file.
Defaults to path + '/catalogs/' + catalog.
The path where the Interchange socket file will reside
Defaults to icroot + '/var/run/socket'
The path where rails will live within the camp; defaults to "rails" under the camp's path.
The lowest port to be used by the Mongrel cluster for balancing Rails children
The ProxyBalancer name used for the Mongrel listeners within the camp Apache.
Rendered Apache ProxyBalancer member configuration directives suitable for placement directly within a virtualhost's <Proxy balancer://...>...</Proxy> container. Defaults to using three mongrel listeners with ports incremented by one starting at the mongrel_base_port.
When using an application server other than those natively supported in camps, set this to the command to initialize your application server. This will run only once during camp creation. For example (using a custom init script):
__CAMP_PATH__/bin/init-magento
When using an application server other than those natively supported in camps, set this to the command to start your application server. For example (using a custom startup script):
__CAMP_PATH__/bin/start-unicorn
Command to stop your application server. An example for Ruby's Unicorn:
pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid
Command to restart your application server. An example for Ruby's Unicorn:
pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill -HUP $pid || __CAMP_PATH__/bin/start-unicorn
Command to be run after the mkcamp process is completed.
The type of database server used for the camp; will be either 'pg' for Postgres or 'mysql' for MySQL.
The database type is determined by the subdirectories within the camp type path; the camp type path must have either a pgsql directory or a mysql directory (for pg or mysql, respectively).
The main database path for the camp's database server; dependent on the database type, will be either 'pgsql' or 'mysql' for Postgres and MySQL respectively.
The database hostname; defaults to 'localhost'.
The custom port to use for your camp's database server. Defaults to 8900 + camp number.
The encoding to use when initializing the database cluster. Defaults to UTF-8.
The directory where the camp's database is expected to store binary data. Defaults to db_path + '/data'.
A temporary directory for varying, ephemeral data like logs, pids, etc., specific for the camp's database server. Defaults to db_path + '/tmp'.
The base database server logfile path; for Postgres, this is in fact the logfile to be used; MySQL uses multiple logging paths ordinarily, so it can be used as a prefix for a full path with MySQL (it probably could be used for all logging too).
Defaults:
The database server configuration file within the camp.
Defaults:
Path for the camp's database server UNIX socket. Defaults to db_tmpdir + 'mysql.$camp_number.sock'
All of the above variables are calculated for you. This calculation is one of the first things performed by Camp::Master; therefore, they can be safely overridden in your base or type configuration files as appropriate for your camp deployment.
A few variables must be provided by your configuration files in order for the camp system to function properly:
Space-separated list of paths to SQL files that should be run upon preparing a camp's database server. You may specify as many as you need, but at least one should be provided.
The paths may be absolute or relative; if relative, they are expected to be relative to the camp type's directory.
Each script specified in the space-separated list is run within its own db client shell; they are executed in order of specification. No assumptions are made for transaction handling; if you want transactions, put them in the scripts themselves.
It's important to know the context in which these run, which differs between database server types due to fundamental structural differences between those servers and how they organize things:
In Postgres, each script is run via a connection to the "postgres" user's "postgres" database. Therefore, your scripts need to provide \connect information in order to make changes against a different database.
In MySQL, the scripts connect using the db_default_database and db_default_user variables, which effectively makes these variables required for MySQL deployments.
Scripts listed in db_source_scripts are normally processed verbatim. List again here any scripts here that you would like to first have preprocessed with variable token substitution. This is useful if, for example, you would like to reference the camp number or camp owner's name in the script.
The paths should be specified identically to how they're listed in db_source_scripts or they will not be tokenized.
Space-separated list of paths to SQL files that should be run upon preparing a camp's MySQL database server, prior to processing the roles for that database; this gives an opportunity to initialize the database to a known set of users/permissions, for instance, or initalize things in other ways needed prior to role creation.
Sometimes when working in a camp a developer has SQL scripts that are in-progress and are needed when a database is refreshed. These SQL files should be placed in a standard directory in each camp and db_working_files_dir should be the path to this dir relative to the camp's path. Files in this dir that end in '.sql' will be executed in the camp's database in lexical order when the database is created or refreshed.
Allows specification of a number of seconds to sleep between starting the database server at camp creation time and processing the roles and db_source_scripts; necessary for UNIX socket connections in which the server needs a few seconds to start up and initialize its socket file.
Sane values appear to be in the 5-15 second range. Defaults to 5 seconds; set to 0 or blank to turn off.
Space-separated list of database names that are expected to live within your camp's database server.
This was formerly used to set up Postgres password access to each database in your user account's .pgpass file, but now each database user's password is applied to all databases ("*") there. It is still up to you to configure Postgres to allow appropriate access.
If using LVM snapshots for cloning databases for each camp, then set this to 'yes'.
When using LVM snapshots for databases, you can specifiy the initial snapshot size. Defaults to 3G. When running the command `camp-lvm resize` or `camp-lvm resize_all`, camp snapshots may also be increased by this same size if their usage exceeds 50%.
The LVM volume name to clone for snapshots. For example, lvm_origin_volume:vg0/CampOriginDB.
A space-separated list of directories that are expected to reside under your camp (naturally, these paths are expected to be relative to the camp directory itself). These directories will be cleared out if you rebuild an existing camp (with the mkcamp script), which is the main reason they need to be provided to Camp::Master.
Space-separated list of filenames that need to be put in place within your camp as Apache/Interchange linker programs. Specify at least one for your main catalog.
The name of the catalog for your camp's Interchange deployment. Specify one, even if you have multiple catalogs; the catroot variable is set based on this. For the majority of End Point Interchange deployments, multiple catalogs are a non-issue.
Some optional variables may also be provided to affect the functioning of your camp deployment:
The locale to use when initializing the database cluster. There is no default, meaning that the cluster would by default initialize to whatever locale Postgres itself uses as the default.
The default database to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the database used for db_source_scripts import on MySQL.
The default user to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the user used for db_source_scripts import on MySQL.
Boolean used to indicate that Postgres instance has SSL activated and therefore a certificate should be generated. Default: off.
The country to use for your camp's Postgres SSL certificate (defaults to value of ssl_C).
The statename to use for your camp's Postgres SSL certificate (defaults to value of ssl_ST).
The locality to use for your camp's Postgres SSL certificate (defaults to value of ssl_L).
The organization name to use for your camp's Postgres SSL certificate (defaults to value of ssl_O).
The country to use for your camp's SSL certificate (defaults to US).
The statename to use for your camp's SSL certificate (defaults to 'New York').
The locality to use for your camp's SSL certificate (defaults to 'New York').
The organization name to use for your camp's SSL certificate (defaults to 'End Point Corporation').
Specifies the default version control system to use for the camp type. This is optional, but is certainly highly convenient for camp users and makes things easier to manage. The main reason to use a different version control system would be for using git-svn in a camp against a Subversion repository.
If this is set for a camp type, then it is not necessary for a user to specify their VCS type when creating an instance of that camp. The value of vcs_default_type has no bearing on existing camps; each extant camp's original VCS type is recorded in the camp database and will not change without heroic efforts on the part of the camp's owner.
Use this to specify the SCM repository path. This is not specific to VCS type and has been preserved for legacy purposes. SVN and Git repository paths can be specified in type-specific manner using repo_path_svn and repo_path_git, respectively; when those are used, they override the repo_path value.
There is no default value for repo_path, so don't count on it being present in your code unless you are quite sure of what you're doing.
Use this to specify the main project Subversion repository path; is is assumed that it will be accessed via the "file://" protocol, meaning that you need only provide the path to the repository (and any subdirectories within that repository).
When determining the Subversion repository location, Camp::Master consults, in this order:
There is no default value for repo_path_svn, however, so do not count on this being available for token substitution unless you explicitly set it in the relevant local-config. The default repository location described above is enforced through logic, but not within the master configuration hash.
Use this to specify the main project Git repository path; no assumptions about the path/URL are made, so you are free to specify a file path (which is most likely as of this writing), a git://* URL, an rsync://* URL, etc.
Similar to determination of Subversion repository location, Camp::Master consults the following items in order when finding a camp type's Git repository location:
There is no default value for repo_path_git, so you cannot count on its presence unless you know what you're doing and are working with a camp type that definitely specifies a value for it.
Options to pass literally to the "git clone ..." call when building a new camp and cloning from the source Git repo. See the "git-clone" man pages for valid options. There is no default, meaning that the effective default behavior is invoking git-clone without options.
Git is efficient in its use of disk space, but by using the various options like local, shared, or reference can potentially be even more efficient and speedy.
Specifies a branch to track and checkout after cloning. Use this in the case that your Git repository's active branch differs from the branch the camp type needs to work with. This would likely be the case if you're using a single master Git repository to track several lines of development; one camp type would work with the default branch "master", while other camp types would need to specify alternate branches.
This assumes that you're using a remote name of "origin". Since git_clone_options allows you to specify a different name for your clone, git_branch introduces some extra semantics: if the value of git_branch contains a forward slash ("/"), the word characters preceding the first slash are treated as the remote name.
If you use git_clone_options "-o foo" to specify a remote name of "foo" rather than "origin",
you can have your camp type check out branch "bar" via git_branch of "foo/bar". However, if you're using the default clone remote name of "origin", then git_branch "bar" would suffice to checkout branch "bar".
This also means that if the desired branch has forward slashes in the name, you'll need to include the remote name as the first portion of your git_branch value. So, if you want a branch named "releases/1.1", you would need a git_branch value of "origin/releases/1.1".
Rendering and installing arbitrary files into a camp from templates is a critical aspect of the camp system. Configuration files for Apache, Interchange, Rails, etc. should be reduced to templates, with things like hostnames, file paths, port numbers, etc. replaced with camp-system tokens of the form described in the section regarding CONFIGURATION VARIABLES. At camp creation time, Camp::Master will parse each template, performing token substitution, and install each parsed template into the appropriate location within the new camp.
The files to parse are specified within the camp type subdirectory's camp-config-files file. Specify one path per line; blanks and lines starting with the pound character are ignored.
All the files specified will undergo the described token substitution and installation into the new camp. The paths specified in camp-config-files are expected to be relative to the camp type directory's 'etc/' subdirectory, and also reflect the target path of the file when copied into the new camp itself. Thus, a file at /home/camp/some_type/etc/blah/foo.conf would be registered in camp-config-files with a path relative to etc/, or "blah/foo.conf", and would be installed at /home/some_user/campNN/blah/foo.conf after parsing.
Also, some assumptions are made about what files will be included if your camp uses Interchange versus Rails:
Note that these are not mutually exclusive. It is theoretically acceptable for a single camp to employ both app server types. If you are bursting with curiosity and sadly deficient in sanity, by all means try this out.
While this can be expected to vary between deployments, a number of files are obvious candidates for inclusion within this templating scheme:
Anything that relies on file paths, ports, domain names, etc. should be abstracted out into such templates. A common pattern (implied by interchange_local.cfg and catalog_local.cfg above) is to encapsulate such details within "local" configuration files that are included from master configuration files; the master configuration files can stay in version control and have no need to vary between camps, production, etc., while the "local" configuration files are small, containing only what is absolutely necessary, and vary in a controlled way between environments by virtue of being managed by the camp system.
VERSION CONTROL SYSTEM OPTIONSThe camp system provides three different version control choices:
Typically speaking, use of Git for a particular camp type implies non-use of Subversion, and the opposite also holds. However, there is no reason that all types cannot be made available for use within the same camp type; the Git software suite comes with '''git-svn''', which can be used to track a Subversion repository via a Git repository. Therefore, all options can be available for a camp type provided the administrator takes the trouble to arrange it. That arrangement takes place largely outside of the camp system itself, and would instead involve post-commit hooks within one or both repositories to ensure that a change in one repository is pushed to the other.
Camp::Master ultimately makes no assumptions about your camp type's VCS offerings; if a camp user attempts to make a new camp using some version control system, Camp::Master will attempt to do it, and if the requested version control system hasn't been configured for the camp type, an error will result. Configuration options to tell Camp::Master about your VCS setup are fairly straightforward; see the configuration variable information for:
While no assumptions are made, the strong recommendation is to use Git. Subversion is slow, inefficient in its use of disk space, relatively inflexible, bad at branching, etc.
GIT USEWhen creating a new camp using Git as the VCS type, Camp::Master assumes an SVN-like flow of operations; the camp type specifies where the Git repository lives, and the new camp will result in a new repository that tracks that central repository. Consequently, the operations used by Camp::Master tend to be clone, pull, and push operations, for remote repository use. When you're working within your Git-based camp, you of course need to keep this in mind, knowing that a checkout or a commit are only relative to the repository of that camp, and that you ultimately need a push to commit your stuff to the central camp repository.
At this time, Camp::Master doesn't provide any tools for creating a Git camp using an alternate repository as the repository to track; this is something that can be done using git directly, after creating a new camp. Create the camp with Git as the VCS choice, then use git directly within that camp to point the camp at an alternate remote repository.
BUGS AND LIMITATIONSThis module has tests. It also probably has bugs. It is, after all, software.
AUTHOREthan Rowe and other contributors
LICENSE AND COPYRIGHTCopyright (C) 2006-2016 End Point Corporation, https://www.endpoint.com/
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see: http://www.gnu.org/licenses/
© 2006–2019 End Point Corporation