Welcome to Vagga’s documentation!¶

Example¶

Let’s make config for hello-world flask application. To start you need to put following in vagga.yaml:

containers:
  flask: ❶
    setup:
    - !Ubuntu trusty ❷
    - !UbuntuUniverse ❸
    - !Install [python3-flask] ❹
commands:
  py3: !Command ❺
    container: flask ❻
    run: python3 ❼

• ❶ – create a container “flask”
• ❷ – install base image of ubuntu
• ❸ – enable the universe repository in ubuntu
• ❹ – install flask from package (from ubuntu universe)
• ❺ – create a simple command “py3”
• ❻ – run command in container “flask”
• ❼ – the command-line is “python3”

To run command just run vagga command_name:

$ vagga py3
[ .. snipped container build log .. ]
Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>

This is just a lazy example. Once your project starts to mature you want to use some specific version of flask and some other dependencies:

containers:
  flask:
    setup:
    - !Ubuntu trusty
    - !Py3Install
      - werkzeug==0.9.4
      - MarkupSafe==0.23
      - itsdangerous==0.22
      - jinja2==2.7.2
      - Flask==0.10.1
      - sqlalchemy==0.9.8

And if another developer does git pull and gets this config, running vagga py3 next time will rebuild container and run command in the new environment without any additional effort:

$ vagga py3
[ .. snipped container build log .. ]
Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask, sqlalchemy
>>>

You probably want to move python dependencies into requirements.txt:

containers:
  flask:
    setup:
    - !Ubuntu trusty
    - !Py3Requirements "requirements.txt"

And vagga is smart enough to rebuild if requirements.txt change.

In case you’ve just cloned the project you might want to run bare vagga to see which commands are available. For example, here are some commands available in vagga project itself:

$ vagga
Available commands:
    make                Build vagga
    build-docs          Build vagga documentation
    test                Run self tests

(the descriptions on the right are added using description key in command)

More Reading¶

• Managing Dependencies with Vagga shows basic concepts of using vagga and what problems it solves.
• The Higher Level Package Manager – discussion of vagga goals and future
• Evaluating Mesos discuss how to run network tolerance tests.

Command-Centric Workflow¶

When you start working on project, you don’t need to know anything about virtual machines, dependencies, paths whatever. You just need to know what you can do with it.

Consider we have an imaginary web application. Let’s see what we can do:

$ git clone git@git.git:somewebapp.git somewebapp
$ cd somewebapp
$ vagga
Available commands:
    build-js    build javascript files needed to run application
    serve       serve a program on a localhost

Ok, now we know that we probably expected to build javascipt files and that we can run a server. We now just do:

$ vagga build-js
# container created, dependencies populated, javascripts are built
$ vagga serve
Now you can go to http://localhost:8000 to see site in action

Compare that to vagrant:

$ vagrant up
# some machine(s) created
$ vagrant ssh
# now you are in new shell. What to do?
$ make
# ok probably something is built (if project uses make), what now?
$ less README
# long reading follows

Or compare that to docker:

$ docker pull someuser/somewebapp
$ docker run --rm --it someuser/somewebapp
# if you are lucky something is run, but how to build it?
# let's see the README

Lazy Container Creation¶

There are few interesting cases where lazy containers help.

containers:

  build:
    setup:
    - !Ubuntu trusty
    - !Install [make, nodejs, uglifyjs]

  serve:
    setup:
    - !Ubuntu trusty
    - !UbuntuUniverse
    - !Install [python-django]

commands:

  build-js: !Command
    container: build
    run: "make build-js"

  serve: !Command
    container: serve
    run: "python manage.py runserver"

containers:

  testing:
    setup:
    - !Ubuntu trusty
    - !UbuntuUniverse
    - !Install [make, nodejs, uglifyjs, python-django, nosetests]

commands:

  test:
    container: testing
    run: [nosetests]

$ vagrant up
# two containers are up at this point
$ vagrant ssh build -- make
# built, now we don't want to waste memory for build virtual machine
$ vagrant halt build
$ vagrant ssh serve -- python manage.py runserver

$ vagga
Available commands:
    md2html         convert markdown to html without installation
    tests           run tests
    example-web     run live demo (flask app)
    example-plugin  example of plugin for markdown parser
$ vagga example-web
Now go to http://localhost:8000 to see the demo

$ ls -R examples
examples/web:
Vagrantfile README flask-app.py

examples/plugin:
Vagrantfile README main.py plugin.py

$ cd examples/web
$ vagrant up && vagrant ssh -- python main.py --help
$ vagrant ssh -- python main.py --port 8000
# ok got it, let's stop it
$ vagrant halt && vagrant destroy

Container Versioning and Rebuilding¶

What if the project dependencies are changed by upstream? No problem:

$ git pull
$ vagga serve
# vagga notes that dependencies changed, and rebuilds container
$ git checkout stable
# moving to stable branch, to fix some critical bug
$ vagga serve
# vagga uses old container that is probably still around

Vagga hashes dependencies, and if the hash changed creates new container. Old ones are kept around for a while, just in case you revert to some older commit or switch to another branch.

How you do this with Vagrant:

$ git pull
$ vagrant ssh -- python manage.py runserver
ImportError
$ vagrant reload
$ vagrant ssh -- python manage.py runserver
ImportError
$ vagrant reload --provision
#  If you are lucky and your provision script is good, dependency installed
$ vagrant ssh -- python manage.py runserver
# Ok it works
$ git checkout stable
$ vagrant ssh -- python manage.py runserver
# Wow, we still running dependencies from "master", since we added
# a dependency it works for now, but may crash when deploying
$ vagrant restart --provision
# We used ``pip install requirements.txt`` in provision
# and it doesn't delete dependencies
$ vagrant halt
$ vagrant destroy
$ vagrant up
# let's wait ... it sooo long.
$ vagrant ssh -- python manage.py runserver
# now we are safe
$ git checkout master
# Oh no, need to rebuild container again?!?!

Using Docker? Let’s see:

$ git pull
$ docker run --rm -it me/somewebapp python manage.py runserver
ImportError
$ docker tag me/somewebapp:latest me/somewebapp:old
$ docker build -t me/somewebapp .
$ docker run --rm -it me/somewebapp python manage.py runserver
# Oh, that was simple
$ git checkout stable
$ docker run --rm -it me/somewebapp python manage.py runserver
# Oh, crap, I forgot to downgrade container
# We were smart to tag old one, so don't need to rebuild:
$ docker run --rm -it me/somewebapp:old python manage.py runserver
# Let's also rebuild dependencies
$ ./build.sh
Running: docker run --rm me/somewebapp_build python manage.py runserver
# Oh crap, we have hard-coded container name in build script?!?!

Well, docker is kinda easier because we can have multiple containers around, but still hard to get right.

Running Multiple Processes¶

Many projects require multiple processes around. E.g. when running web application on development machine there are at least two components: database and app itself. Usually developers run database as a system process and a process in a shell.

When running in production one usually need also a cache and a webserver. And developers are very lazy to run those components on development system, just because it’s complex to manage. E.g. if you have a startup script like this:

#!/bin/sh
redis-server ./config/redis.conf &
python manage.py runserver

You are going to loose redis-server running in background when python process dead or interrupted. Running them in different tabs of your terminal works while there are two or three services. But today more and more projects adopt service-oriented architecture. Which means there are many services in your project (e.g. in our real-life example we had 11 services written by ourselves and we also run two mysql and two redis nodes to emulate clustering).

This means either production setup and development are too diverse, or we need better tools to manage processes.

How vagrant helps? Almost in no way. You can run some services as a system services inside a vagrant. And you can also have multiple virtual machines with services, but this doesn’t solve core problem.

How docker helps? It only makes situation worse, because now you need to follow logs of many containers, and remember to docker stop and docker rm the processes on every occasion.

Vagga’s way:

commands:
  run_full_app: !Supervise
    children:
      web: !Command
        container: python
        run: "python manage.py runserver"
      redis: !Command
        container: redis
        run: "redis-server ./config/redis.conf"
      celery: !Command
        container: python
        run: "python manage.py celery worker"

Now just run:

$ vagga run_full_app
# two python processes and a redis started here

It not only allows you to start processes in multiple containers, it also does meaningful monitoring of them. The stop-on-failure mode means if any process failed to start or terminated, terminate all processes. It’s opposite to the usual meaning of supervising, but it’s super-useful development tool.

Let’s see how it’s helpful. In example above celery may crash (for example because of misconfiguration, or OOM, or whatever). Usually when running many services you have many-many messages on startup, so you may miss it. Or it may crash later. So you click on some task in web app, and wait when the task is done. After some time, you think that it may be too long, and start looking in logs here and there. And after some tinkering around you see that celery is just down. Now, you lost so much time just waiting. Wouldn’t it be nice if everything is just crashed and you notice it immediately? Yes it’s what stop-on-failure does.

Then if you want to stop it, you just press Ctrl+C and wait for it to shut down. If it hangs for some reason (may be you created a bug), you repeat or press Ctrl+/ (which is SIGQUIT), or just do kill -9 from another shell. In any case vagga will not exit until all processes are shut down and no hanging processes are left ever (Yes, even with kill -9).

User Namespaces¶

As you might noticed that adding user to docker group (if your docker socket is accessed by docker group), is just like giving him a paswordless sudo. This is because root user in docker container is same root that one on host. Also user that can start docker container can mount arbitrary folder in host filesystem into the container (So he can just mount /etc and change /etc/passwd).

Vagga is different as it uses a user namespaces and don’t need any programs running as root or setuid programs or sudo (except systems’ builtin newuidmap/newgidmap if you want more that one user inside a container, but newuidmap setuid binary is very small functionally and safe).

No Central Daemon¶

Vagga keeps your containers in .vagga dir inside your project. And runs them just like any other command from your shell. I.e. command run with vagga is child of your shell, and if that process is finished or killed, its just done. No need to delete container in some central daemon like docker has (i.e. docker doesn’t always remove containers even when using --rm).

Docker also shares some daemon configuration between different containers even run by different users. There is no such sharing in vagga.

Also not having central daemon shared between users allows us to have a user-defined settings file in $HOME/.config/vagga/.

Children Processes¶

Running processes as children of current shell has following advantages:

• You can monitor process and restart when dead (needs polling in docker), in fact there a command type supervise that does it for you)
• File descriptors may be passed to process
• Processes/containers may be socket-activated (e.g. using systemd --user)
• Stdout and stderr streams are just inherited file descriptors, and they are separate (docker mixes the two; it also does expensive copying of the stream from the container to the client using HTTP api)

Filesystems¶

All files in vagga is kept in .vagga/container_name/ so you can inspect all persistent filesystems easily, without finding cryptic names in some system location, and without sudo

Filesystem Permissions¶

Docker by default runs programs in container as root. And it’s also a root on the host system. So usually in your development project you get files with root owner. While it’s possible to specify your uid as a user for running a process in container, it’s not possible to do it portable. I.e. your uid in docker container should have passwd entry. And somebody else may have another uid so must have a different entry in /etc/passwd. Also if some process realy needs to be root inside the container (e.g. it must spawn processes by different users) you just can’t fix it.

With help of user namespaces Vagga runs programs as a root inside a container, but it looks like your user outside. So all your files in project dir are still owned by you.

Security¶

While docker has enterprise support, including security updates. Vagga doesn’t have such (yet).

However, Vagga runs nothing with root privileges. So even running root process in guest system is at least as secure as running any unprivileged program in host sytem. It also uses chroot and linux namespaces for more isolation. Compare it to docker which doesn’t consider running as root inside a container secure.

You can apply selinux or apparmor rules for both.

Filesystem Redundancy¶

Vagga creates each container in .vagga as a separate directory. So theoretically it uses more space than layered containers in docker. But if you put that dir on btrfs filesystem you can use bedup to achieve much better redundancy than what docker provides.

Containers¶

While vagrant emulates full virtual machine, vagga uses linux containers. So you don’t need hardware virtualization and a supervisor. So usually vagga is more light on resources.

Also comparing to vagrant where you run project inside a virtual machine, vagga is suited to run commands inside a container, not a full virtual machine with SSH. In fact many vagga virtual machines don’t have a shell and/or a package manager inside.

Commands¶

While vagrant is concentrated around vagrant up and VM boot process. Light containers allows you to test your project in multiple environments in fraction of second without waiting for boot or having many huge processes hanging around.

So instead of having vagrant up and vagrant ssh we have user-defined commands like vagga build or vagga run or vagga build-a-release-tarball.

Linux-only¶

While vagrant works everywhere, vagga only works on linux systems with recent kernel and userspace utilities.

If you use a mac, just run vagga inside a vagrant container, just like you used to run docker :)

Half-isolation¶

Being only a container allows vagga to share memory with host system, which is usually a good thing.

Memory and CPU usage limits can be enforced on vagga programs using cgroups, just like on any other process in linux. Vagga runs only on quite recent linux kernels, which has much more limit capabilities than previous ones.

Also while vagrant allows to forward selected network ports, vagga by default shares network interface with the host system. Isolating and forwarding ports will be implemented soon.

Containers¶

Example of one container defined:

containers:
  sphinx:
    setup:
    - !Ubuntu trusty
    - !Install [python-sphinx, make]

The YAML above defines a container named sphinx, which is built with two steps: download and unpack ubuntu trusty base image, and install install packages name python-sphinx, make inside the container.

Commands¶

Example of command defined:

commands:
  build-docs: !Command
    description: Build vagga documentation using sphinx
    container: sphinx
    work-dir: docs
    run: make

The YAML above defines a command named build-docs, which is run in container named sphinx, that is run in docs/ sub dir of project, and will run command make in container. So running:

$ vagga build-docs html

Builds html docs using sphinx inside a container.

See commands for comprehensive description of how to define commands.

Common Parameters¶

These parameters work for both kinds of commands:

description¶

Description that is printed in when vagga is run without arguments

banner¶

The message that is printed before running process(es). Useful for documenting command behavior.

banner-delay¶

The seconds to sleep before printing banner. For example if commands run a web service, banner may provide a URL for accessing the service. The delay is used so that banner is printed after service startup messages not before. Note that currently vagga sleeps this amount of seconds even if service is failed immediately.

epilog¶

The message printed after command is run. It’s printed only if command returned zero exit status. Useful to print further instructions, e.g. to display names of build artifacts produced by command.

Parameters of !Command¶

container¶

The container to run command in

tags¶

The list of tags for this command. Tags are used for processes filtering (with --only and --exclude) when running any !Supervise command.

Simple example:

commands:
  run: !Supervise
    children:
      postgres: !Command
        tags: [service]
        run: ...
      redis: !Command
        tags: [service]
        run: ...
      app: !Command
        tags: [app]
        run: ...

$ vagga run --only service  # will start only postgres and redis processes

run¶

The command to run. It can be:

• either a string encompassing a shell command line (which is feeded to /bin/sh -c)
• or a list containing first the full path to the executable to run and then possibly static arguments.

work-dir¶

The working directory to run in. Path relative to project root. By default command is run in the same directory where vagga started (sans the it’s mounted as /work so the output of pwd would seem to be different)

accepts-arguments¶

Denotes whether command accepts additional arguments. Defaults to:

• false for a shell command line (if run is a string);
• true if command is an executable (if run is a list).

NB: If command is a shell command line - even if it’s composed of only one call to an executable -, arguments are given to its executing context, not appended to it.

environ¶

The mapping of environment to pass to command. This overrides environment specified in container on value by value basis.

pid1mode¶

This denotes what is run as pid 1 in container. It may be wait, wait-all-children or exec. The default wait is ok for most regular processes. See What’s Special With Pid 1? for more info.

write-mode¶

The parameter specifies how container’s base file system is used. By default container is immutable (corresponds to the read-only value of the parameter), which means you can only write to the /tmp or to the /work (which is your project directory).

Another option is transient-hard-link-copy, which means that whenever command is run, create a copy of the container, consisting of hard-links to the original files, and remove the container after running command. Should be used with care as hard-linking doesn’t prevent original files to be modified. Still very useful to try package installation in the system. Use vagga _build --force container_name to fix base container if that was modified.

user-id¶

The user id to run command as. If the external-user-id is omitted this has same effect like using sudo -u inside container (except it’s user id instead of user name)

external-user-id¶

(experimental) This option allows to map the user-id as seen by command itself to some other user id inside container namespace (the namespace which is used to build container). To make things a little less confusing, the following two configuration lines:

user-id: 1
external-user-id: 0

Will make your command run as user id 1 visible inside the container (which is “daemon” or “bin” depending on distribution). But outside the container it will be visible as your user (i.e. user running vagga). Which effectively means you can create/modify files in project directory without permission errors, but tar and other commands which have different behaviour when running as root would think they are not root (but has user id 1)

Parameters of !Supervise¶

mode¶

The set of processes to supervise and mode. See Supervision for more info

children¶

A mapping of name to child definition of children to run. All children are started simultaneously.

kill-unresponsive-after¶

(default 2 seconds) If some process exits (in stop-on-failure mode), vagga will send TERM signal to all the other processes. If they don’t finish in the specified number of seconds, vagga will kill them with KILL signal (so they finish without being able to intercept signal unconditionally). If you don’t like this behavior set the parameter to some large value.

Generic Installers¶

To run arbitrary shell command use !Sh:

setup:
- !Ubuntu trusty
- !Sh "apt-get install -y python"

If you have more than one-liner you may use YAMLy literal syntax for it:

setup:
- !Ubuntu trusty
- !Sh |
   wget somepackage.tar.gz
   tar -xzf somepackage.tar.gz
   cd somepackage
   make && make install

To run !Sh you need /bin/sh. If you don’t have shell in container you may use !Cmd that runs command directly:

setup:
# ...
- !Cmd [/usr/bin/python, '-c', 'print "hello from build"']

To install a package of any (supported) linux distribution just use !Install command:

containers:

  ubuntu:
    setup:
    - !Ubuntu trusty
    - !Install [python]

  ubuntu-precise:
    setup:
    - !Ubuntu precise
    - !Install [python]

  alpine:
    setup:
    - !Alpine v3.1
    - !Install [python]

Occasionally you need some additional packages to use for container building, but not on final machine. Use !BuildDeps for them:

setup:
- !Ubuntu trusty
- !Install [python]
- !BuildDeps [python-dev, gcc]
- !Sh "make && make install"

The python-dev and gcc packages from above will be removed after building whole container.

To add some environment arguments to subsequent build commands use !Env:

setup:
# ...
- !Env
  VAR1: value1
  VAR2: value2
- !Sh "echo $VAR1 / $VAR2"

Sometimes you want to rebuild container when some file changes. For example if you have used the file in the build. There is a !Depends command which does nothing per se, but add a dependency. The path must be relative to your project directory (the dir where vagga.yaml is). For example:

setup:
# ...
- !Depends requirements.txt
- !Sh "pip install -r requirements.txt"

To download and unpack tar archive use !Tar command:

setup:
- !Tar
  url: http://something.example.com/some-project-1.0.tar.gz
  sha256: acd1234...
  path: /
  subdir: some-project-1.0

Only url field is mandatory. If url starts with dot . it’s treated as filename inside project directory. The path is target path to unpack into, and subdir is a dir inside tar file. By default path is root of new filesystem. The subdir is a dir inside the tar file, if omitted whole tar archive will be unpacked.

You can use !Tar command to download and unpack the root filesystem from scratch.

There is a shortcut to download tar file and build and install from there, which is !TarInstall:

setup:
- !TarInstall
  url: http://static.rust-lang.org/dist/rust-0.12.0-x86_64-unknown-linux-gnu.tar.gz
  sha256: abcd1234...
  subdir: rust-0.12.0-x86_64-unknown-linux-gnu
  script: ./install.sh --prefix=/usr

Only the url is mandatory here too. Similarly, if url starts with dot . it’s treated as filename inside project directory. The script is by default ./configure --prefix=/usr; make; make install. It’s run in subdir of unpacked archive. If subdir is omitted it’s run in the only subdirectory of the archive. If archive contains more than one directory and subdir is empty, it’s an error, however you may use . as subdir.

To remove some data from the image after building use !Remove command:

setup:
# ...
- !Remove /var/cache/something

To clean directory but ensure that directory exists use !EmptyDir command:

setup:
# ...
- !EmptyDir /tmp

To ensure that directory exists use !EnsureDir command. It’s very often used for future mount points:

setup:
# ...
- !EnsureDir /sys
- !EnsureDir /dev
- !EnsureDir /proc

Sometimes you want to keep some cache between builds of container or similar containers. Use !CacheDirs for that:

setup:
# ...
- !CacheDirs { "/var/cache/apt": "apt-cache" }

Mutliple directories may be specified at once.

Sometimes you just want to write a file in target system:

setup:
# ...
- !Text
  /etc/locale.conf: |
     LANG=en_US.UTF-8
     LC_TIME=uk_UA.UTF-8

Ubuntu¶

To install base ubuntu system use:

setup:
- !Ubuntu trusty

Potentially any ubuntu long term support release instead of trusty should work. To install a non LTS release, use:

setup:
- !UbuntuRelease { version: 14.10 }

To install any ubuntu package use generic !Install command:

setup:
- !Ubuntu trusty
- !Install python

Many interesting ubuntu packages are in the “universe” repository, you may add it by series of !UbuntuRepo commands (see below), but there is shortcut !UbuntuUniverse:

setup:
- !Ubuntu trusty
- !UbuntuUniverse
- !Install [checkinstall]

The !UbuntuRepo command adds additional repository. For example, to add marathon repository you may write:

setup:
- !Ubuntu trusty
- !UbuntuRepo
  url: http://repos.mesosphere.io/ubuntu
  suite: trusty
  components: [main]
- !Install [mesos, marathon]

This effectively adds the repository and installs mesos and marathon packages.

Alpine¶

To install base alpine system use:

setup:
- !Alpine v3.1

Potentially any alpine version instead of v3.1 should work.

To install any alpine package use generic !Install command:

setup:
- !Alpine v3.1
- !Install [python]

Npm Installer¶

You can build somewhat default nodejs environment using !NpmInstall command. For example:

setup:
- !Ubuntu trusty
- !NpmInstall [react-tools]

All node packages are installed as --global which should be expected. If no distribution is specified before the !NpmInstall command, the implicit !Alpine v3.1 (in fact the latest version) will be executed.

setup:
- !NpmInstall [react-tools]

So above should just work as expected if you don’t need any special needs. E.g. it’s usually perfectly ok if you only use node to build static scripts.

The following npm features are supported:

• Specify package@version to install specific version (recommended)
• Use git:// url for the package. In this case git will be installed for the duration of the build automatically
• Bare package_name (should be used only for one-off environments)

Other forms may work, but are unsupported for now.

Python Installer¶

There are two separate commands for installing packages for python2 and python3. Here is a brief example:

setup:
- !Ubuntu trusty
- !Py2Install [sphinx]

We always fetch latest pip for installing dependencies. The python-dev headers are installed for the time of the build too. Both python-dev and pip are removed when installation is finished.

The following pip package specification formats are supported:

• The package_name==version to install specific version (recommended)
• Bare package_name (should be used only for one-off environments)
• The git+ and hg+ links (the git and mercurial are installed as build dependency automatically), since vagga 0.4 git+https and hg+https are supported too (required installing ca-ceritificates manually before)

All other forms may work but not supported. Specifying command-line arguments instead of package names is not supported. To configure pip use !PipConfig directive. In the example there are full list of parameters:

setup:
- !Ubuntu trusty
- !PipConfig
  index-urls: ["http://internal.pypi.local"]
  find-links: ["http://internal.additional-packages.local"]
  dependencies: true
- !Py2Install [sphinx]

They should be self-descriptive. Note unlike in pip command line we use single list both for primary and “extra” indexes. See pip documentation for more info about options

Better way to specify python dependencies is to use “requirements.txt”:

setup:
- !Ubuntu trusty
- !Py3Requirements "requirements.txt"

This works the same as Py3Install including auto-installing of version control packages and changes tracking. I.e. It will rebuild container when “requirements.txt” change. So ideally in python projects you may use two lines above and that’s it.

The Py2Requirements command exists too.

PHP/Composer Installer¶

Composer packages can be installed either explicitly or from composer.json. For example:

setup:
- !Ubuntu trusty
- !ComposerInstall [laravel/installer]

The packages will be installed using Composer’s global require at /usr/local/lib/composer/vendor. This is only useful for installing packages that provide binaries used to bootstrap your project (like the Laravel installer, for instance):

setup:
- !Ubuntu trusty
- !ComposerInstall [laravel/installer]
- !Sh laravel new src

Alternatively, you can use Composer’s crate-project command:

setup:
- !Ubuntu trusty
- !ComposerInstall # just to have composer available
- !Sh composer create-project --prefer-dist laravel/laravel src

For your project dependencies, you should install packages from your composer.json. For example:

setup:
- !Ubuntu trusty
- !ComposerDependencies

This command will install packages (including dev) from composer.json into /usr/local/lib/composer/vendor using Composer’s install command.

You can also specify some options available from Composer command line, for example:

setup:
- !Ubuntu trusty
- !ComposerDependencies
  working_dir: src # run command inside src directory
  dev: false # do not install dev dependencies
  optimize_autoloader: true

If you want to use hhvm, you can disable the installation of the php runtime:

setup:
- !Ubuntu trusty
- !ComposerConfig
  install_runtime: false
  runtime_exe: hhvm

Note that you will have to manually install hhvm and set the include_path:

setup:
- !Ubuntu trusty
- !UbuntuUniverse
- !AptTrust keys: ["hhvm apt key here"]
- !UbuntuRepo
  url: http://dl.hhvm.com/ubuntu
  suite: trusty
  components: [main]
- !Install [hhvm]
- !ComposerConfig
  install_runtime: false
  runtime_exe: hhvm
- !Sh echo '.:/usr/local/lib/composer' >> /etc/hhvm/php.ini

Ruby Installer¶

Ruby gems can be installed either by providing a list of gems or from a Gemfile using bundler. For example:

setup:
- !Alpine v3.3
- !GemInstall [rake]

We will update gem to the latest version (unless specified not to) for installing gems. The ruby-dev headers are installed for the time of the build too and are removed when installation is finished.

The following gem package specification formats are supported:

• The package_name:version to install specific version (recommended)
• Bare package_name (should be used only for one-off environments)

setup:
- !Alpine v3.3
- !Install [libxml2, libxslt, zlib, sqlite-libs]
- !BuildDeps [libxml2-dev, libxslt-dev, zlib-dev, sqlite-dev]
- !Env
  NOKOGIRI_USE_SYSTEM_LIBRARIES: 1
  HOME: /tmp
- !GemInstall [rails]
- !Sh rails new . --skip-bundle

Bundler is also available for installing gems from Gemfile. For example:

setup:
- !Alpine v3.3
- !GemBundle

You can also specify some options to Bundler, for example:

setup:
- !Alpine v3.3
- !GemBundle
  gemfile: src/Gemfile # use this Gemfile
  without: [development, test] # groups to exclude when installing gems
  trust_policy: HighSecurity

It is possible to avoid installing ruby if you are providing it yourself:

setup:
- !Alpine v3.3
- !GemSettings
  install_ruby: false
  gem_exe: /usr/bin/gem

Dependent Containers¶

Sometimes you want to build on top of another container. For example, container for running tests might be based on production container, but it might add some test utils. Use !Container command for that:

container:
  base:
    setup:
    - !Ubuntu trusty
    - !Py3Install [django]
  test:
    setup:
    - !Container base
    - !Py3Install [nosetests]

It’s also sometimes useful to freeze some part of container and test next build steps on top of it. For example:

container:
  temporary:
    setup:
    - !Ubuntu trusty
    - !TarInstall
      url: http://download.zeromq.org/zeromq-4.1.0-rc1.tar.gz
  web:
    setup:
    - !Container temporary
    - !Py3Install [pyzmq]

In this case when you try multiple different versions of pyzmq, the zeromq itself will not be rebuilt. When you’re done, you can append build steps and remove the temporary container.

Sometimes you need to generate (part of) vagga.yaml itself. For some things you may just use shell scripting. For example:

container:
  setup:
  - !Ubuntu trusty
  - !Env { VERSION: 0.1.0 }
  - !Sh "apt-get install somepackage==$VERSION"

In more complex scenarios you may want to generate real vagga.yaml. You may use that with ancillary container and !SubConfig command. For example, here is how we use a docker2vagga script to transform Dockerfile to vagga config:

docker-parser: ❶
  setup:
  - !Alpine v3.1
  - !Install [python]
  - !Depends Dockerfile ❷
  - !Depends docker2vagga.py ❷
  - !Sh 'python ./docker2vagga.py > /docker.yaml' ❸

somecontainer:
  setup:
  - !SubConfig
    source: !Container docker-parser ❶
    path: docker.yaml ❹
    container: docker-smart ❺

Few comments:

• ❶ – container used for build, it’s rebuilt automatically as a dependency for “somecontainer”
• ❷ – normal dependency rules apply, so you must add external files that are used to generate the container and vagga file in it
• ❸ – put generated vagga file inside a container
• ❹ – the “path” is relative to the source if the latter is set
• ❺ – name of the container used inside a “docker.yaml”

The !SubConfig command may be used to include some commands from another file without building container. Just omit source command:

subdir:
  setup:
  - !SubConfig
    path: subdir/vagga.yaml
    container: containername

The YAML file used may be a partial container, i.e. it may contain just few commands, installing needed packages. The other things (including the name of the base distribution) can be set by original container:

# vagga.yaml
containers:
  ubuntu:
    setup:
    - !Ubuntu trusty
    - !SubConfig
      path: packages.yaml
      container: packages
  alpine:
    setup:
    - !Alpine v3.1
    - !SubConfig
      path: packages.yaml
      container: packages

# packages.yaml
containers:
  packages:
    setup:
    - !Install [redis, bash, make]

Container Bootstrap¶

Command that can be used to bootstrap a container (i.e. may work on top of empty container):

• Alpine
• Ubuntu
• UbuntuRelease
• SubConfig
• Container
• Tar

Ubuntu Commands¶

Ubuntu¶

Simple and straightforward way to install Ubuntu LTS release.

Example:

setup:
- !Ubuntu trusty

The value is single string having the codename of release trusty or precise known to work at the time of writing.

The Ubuntu LTS images are updated on daily basis. But vagga downloads and caches the image. To update the image that was downloaded by vagga you need to clean the cache.

UbuntuRelease¶

This is more exensible but more cumbersome way to setup ubuntu (comparing to Ubuntu). For example to install trusty you need:

- !UbuntuRelease { version: 14.04 }

But you can install non-lts version with this command:

- !UbuntuRelease { version: 15.10 }

All options:

version

The verison of ubuntu to install. This must be digital YY.MM form, not a code name. Required.

arch

The architecture to install. Defaults to amd64.

keep-chfn-command

(default false) This may be set to true to enable /usr/bin/chfn command in the container. This often doesn’t work on different host systems (see #52 as an example). The command is very rarely useful, so the option here is for completeness only.

AptTrust¶

This command fetches keys with apt-key and adds them to trusted keychain for package signatures. The following trusts a key for fkrull/deadsnakes repository:

- !AptTrust keys: [5BB92C09DB82666C]

By default this uses keyserver.ubuntu.com, but you can specify alternative:

- !AptTrust
  server: hkp://pgp.mit.edu
  keys: 1572C52609D

This is used to get rid of the error similar to the following:

WARNING: The following packages cannot be authenticated!
  libpython3.5-minimal python3.5-minimal libpython3.5-stdlib python3.5
E: There are problems and -y was used without --force-yes

Options:

server

(default keyserver.ubuntu.com) Server to fetch keys from. May be a hostname or hkp://hostname:port form.

keys

(default []) List of keys to fetch and add to trusted keyring. Keys can include full fingerprint or suffix of the fingerprint. The most common is the 8 hex digits form.

UbuntuRepo¶

Adds arbitrary debian repo to ubuntu configuration. For example to add newer python:

- !UbuntuRepo
  url: http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu
  suite: trusty
  components: [main]
- !Install [python3.5]

See UbuntuPPA for easier way for dealing specifically with PPAs.

Options:

url

Url to the repository. Required.

suite

Suite of the repository. The common practice is that the suite is named just like the codename of the ubuntu release. For example trusty. Required.

components

List of the components to fetch packages from. Common practice to have a main component. So usually this setting contains just single element components: [main]. Required.

UbuntuPPA¶

A shortcut to UbuntuRepo that adds named PPA. For example, the following:

- !Ubuntu trusty
- !AptTrust keys: [5BB92C09DB82666C]
- !UbuntuPPA fkrull/deadsnakes
- !Install [python3.5]

Is equivalent to:

- !Ubuntu trusty
- !UbuntuRepo
  url: http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu
  suite: trusty
  components: [main]
- !Install [python3.5]

UbuntuUniverse¶

The singleton step. Just enables an “universe” repository:

- !Ubuntu trusty
- !UbuntuUniverse
- !Install [checkinstall]

Alpine Commands¶

Alpine¶

setup:
- !Alpine v3.2

Distribution Commands¶

These commands work for any linux distributions as long as distribution is detected by vagga. Latter basically means you used Alpine, Ubuntu, UbuntuRelease in container config (or in parent config if you use SubConfig or Container)

Install¶

setup:
- !Ubuntu trusty
- !Install [gcc, gdb]        # On Ubuntu, equivalent to `apt-get install gcc gdb -y`
- !Install [build-essential] # `apt-get install build-essential -y`
# Note that `apt-get install` is run 2 times in this example

BuildDeps¶

setup:
- !Ubuntu trusty
- !BuildDeps [wget]
- !Sh echo "We can use wget here, but no curl"
- !BuildDeps [curl]
- !Sh echo "We can use wget and curl here"
# Container built. Now, everything in BuildDeps(wget and curl) is removed from the container.

Generic Commands¶

Sh¶

Runs arbitrary shell shell command, for example:

- !Ubuntu trusty
- !Sh "apt-get install -y package"

If you have more than one-liner you may use YAMLy literal syntax for it:

setup:
- !Alpine v3.2
- !Sh |
   if [ ! -z "$(which apk)" ] && [ ! -z "$(which lbu)" ]; then
     echo "Alpine"
   fi
- !Sh echo "Finished building the Alpine container"

Warning

To run !Sh you need /bin/sh in the container. See Cmd for more generic command runner.

Note

The !Sh command is run by /bin/sh -exc. With the flags meaning -e – exit if any command fails, -x – print command before executing, -c – execute command. You may undo -ex by inserting set +ex at the start of the script. But it’s not recommended.

Cmd¶

Runs arbitrary command in the container. The argument provided must be a YAML list. For example:

   setup:
   - !Ubuntu trusty
   - !Cmd ["apt-get", "install", "-y", "python"]

You may use YAMLy features to get complex things. To run complex python
code you may use::

    setup:
    - !Cmd
      - python
      - -c
      - |
        import socket
        print("Builder host", socket.gethostname())

Or to get behavior similar to :step:`Sh` command, but with different shell:

    setup:
    - !Cmd
      - /bin/bash
      - -exc
      - |
        echo this is a bash script

Download¶

Downloads file and puts it somewhere in the file system.

Example:

- !Download
  url: https://jdbc.postgresql.org/download/postgresql-9.4-1201.jdbc41.jar
  path: /opt/spark/lib/postgresql-9.4-1201.jdbc41.jar

Note

This step does not require any download tool to be installed in the container. So may be used to put static binaries into container without a need to install the system.

Options:

url

(required) URL to download file from

path

(required) Path where to put file. Should include the file name (vagga doesn’t try to guess it for now). Path may be in /tmp to be used only during container build process.

mode

(default ‘0o644’) Mode (permissions) of the file. May be used to make executable bit enabled for downloaded script

Warning

The download is cached similarly to other commands. Currently there is no way to control the caching. But it’s common practice to publish every new version of archive with different URL (i.e. include version number in the url itself)

Tar¶

Unpacks Tar archive into container’s filesystem.

Example:

- !Tar
  url: http://something.example.com/some-project-1.0.tar.gz
  path: /
  subdir: some-project-1.0

Downloaded file is stored in the cache and reused indefinitely. It’s expected that the new version of archive will have a new url. But occasionally you may need to clean the cache to get the file fetched again.

url

Required. The url or a path of the archive to fetch. If the url startswith dot . it’s treated as a file name relative to the project directory. Otherwise it’s a url of the file to download.

path

(default /). Target path where archive should be unpacked to. By default it’s a root of the filesystem.

subdir

Subdirectory inside the archive to extract. May be . to extract the root of the archive.

This command may be used to populate the container from scratch

TarInstall¶

Similar to Tar but unpacks archive into a temporary directory and runs installation script.

Example:

setup:
- !TarInstall
  url: http://static.rust-lang.org/dist/rust-1.4.0-x86_64-unknown-linux-gnu.tar.gz
  script: ./install.sh --prefix=/usr

url

Required. The url or a path of the archive to fetch. If the url startswith dot . it’s treated as a file name relative to the project directory. Otherwise it’s a url of the file to download.

subdir

Subdirectory which command is run in. May be . to run command inside the root of the archive.

The common case is having a single directory in the archive, and that directory is used as a working directory for script by default.

script

The command to use for installation of the archive. Default is effectively a ./configure --prefix=/usr && make && make install.

The script is run with /bin/sh -exc, to have better error hadling and display. Also this means that dash/bash-compatible shell should be installed in the previous steps under path /bin/sh.

Git¶

Check out a git repository into a container. This command doesn’t require git to be installed in the container.

Example:

setup:
- !Alpine v3.1
- !Install [python]
- !Git
  url: git://github.com/tailhook/injections
  path: /usr/lib/python3.5/site-packages/injections

(the example above is actually a bad idea, many python packages will work just from source dir, but you may get improvements at least by precompiling *.pyc files, see GitInstall)

Options:

url

(required) The git URL to use for cloning the repository

revision

(optional) Revision to checkout from repository. Note if you don’t specify a revision, the latest one will be checked out on the first build and then cached indefinitely

branch

(optional) A branch to check out. Usually only useful if revision is not specified

path

(required) A path where to store the repository.

GitInstall¶

Check out a git repository to a temporary directory and run script. This command doesn’t require git to be installed in the container.

Example:

setup:
- !Alpine v3.1
- !Install [python, py-setuptools]
- !GitInstall
  url: git://github.com/tailhook/injections
  script: python setup.py install

Options:

url

(required) The git URL to use for cloning the repository

revision

(optional) Revision to checkout from repository. Note if you don’t specify a revision, the latest one will be checked out on the first build and then cached indefinitely

branch

(optional) A branch to check out. Usually only useful if revision is not specified

subdir

(default root of the repository) A subdirectory of the repository to run script in

script

(required) A script to run inside the repository. It’s expected that script does compile/install the software into the container. The script is run using /bin/sh -exc

Files and Directories¶

Text¶

Writes a number of text files into the container file system. Useful for wrinting short configuration files (use external files and file copy or symlinks for writing larger configs)

Example:

setup:
- !Text
  /etc/locale.conf: |
     LANG=en_US.UTF-8
     LC_TIME=uk_UA.UTF-8

Copy¶

Copy file or directory into the container. Useful either to put build artifacts from temporary location into permanent one, or to copy files from the project directory into the container.

Example:

setup:
- !Copy
  source: /work/config/nginx.conf
  path: /etc/nginx/nginx.conf

For directories you might also specify regular expression to ignore:

setup:
- !Copy
  source: /work/mypkg
  path: /usr/lib/python3.4/site-packages/mypkg
  ignore-regex: "(~|.py[co])$"

Symlinks are copied as-is. Path translation is done neither for relative nor for absolute symlinks. Hint: relative symlinks pointing inside the copied directory work well, as well as absolute symlinks that point to system locations.

Note

The command fails if any file name has non-utf-8 decodable names. This is intentional. If you really need bad filenames use traditional cp or rsync commands.

Options:

source

(required) Absolute to directory or file to copy. If path starts with /work files are checksummed to get the version of the container.

path

(required) Destination path

ignore-regex

(default (^|/)\.(git|hg|svn|vagga)($|/)|~$|\.bak$|\.orig$|^#.*#$) Regular expression of paths to ignore. Default regexp ignores common revision control folders and editor backup files.

owner-uid, owner-gid

(preserved by default) Override uid and gid of files and directories when copying. It’s expected that most useful case is owner-uid: 0 and owner-gid: 0 but we try to preserve the owner by default. Note that unmapped users (the ones that don’t belong to user’s subuid/subgid range), will be set to nobody (65535).

Warning

If the source directory starts with /work all the files are read and checksummed on each run of the application in the container. So copying large directories for this case may influence container startup time even if rebuild is not needed.

This command is useful for making deployment containers (i.e. to put application code to the container file system). For this case checksumming issue above doesn’t apply. It’s also useful to enable auto-clean for such containers.

Remove¶

Remove file or a directory from the container and keep it clean on the end of container build. Useful for removing cache directories.

This is also inherited by subcontainers. So if you know that some installer leaves temporary (or other unneeded files) after a build you may add this entry instead of using shell rm command. The /tmp directory is cleaned by default. But you may also add man pages which are not used in container.

Example:

setup:
- !Remove /var/cache/something

For directories consider use EmptyDir if you need to keep cleaned directory in the container.

EnsureDir¶

setup:
#...
- !EnsureDir /var/cache/downloads
- !Sh if [ -d "/var/cache/downloads" ]; then echo "Directory created"; fi;
- !EnsureDir /creates/parent/directories

EmptyDir¶

Cleans up a directory. It’s similar to the Remove but keeps directory created.

CacheDirs¶

Adds build cache directories. Example:

- !CacheDirs
  /tmp/pip-cache/http: pip-cache-http
  /tmp/npm-cache: npm-cache

This maps /tmp/pip-cache/http into the cache directory of the vagga, by default it’s ~/.vagga/.cache/pip-cache-http. This allows to reuse same download cache by multiple rebuilds of the container. And if shared cache is used also reuses the cache between multiple projects.

Be picky on the cache names, if file conficts there may lead to unexpected build results.

Note

Vagga uses a lot of cache dirs for built-in commands. For example the ones described above are used whenever you use Py* and Npm* commands respectively. You don’t need to do anything special to use cache.

Meta Data¶

Env¶

Set environment variables for the build.

Example:

setup:
- !Env HOME: /root

Note

The variables are used only for following build steps, and are inherited on the Container directive. But they are not used when running the container.

Depends¶

Rebuild the container when a file changes. For example:

setup:
# ...
- !Depends requirements.txt
- !Sh "pip install -r requirements.txt"

The example is not the best one, you could use Py3Requirements for the same task.

Only the hash of the contents of a file is used in versioning the container not an owner or permissions. Consider adding the auto-clean option if it’s temporary container that depends on some generated file (sometimes useful for tests).

Sub-Containers¶

Container¶

Build a container based on another container:

container:
  base:
    setup:
    - !Ubuntu trusty
    - !Py3Install [django]
  test:
    setup:
    - !Container base
    - !Py3Install [nosetests]

There two known use cases of functionality:

1. Build test/deploy containers on top of base container (example above)
2. Cache container build partially if you have to rebuild last commands of the container frequently

In theory, the container should behave identically as if the commands would be copy-pasted to the setup fo dependent container, but sometimes things doesn’t work. Known things:

1. The packages in a BuildDeps are removed
2. Remove and EmptyDir will empty the directory
3. Build with temporary-mount is not mounted

If you have any other bugs with container nesting report in the bugtracker.

Note

Container step doesn’t influence environ and volumes as all other options of the container in any way. It only somewhat replicate setup sequence. We require whole environment be declared manually (you you can use YAMLy aliases)

SubConfig¶

This feature allows to generate (parts of) vagga.yaml for the container. For example, here is how we use a docker2vagga script to transform Dockerfile into vagga config:

docker-parser: ❶
  setup:
  - !Alpine v3.1
  - !Install [python]
  - !Depends Dockerfile ❷
  - !Depends docker2vagga.py ❷
  - !Sh 'python ./docker2vagga.py > /docker.yaml' ❸

somecontainer:
  setup:
  - !SubConfig
    source: !Container docker-parser ❶
    path: docker.yaml ❹
    container: docker-smart ❺

Few comments:

• ❶ – container used for build, it’s rebuilt automatically as a dependency for “somecontainer”
• ❷ – normal dependency rules apply, so you must add external files that are used to generate the container and vagga file in it
• ❸ – put generated vagga file inside a container
• ❹ – the “path” is relative to the source if the latter is set
• ❺ – name of the container used inside a “docker.yaml”

Warning

The functionality of !SubConfig is experimental and is a subject to change in future. In particular currently the /work mount point and current directory used to build container are those of initial vagga.yaml file. It may change in future.

The !SubConfig command may be used to include some commands from another file without building container. Just omit generator command:

subdir:
  setup:
  - !SubConfig
    path: subdir/vagga.yaml
    container: containername

The YAML file used may be a partial container, i.e. it may contain just few commands, installing needed packages. The other things (including the name of the base distribution) can be set by original container:

# vagga.yaml
containers:
  ubuntu:
    setup:
    - !Ubuntu trusty
    - !SubConfig
      path: packages.yaml
      container: packages
  alpine:
    setup:
    - !Alpine v3.1
    - !SubConfig
      path: packages.yaml
      container: packages

# packages.yaml
containers:
  packages:
    setup:
    - !Install [redis, bash, make]

Build¶

This command is used to build some parts of the container in another one. For example:

containers:
  webpack: ❶
    setup:
    - !NpmInstall [webpack]
    - !NpmDependencies
  jsstatic:
    setup:
    - !Container webpack ❶
    - !Copy ❷
        source: /work/frontend
        path: /tmp/js
    - !Sh |
        cd /tmp/js
        webpack --output-path /var/javascripts
    auto-clean: true ❸
  nginx:
    setup:
    - !Alpine v3.3
    - !Install [nginx]
    - !Build
      container: jsstatic
      source: /var/javascripts
      path: /srv/www

Note the following things:

• ❶ – We use separate container for npm dependencies so we don’t have to rebuild it on each change of the sources
• ❷ – We copy javascript sources into our temporary container. The important part of copying operation is that all the sources are hashed and versioned when copying. So container will be rebuild on source changes. Since we don’t need sources in the container we just put them in temporary folder.
• ❸ – The temporary container is cleaned automatically (there is low chance that it will ever be reused)

Technically it works similar to !Container except it doesn’t apply configuration from the source container and allows to fetch only parts of the resulting container.

Another motivating example is building a package:

containers:
  pkg:
    setup:
    - !Ubuntu trusty
    - !Install [build-essential]
    - !EnsureDir /packages
    - !Sh |
        checkinstall --pkgname=myapp --pakdir=/packages make
    auto-clean: true
  nginx:
    setup:
    - !Ubuntu trusty
    - !Build
      container: pkg
      source: /packages
      temporary-mount: /tmp/packages
    - !Sh dpkg -i /tmp/packages/mypkg_0.1.deb

Normal versioning of the containers apply. This leads to the following consequences:

• Putting multiple Build steps with the same container will build container only once (this way you may extract multiple folders from the single container).
• Despite the name Build dependencies are not rebuilt.
• The Build command itself depends only on the container but on on the individual files. You need to ensure that the source container is versioned well (sometimes you need Copy or Depends for the task)

Options:

container

(required) Name of the container to build and to extract data from

source

(default /) Source directory (absolute path inside the source container) to copy files from

path

Target directory (absolue path inside the resulting container) to copy (either path or temporary-mount required)

temporary-mount

A directory to mount source into. This is useful if you don’t want to copy files, but rather want to use files from there. The directory is created automatically if not exists, but not parent directories. It’s probably good idea to use a subdirectory of the temporary dir, like /tmp/package. The mount is read-only and persists until the end of the container build and is not propagated through Container step.

Node.JS Commands¶

NpmInstall¶

Example:

setup:
- !NpmInstall [babel-loader@6.0, webpack]

Install a list of node.js packages. If no linux distributions were used yet !NpmInstall installs the latest Alpine distribution. Node is installed automatically and analog of the node-dev package is also added as a build dependency.

Note

Packages installed this way (as well as those installed by !NpmDependencies are located under /usr/lib/node_modules. In order for node.js to find them, one should set the environment variable NODE_PATH, making the example become

Example:

setup:
- !NpmInstall [babel-loader@6.0, webpack]
environ:
  NODE_PATH: /usr/lib/node_modules

NpmDependencies¶

Works similarly to NpmInstall but installs packages from package.json. For example:

- !NpmDependencies

This installs dependencies and devDependencies from package.json into a container (with --global flag).

You may also customize package.json and install other kinds of dependencies:

- !NpmDependencies
  file: frontend/package.json
  peer: true
  optional: true
  dev: false

Note

Since npm supports a whole lot of different versioning schemes and package sources, some features may not work or may not version properly. You may send a pull request for some unsupported scheme. But we are going to support only the popular ones. Generally, it’s safe to assume that we support a npmjs.org packages and git repositories with full url.

Note

We don’t use npm install . to execute this command but rather use a command-line to specify every package there. It works better because npm install --global . tries to install this specific package to the system, which is usually not what you want.

Options:

file

(default package.json) A file to get dependencies from

package

(default true) Whether to install package dependencies (i.e. the ones specified in dependencies key)

dev

(default true) Whether to install devDependencies (we assume that vagga is mostly used for develoment environments so dev dependencies should be on by default)

peer

(default false) Whether to install peerDependencies

bundled

(default true) Whether to install bundledDependencies (and bundleDependencies too)

optional

(default false) Whether to install optionalDependencies. By default npm tries to install them, but don’t fail if it can’t install. Vagga tries its best to guarantee that environment is the same, so dependencies should either install everywhere or not at all. Additionally because we don’t use “npm install package.json” as described earlier we can’t reproduce npm’s behavior exactly. But optional dependencies of dependencies will probably try to install.

Warning

This is a new command. We can change default flags used, if that will be more intuitive for most users.

NpmConfig¶

The directive configures various settings of npm commands above. For example, you may want to turn off automatic nodejs installation so you can use custom oversion of it:

- !NpmConfig
    install_node: false
    npm_exe: /usr/local/bin/npm
- !NpmInstall [webpack]

Note

Every time NpmConfig is specified, options are replaced rather than augmented. In other words, if you start a block of npm commands with NpmConfig, all subsequent commands will be executed with the same options, no matter which NpmConfig settings were before.

All options:

npm-exe

(default is npm) The npm command to use for installation of packages.

install-node

(default true) Whether to install nodejs and npm automatically. Setting the option to false is useful for setting up custom version of the node.js.

Python Commands¶

PipConfig¶

The directive configures various settings of pythonic commands below. The mostly used option is dependencies:

- !PipConfig
    dependencies: true
- !Py3Install [flask]

Most options directly correspond to the pip command line options so refer to pip help for more info.

Note

Every time PipConfig is specified, options are replaced rather than augmented. In other words, if you start a block of pythonic commands with PipConfig, all subsequent commands will be executed with the same options, no matter which PipConfig settings were before.

All options:

dependencies

(default false) allow to install dependencies. If the option is false (by default) pip is run with pip --no-deps

index-urls

(default []) List of indexes to search for packages. This corresponds to --index-url (for the first element) and --extra-index-url (for all subsequent elements) options on the pip command-line.

When the list is empty (default) the pypi.python.org is used.

find-links

(default []) List of URLs to HTML files that need to be parsed for links that indicate the packages to be downloaded.

trusted-hosts

(default []) List of hosts that are trusted to download packages from.

cache-wheels

(default true) Cache wheels between different rebuilds of the container. The downloads are always cached. Only binary wheels are toggled with the option. It’s useful to turn this off if you build many containers with different dependencies.

Starting with vagga v0.4.1 cache is namespaced by linux distribution and version. It was single shared cache in vagga <= v0.4.0

install-python

(default true) Install python automatically. This will install either python2 or python3 with a default version of your selected linux distribution. You may set this parameter to false and install python yourself. This flag doesn’t disable automatic installation of pip itself and version control packages. Note that by default python-dev style packages are as build dependencies installed too.

python-exe

(default is either python2 or python3 depending on which command is called, e.g. Py2Install or Py3Install) This allows to change executable of python. It may be either just name of the specific python interpreter (python3.5) or full path. Note, when this is set, the command will be called both for Py2* commands and Py3* commands.

Py2Install¶

Installs python package for Python 2.7 using pip. Example:

setup:
- !Ubuntu trusty
- !Py2Install [sphinx]

We always fetch latest pip for installing dependencies. The python-dev headers are installed for the time of the build too. Both python-dev and pip are removed when installation is finished.

The following pip package specification formats are supported:

• The package_name==version to install specific version (recommended)
• Bare package_name (should be used only for one-off environments)
• The git+ and hg+ links (the git and mercurial are installed as build dependency automatically), since vagga 0.4 git+https and hg+https are supported too (required installing ca-ceritificates manually before)

All other forms may work but not supported. Specifying command-line arguments instead of package names is not supported.

See Py2Requirements for the form that is both more convenient and supports non-vagga installations better.

Note

If you configure python-exe in PipConfig there is no difference between Py2Install and Py3Install.

Py2Requirements¶

This command is similar to Py2Install but gets package names from the file. Example:

setup:
- !Ubuntu trusty
- !Py2Requirements "requirements.txt"

See Py2Install for more details on package installation and PipConfig for more configuration.

Py3Install¶

Same as Py2Install but installs for Python 3.x by default.

setup:
- !Alpine v3.3
- !Py3Install [sphinx]

See Py2Install for more details on package installation and PipConfig for more configuration.

Py3Requirements¶

This command is similar to Py3Install but gets package names from the file. Example:

setup:
- !Alpine v3.3
- !Py3Requirements "requirements.txt"

See Py2Install for more details on package installation and PipConfig for more configuration.

PHP/Composer Commands¶

ComposerInstall¶

Example:

setup:
- !Alpine v3.3
- !ComposerInstall ["phpunit/phpunit:~5.2.0"]

Install a list of php packages using composer global require --prefer-dist --update-no-dev. Packages are installed in /usr/local/lib/composer/vendor.

Binaries are automatically installed to /usr/local/bin by Composer so they are available in your PATH.

Composer itself is located at /usr/local/bin/composer and available in your PATH as well. After container is built, the Composer executable is no longer available.

ComposerDependencies¶

Install packages from composer.json using composer install. For example:

- !ComposerDependencies

Similarly to ComposerInstall, packages are installed at /usr/local/lib/composer/vendor, including those listed at require-dev, as Composer default behavior.

Options correspond to the ones available to the composer install command line so refer to composer cli docs for detailed info.

Options:

working_dir

(default None) Use the given directory as working directory

dev

(default true) Whether to install require-dev (this is Composer default behavior).

prefer

(default None) Preferred way to download packages. Can be either source or dist. If no specified, will use Composer default behavior (use dist for stable).

ignore_platform_reqs

(default false) Ignore php, hhvm, lib-* and ext-* requirements.

no_autoloader

(default false) Skips autoloader generation.

no_scripts

(default false) Skips execution of scripts defined in composer.json.

no_plugins

(default false) Disables plugins.

optimize_autoloader

(default false) Convert PSR-0/4 autoloading to classmap to get a faster autoloader.

classmap_authoritative

(default false) Autoload classes from the classmap only. Implicitly enables optimize_autoloader.

ComposerConfig¶

The directive configures various settings of composer commands above. For example, you may want to use hhvm instead of php:

- !ComposerConfig
     install_runtime: false
     runtime_exe: hhvm
 - !ComposerInstall [phpunit/phpunit]

Note

Every time ComposerConfig is specified, options are replaced rather than augmented. In other words, if you start a block of composer commands with ComposerConfig, all subsequent commands will be executed with the same options, no matter which ComposerConfig settings were before.

All options:

runtime_exe

(default php) The command to use for running Composer.

install_runtime

(default true) Whether to install the default runtime (php) automatically. Setting the option to false is useful when using hhvm, for example.

install_dev

(default false) Whether to install development packages (php-dev). Defaults to false since it is rare for php projects to build modules and it may require manual configuration.

include_path

(default .:/usr/local/lib/composer) Set include_path. This option overrides the default include_path instead of appending to it;

Note

Setting install_runtime to false still installs Composer.

Ruby Commands¶

GemInstall¶

Example:

setup:
- !Alpine v3.3
- !GemInstall [rake]

Install a list of ruby gems using gem install --bindir /usr/local/bin --no-document.

The --bindir option instructs gem to install binaries in /usr/local/bin so they are available in your PATH.

GemBundle¶

Install gems from Gemfile using bundle install --system --binstubs /usr/local/bin. For example:

- !GemBundle

Options correspond to the ones available to the bundle install command line, so refer to bundler documentation for detailed info.

Options:

gemfile

(default Gemfile) Use the specified gemfile instead of Gemfile.

without

(default []) Exclude gems that are part of the specified named group.

trust_policy

(default None) Sets level of security when dealing with signed gems. Accepts LowSecurity, MediumSecurity and HighSecurity as values.

GemConfig¶

The directive configures various settings of ruby commands above:

- !GemConfig
     install_ruby: true
     gem_exe: gem
     update_gem: true
 - !GemInstall [rake]

Note

Every time GemConfig is specified, options are replaced rather than augmented. In other words, if you start a block of ruby commands with GemConfig, all subsequent commands will be executed with the same options, no matter which GemConfig settings were before.

All options:

install_ruby

(default true) Whether to install ruby.

gem_exe

(default /usr/bin/gem) The rubygems executable.

update_gem

(default true) Whether to update rubygems itself.

Note

If you set install_ruby to false you will also have to provide rubygems if needed.

Note

If you set gem_exe, vagga will no try to update rubygems.

Upgrading 0.4.1 -> 0.5.0¶

This release doesn’t introduce any severe incompatibilities. Except in the networking support:

• Change gateway network from 172.18.0.0/16 to 172.23.0.0/16, hopefully this will have less collisions

The following are minor changes during the container build:

• The stdin redirected from /dev/null and stdout is redirected to stderr during the build. If you really need asking a user (which is an antipattern) you may open a /dev/tty.
• The .vagga/.mnt is now unmounted during build (fixes bugs with bad tools)
• !Depends doesn’t resolve symlinks but depends on the link itself
• !Remove removes files when encountered (previously removed only when container already built), also the command works with files (not only dirs)

The following are bugfixes in container runtime:

• The TERM and *_proxy env vars are now propagated for supervise commands in the same way as with normal commands (previously was absent)
• Pseudo-terminals in vagga containers now work
• Improved SIGINT handling, now Ctrl+C in interactive processes such as python (without arguments) works as expected
• The signal messages (“Received SIGINT...”) are now printed into stderr rather than stdout (for !Supervise type of commands)
• Killing vagga supervise with TERM mistakenly reported SIGINT on exit, fixed

And the following changes the hash of containers (this should not cause a headache, just will trigger a container rebuild):

• Add support for arch parameter in !UbuntuRelease this changes hash sum of all containers built using !UbuntuRelease

See Release Notes and Github for all changes.

Upgrading 0.4.0 -> 0.4.1¶

This is minor release so it doesn’t introduce any severe incompatibilities. The pip cache in this release is namespaced over distro and version. So old cache will be inactive now. And should be removed manually by cleaning .vagga/.cache/pip-cache directory. You may do that at any time

See Release Notes and Github for all changes.

Upgrading 0.3.x -> 0.4.x¶

The release is focused on migrating from small amount of C code to “unshare” crate and many usability fixes, including ones which have small changes in semantics of configuration. The most important changes:

• The !Sh command now runs shell with -ex this allows better error reporting (but may change semantics of script for some obscure cases)
• There is now kill-unresponsive-after setting for !Supervise commands with default value of 2. This means that processes will shut down unconditionally two seconds after Ctrl+C.

See Release Notes and Github for all changes.

Upgrading 0.2.x -> 0.3.x¶

This upgrade should be seamless. The release is focused on migrating code from pre-1.0 Rust to... well... rust 1.2.0.

Other aspect of code migration is that it uses musl libc. So building vagga from sources is more complex now. (However it’s as easy as previous version if you build with vagga itself, except you need to wait until rust builds for the first time).

Upgrading 0.1.x -> 0.2.x¶

There are basically two things changed:

1. The way how containers (images) are built
2. Differentiation of commands

rust:
  builder: ubuntu
  parameters:
    repos: universe
    packages: make checkinstall wget git uidmap
  provision: |
    wget https://static.rust-lang.org/dist/rust-0.12.0-x86_64-unknown-linux-gnu.tar.gz
    tar -xf rust-0.12.0-x86_64-unknown-linux-gnu.tar.gz
    cd rust-0.12.0-x86_64-unknown-linux-gnu
    ./install.sh --prefix=/usr

rust:
  setup:
  - !Ubuntu trusty
  - !UbuntuUniverse ~
  - !TarInstall
    url: http://static.rust-lang.org/dist/rust-1.0.0-alpha-x86_64-unknown-linux-gnu.tar.gz
    script: "./install.sh --prefix=/usr"
  - !Install [make, checkinstall, git, uidmap]
  - !Sh "echo Done"

Supervision Modes¶

There are three basic modes of operation:

• stop-on-failure – stops all processes as soon as any single one is dead (default)
• wait-all – wait for all processes to finish
• restart – always restart dead processes

In any mode of operation supervisor itself never exits until all the children are dead. Even when you kill supervisor with kill -9 or kill -KILL all children will be killed with -KILL signal too. I.e. with the help of namespaces and good old PR_SET_PDEATHSIG we ensure that no process left when supervisor killed, no one is reparented to init, all traces of running containers are cleared. Seriously. It’s very often a problem with many other ways to run things on development machine.

commands:
  run_full_app: !Supervise
    mode: stop-on-failure
    children:
      web: !Command
        container: python
        run: "python manage.py runserver"
      celery: !Command
        container: python
        run: "python manage.py celery worker"

Tips¶

# run everything, except the web process we are debugging
$ vagga run_full_app --exclude web
# then in another tab
$ vagga run_full_app --only web

Builtin Commands¶

All commands have --help, so we don’t duplicate all command-line flags here

vagga _run CONTAINER CMD ARG...

run arbitrary command in container defined in vagga.yaml

vagga _build CONTAINER

Builds container without running a command.

More useful in the form:

$ vagga _build --force container_name

To rebuid container that has previously been built.

vagga _clean

Removes images and temporary files created by vagga.

The following command removes containers that are not used by current vagga config (considering the state of all files that vagga.yaml depends on):

$ vagga _clean --unused

There is a faster option for removing unused containers:

$ vagga _clean --old

This is different because it only looks at symlinks in .vagga/*. So may be wrong (if you changed vagga.yaml and did not run the command(s)). It’s faster because it doesn’t calculate the hashsums. But the difference in speed usually not larger than a few seconds (on large configs). The existence of the two commands should probably be treated as a historical accident and --unused variant preferred.

For other operations and paremeters see vagga _clean --help

vagga _list

List of commands (similar to running vagga without command)

vagga _version_hash CONTAINER

Prints version hash for the container. In case the image has not been built (or config has been updated since) it should return new hash. But sometimes it’s not possible to determine the hash in advance. In this case command returns an error.

Might be used in some automation scripts.

vagga _init_storage_dir

If you have configured a storage-dir in settings, say /vagga-storage, when you run vagga _init_storage_dir abc will create a /vagga-storage/abc and .vagga with .vagga/.lnk pointing to the directory. The command ensures that the storage dir is not used for any other folder.

This is created for buildbots which tend to clean .vagga directory on every build (like gitlab-ci) or just very often.

vagga _pack_image IMAGE_NAME

Pack image into the tar archive, optionally compressing and output it into stdout (use shell redirection > file.tar to store it into the file).

It’s very similar to tar -cC .vagga/IMAGE_NAME/root except it deals with file owners and permissions correctly. And similar to running vagga _run IMAGE_NAME tar -c / except it ignores mounted file systems.

vagga _push_image IMAGE_NAME

Push container image IMAGE_NAME into the image cache.

Actually it boils down to packing an image into tar (vagga _pack_image) and running push-image-script, see the documentation of the setting to find out how to configure image cache.

Normal Commands¶

If command declared as !Command you get a command with the following usage:

Usage:
    vagga [OPTIONS] some_command [ARGS ...]

Runs a command in container, optionally builds container if that does not
exists or outdated. Run `vagga` without arguments to see the list of
commands.

positional arguments:
  some_command          Your defined command
  args                  Arguments for the command

optional arguments:
  -h,--help             show this help message and exit
  -E,--env,--environ NAME=VALUE
                        Set environment variable for running command
  -e,--use-env VAR      Propagate variable VAR into command environment
  --no-build            Do not build container even if it is out of date.
                        Return error code 29 if it's out of date.
  --no-version-check    Do not run versioning code, just pick whatever
                        container version with the name was run last (or
                        actually whatever is symlinked under
                        `.vagga/container_name`). Implies `--no-build`

All the ARGS that follow command are passed to the command even if they start with dash -.

Supervise Commands¶

If command declared as !Supervise you get a command with the following usage:

Usage:
    vagga run [OPTIONS]

Run full server stack

optional arguments:
  -h,--help             show this help message and exit
  --only PROCESS_NAME [...]
                        Only run specified processes
  --exclude PROCESS_NAME [...]
                        Don't run specified processes
  --no-build            Do not build container even if it is out of date.
                        Return error code 29 if it's out of date.
  --no-version-check    Do not run versioning code, just pick whatever
                        container version with the name was run last (or
                        actually whatever is symlinked under
                        `.vagga/container_name`). Implies `--no-build`

Currently there is no way to provide additional arguments to commands declared with !Supervise.

The --only and --exclude arguments are useful for isolating some single app to a separate console. For example, if you have vagga run that runs full application stack including a database, cache, web-server and your little django application, you might do the following:

$ vagga run --exclude django

Then in another console:

$ vagga run --only django

Now you have just a django app that you can observe logs from and restart independently of other applications.

Global Settings¶

Settings are searched for in one of the following files:

• $HOME/.config/vagga/settings.yaml
• $HOME/.vagga/settings.yaml
• $HOME/.vagga.yaml

Supported settings:

storage-dir¶

Directory where to put images build by vagga. Usually they are stored in .vagga subdirectory of the project dir. It’s mostly useful when the storage-dir points to a directory on a separate partition. Path may start with ~/ which means path is inside the user’s home directory.

cache-dir¶

Directory where to put cache files during the build. This is used to speed up the build process. By default cache is put into .vagga/.cache in project directory but this setting allows to have cache directory shared between multiple projects. Path may start with ~/ which means path is inside the user’s home directory.

site-settings¶

(experimental) The mapping of project paths to settings for this specific project.

proxy-env-vars¶

Enable forwarding for proxy environment variables. Default true. Environment variables currently that this setting influence currently: http_proxy, https_proxy, ftp_proxy, all_proxy, no_proxy.

external-volumes¶

A mapping of volume names to the directories inside the host file system.

Note

The directories must exist even if unused in any vagga.yaml.

For example, here is how you might export home:

external-volumes:
  home: /home/user

Then in vagga.yaml you use it as follows (prepend with /volumes):

volumes:
  /root: !BindRW /volumes/home

See Volumes for more info about defining mount points.

Warning

1. Usage of volume is usually a subject for filesystem permissions. I.e. your user becomes root inside the container, and many system users are not mapped (not present) in container at all. This means that mounting /var/lib/mysql or something like that is useless, unless you chown the directory
2. Any vagga project may use the volume if it’s defined in global config. You may specify the volume in site-settings if you care about security (and you should).

push-image-script¶

A script to use for uploading a container image when you run vagga _push_image.

To push image using webdav:

push-image-script: "curl -T ${image_path} \
    http://example.org/${container_name}.${short_hash}.tar.xz"

To push image using scp utility (SFTP protocol):

push-image-script: "scp ${image_path} \
   user@example.org:/target/path/${container_name}.${short_hash}.tar.xz"

The FTP(s) (for exxample, using lftp utility) or S3 (using s3cmd) are also valid choices.

Note

This is that rare case where command is run by vagga in your host filesystem. This allows you to use your credentials in home directory, and ssh-agent’s socket. But also this means that utility to upload images must be installed in host system.

Variables:

container_name

The name of the container as declared in vagga.yaml

short_hash

The short hash of container setup. This is the same hash that is used to detect whether container configuration changed and is needed to be rebuilt. And the same hash used in directory name .vagga/.roots.

All project-local settings are also allowed here.

Project-Local Settings¶

Project-local settings may be in the project dir in:

• .vagga.settings.yaml
• .vagga/settings.yaml

All project-local settings are also allowed in global config.

While settings can potentially be checked-in to version control it’s advised not to do so.

version-check¶

If set to true (default) vagga will check if the container that is already built is up to date with config. If set to false vagga will use any container with same name already built. It’s only useful for scripts for performance reasons or if you don’t have internet and containers are not too outdated.

ubuntu-mirror¶

Set to your preferred ubuntu mirror. By default it’s mirror://mirrors.ubuntu.com/mirrors.txt which means mirror will be determined automatically. Note that it’s different from default in ubuntu itself where http://archive.ubuntu.com/ubuntu/ is the default.

alpine-mirror¶

Set to your preferred alpine mirror. By default it’s the random one is picked from the list.

Note

Alpine package manager is used not only for building Alpine distribution, but also internally for fetching tools that are outside of the container filesystem (for example to fetch git for Git or GitInstall command(s))

Could not read /etc/subuid or /etc/subgid¶

The full error might look like:

ERROR:vagga::container::uidmap: Error reading uidmap: Can't open /etc/subuid: No such file or directory (os error 2)
WARN:vagga::container::uidmap: Could not read /etc/subuid or /etc/subgid (see http://bit.ly/err_subuid)
error setting uid/gid mappings: Operation not permitted (os error 1)

This means there is no /etc/subuid file. It probably means you need to create one. The recommended contents are following:

your_user_name:100000:65536

You may get another similar error:

ERROR:vagga::container::uidmap: Error reading uidmap: /etc/subuid:2: Bad syntax: "user:100000:100O"
WARN:vagga::container::uidmap: Could not read /etc/subuid or /etc/subgid (see http://bit.ly/err_subuid)
error setting uid/gid mappings: Operation not permitted (os error 1)

This means somebody has edited /etc/subuid and made an error. Just open the file (note it’s owned by root) and fix the issue (in the example the last character should be zero, but it’s a letter “O”).

Can’t find newuidmap or newgidmap¶

Full error usually looks like:

WARN:vagga::process_util: Can't find `newuidmap` or `newuidmap` (see http://bit.ly/err_idmap)
error setting uid/gid mappings: No such file or directory (os error 2)

There might be two reasons for this:

1. The binaries are not installed (see below)
2. The commands are not in PATH

In the latter case you should fix your PATH.

The packages for Ubuntu >= 14.04:

$ sudo apt-get install uidmap

The Ubuntu 12.04 does not have the package. But you may use the package from newer release (the following version works fine on 12.04):

$ wget http://gr.archive.ubuntu.com/ubuntu/pool/main/s/shadow/uidmap_4.1.5.1-1ubuntu9_amd64.deb
$ sudo dpkg -i uidmap_4.1.5.1-1ubuntu9_amd64.deb

Most distributions (known: Nix, Archlinux, Fedora) have binaries as part of “shadow” package, so have them installed on every system.

You should not run vagga as root¶

Well, sometimes users get some permission denied errors and try to run vagga with sudo. Running as root is never an answer.

Here is a quick check list on permission checks:

• Check owner (and permission bits) of .vagga subdirectory if it exists, otherwise the directory where vagga.yaml is (project dir). In case you have already run vagga as root just do sudo rm -rf .vagga
• Could not read /etc/subuid or /etc/subgid
• Can’t find newuidmap or newgidmap
• Check uname -r to have version of 3.9 or greater
• Check sysctl kernel.unprivileged_userns_clone the setting must either not exist at all or have value of 1
• Check zgrep CONFIG_USER_NS /proc/config.gz or grep CONFIG_USER_NS "/boot/config-`uname -r`" (ubuntu) the setting should equal to y

The error message might look like:

You should not run vagga as root (see http://bit.ly/err_root)

Or it might look like a warning:

WARN:vagga::launcher: You are running vagga as a user different from the owner of project directory. You may not have needed permissions (see http://bit.ly/err_root)

Both show that you don’t run vagga with the user that owns the project. The legitimate reasons to run vagga as root are:

• If you run vagga in container (i.e. in vagga itself) and the root is not a real root
• If your project dir is owned by root (for whatever crazy reason)

Both cases should inhibit the warning automatically, but as a last resort you may try vagga --no-owner-check. If you have good case where this works, please file an issue and we might make the check better.