# Puppet provisioning ReadMe

This document will describe how to set up a Puppet-provisioned Vagrant VM ready for development, as well as the inner working of the provisioning.

## How to use:

### Requirements:
- Oracle VirtualBox 4.3.12
- Vagrant 1.6.5
- Mercurial (to get the source code of the project)
- Git (to clone the required subrepositories if necessary)

NOTE: Make sure your git and mercurial pulls are set to clone the file "as-is", especially if you're on Windows, due to all kind of line ending problems that can occur if you don't.

### Installing

Once you have installed VirtualBox and Vagrant on your machine, pull the mercurial repository in a directory of your choice (we'll refer to it as /root/ from now on). Check if you have the following directories in /root/dev/modules:

- apt
- concat
- elasticsearch
- postgresql
- stdlib
- sysconfig

If you miss all the modules save sysconfig, this means the subrepositories didn't import properly. To remedy this, look up the /root/.hgsub files and use git clone commands to manually import the missing modules in their respective directories.

You can customize the installation (modifying database names and informations, or virtualenv location for instance) by creating a custom.yaml file from the custom.yaml.tmpl file in the /root/dev/ directory and uncommenting values you want to override. This is by no means necessary and puppet will manage to set up your machine using default values if you don't.

Once all the files are here and your options are properly configured, you just have to run the following command from /root/dev:

    vagrant up
    
Once the vagrant up command is done setting up the machine, run this command

    vagrant reload
    
So it fixes a bug with shared folders (see below).

This will set up the virtualbox and provision it via puppet. From there, you can ssh into your machine, activate your virtualenv from the directory you specified in the config file (default is /home/vagrant/myspelenv/).

Your /root/ directory will be located on the virtual machine in the /srv/ directory.

You can then run the following command (from the /srv/root/src directory) for testing purpose:

    python manage.py runserver 0.0.0.0:8000

You'll have to open a browser page to 127.0.0.1:8001 to see your test site.

###Notes and known bugs

Windows users can experience a bug with shared folder (shared folders would only work the first time you setup the vm) due to the way Windows (doesn't) handle symlinks. Here are the steps to fix it manually:

* ssh into the machine
* run the following command:

        sudo /etc/init.d/vboxadd setup
    
* then from the host machine vagrantfile directory, run

        vagrant reload

The machine should then set up correctly when you use 'vagrant up'



#How it works

## Some little things about Puppet

Puppet syntax is declarative. This means that for Puppet to automatically set up your VM, you basically just have to tell Puppet the resources you need installed on your VM, the commands it must run when required, as well as the order in which Puppet should do all this.

Puppet will need 2 things to run properly:
* a .pp manifest file and the path to it, ours is site.pp, which we put in the root/dev/manifests directory. This manifest is the backbone of the puppet script and the one we use to arrange all our provisioning so Puppet installs everything in the right order.
* a path to find required modules, which we put in the /root/dev/modules directory.

These informations must be provided to Vagrant so it knows that we will set up our VM with puppet. If you open the VagrantFile, you will see these lines around the end of the file:

    config.vm.provision :puppet do |puppet|
        puppet.manifests_path = "manifests"
        puppet.manifest_file  = "site.pp"
        puppet.module_path    = "modules"
        puppet.options        = "--hiera_config /vagrant/hiera.yaml"
        puppet.facter = {
          "vagrant_base_path" => File.dirname(__FILE__)
    }
    end
    
This bit of code tells Vagrant to use Puppet to provision the machine, and specify the different informations we talked about just above. The last one we need to describe is the puppet.option line, which tells puppet to check the hiera.yaml file. This file is used to tell puppet where to find the files containing the definitions of the global configuration variables: basically it tells puppet to check our config.yaml file and from there set the appropriate global variables.

## A look at site.pp

Here is our site.pp file:

    exec {
    'apt_update_site':
      command     => '/usr/bin/apt-get update',
      timeout     => 2400,
      returns     => [ 0, 100 ];
      #refreshonly => true;
    
    }
    
    Exec["apt_update_site"] -> Package <| |>
    
    # upgrade system
    class { 'sysconfig::sys_upgrade': }->
    
    # params
    class { 'sysconfig::params': }->
    
    # install packages
    class { 'sysconfig::packages': }->
    
    # install postgres
    class { 'sysconfig::postgresql': }->
    
    # set up the virtualenv
    class { 'sysconfig::virtualenv': }->
    
    # write config
    class { 'sysconfig::config': }->
    
    # write django_init
    class { 'sysconfig::django_init': }->
    
    exec { 'shared_folder_fix_OracleVM_4.3.12':
       command  => 'sudo /etc/init.d/vboxadd setup'
    }
    
The first exec block runs the apt-get update command. Just below this we have the Exec["apt_update_site"] -> Package <| |> which is meant to tell Puppet "do the "apt_update_site" command before installing any package resource". 

Then we have a number of class declarations, which are all subclasses of the "sysconfig" class from the "sysconfig" submodule, which is the custom module for puppet provisioning we are building. Each class has a corresponding .pp file with the relevant instructions in, located in /root/dev/modules/sysconfig/manifests/. This way we can properly identify what each file do and most importantly when. You will notice the -> at the end of class declarations, this is used to tell Puppet in which order it must process the different classes we have declared.

At the end of the file is an exec command that will be executed last, meant to fix the shared folders problems we can encounter on Windows.

### The Sysconfig submodule

The Sysconfig submodule is our main module. All the .pp manifests can be found in the /root/dev/modules/sysconfig/manifest folder. A notable one is init.pp which is a mandatory manifest in any Puppet submodule containing the sysconfig class definition. All the other manifests are the subclasses of the sysconfig class that are called in site.pp.

* sys_upgrade class

This class executes commands such as apt-get upgrade and apt-get dist upgrade, as well as adding the apt postgresql repository and key to the sources.list.d folder so the postgresql module will get it from there when installing postgresql.

* params class

This class has a number of hiera(var_name, default_value) calls, which Puppet uses to set a the variable named var_name to the corresponding value from the config file mentioned in the hiera.yaml file. If there is no config.yaml file or it can't find your config.yaml file (it can be because you didn't make one), it will set the variable to default_value. 

This is the class where you can find all the default values for the variables used in the puppet provisioning.

* packages class

A straightforward class that will install all the required packages for your provisioning that you would normally install via apt-get install command if you didn't use puppet. It use the puppet Package resource for this. The package list is not generated so if you need a new package at some point, you'll have to add it here for it to be installed automatically on new VMs.

* postgres class

This class will install and setup Postgresql according to the parameters specified in config.yaml. 

* virtualenv class

This class will set up the virtual env in the directory mentioned in config.yaml and install all python packages into it, using the create_python_env.py script and the resulting project-boot.py script.

* config class
 
The config class generates the config.py file using a template you can find in /root/dev/modules/sysconfig/template. To do this we use the File resource of Puppet with the attribute "ensure => 'present' " and "content => template(templatepath)"

* django_init class
 
Finally, this class initialize Django, by doing the syncb and superuser creation. Additionally for spel, it does the manip with south commenting for migrations by calling a Python script (stored in root/dev/files) that parse the file and replace "'south'," by "#'south'," and vice-versa.