1 # Puppet provisioning ReadMe

     2 

     3 This document will describe how to set up a Puppet-provisioned Vagrant VM ready for development, as well as the inner working of the provisioning.

     4 

     5 ## How to use:

     6 

     7 ### Requirements:

     8 - Oracle VirtualBox 4.3.12

     9 - Vagrant 1.6.5

    10 - Mercurial (to get the source code of the project)

    11 - Git (to clone the required subrepositories if necessary)

    12 

    13 NOTE: Make sure your git and mercurial pulls are set to clone the file "as-is", especially if you're on Windows, due to all kind of line ending problems that can occur if you don't.

    14 

    15 ### Installing

    16 

    17 Once you have installed VirtualBox and Vagrant on your machine, pull the mercurial repository in a directory of your choice (we'll refer to it as /root/ from now on). Check if you have the following directories in /root/dev/modules:

    18 

    19 - apt

    20 - concat

    21 - elasticsearch

    22 - postgresql

    23 - stdlib

    24 - sysconfig

    25 

    26 If you miss all the modules save sysconfig, this means the subrepositories didn't import properly. To remedy this, look up the /root/.hgsub files and use git clone commands to manually import the missing modules in their respective directories.

    27 

    28 You can customize the installation (modifying database names and informations, or virtualenv location for instance) by creating a custom.yaml file from the custom.yaml.tmpl file in the /root/dev/ directory and uncommenting values you want to override. This is by no means necessary and puppet will manage to set up your machine using default values if you don't.

    29 

    30 Once all the files are here and your options are properly configured, you just have to run the following command from /root/dev:

    31 

    32     vagrant up

    33     

    34 Once the vagrant up command is done setting up the machine, run this command

    35 

    36     vagrant reload

    37     

    38 So it fixes a bug with shared folders (see below).

    39 

    40 This will set up the virtualbox and provision it via puppet. From there, you can ssh into your machine, activate your virtualenv from the directory you specified in the config file (default is /home/vagrant/myspelenv/).

    41 

    42 Your /root/ directory will be located on the virtual machine in the /srv/ directory.

    43 

    44 You can then run the following command (from the /srv/root/src directory) for testing purpose:

    45 

    46     python manage.py runserver 0.0.0.0:8000

    47 

    48 You'll have to open a browser page to 127.0.0.1:8001 to see your test site.

    49 

    50 ###Notes and known bugs

    51 

    52 Windows users can experience a bug with shared folder (shared folders would only work the first time you setup the vm) due to the way Windows (doesn't) handle symlinks. Here are the steps to fix it manually:

    53 

    54 * ssh into the machine

    55 * run the following command:

    56 

    57         sudo /etc/init.d/vboxadd setup

    58     

    59 * then from the host machine vagrantfile directory, run

    60 

    61         vagrant reload

    62 

    63 The machine should then set up correctly when you use 'vagrant up'

    64 

    65 

    66 

    67 #How it works

    68 

    69 ## Some little things about Puppet

    70 

    71 Puppet syntax is declarative. This means that for Puppet to automatically set up your VM, you basically just have to tell Puppet the resources you need installed on your VM, the commands it must run when required, as well as the order in which Puppet should do all this.

    72 

    73 Puppet will need 2 things to run properly:

    74 * a .pp manifest file and the path to it, ours is site.pp, which we put in the root/dev/manifests directory. This manifest is the backbone of the puppet script and the one we use to arrange all our provisioning so Puppet installs everything in the right order.

    75 * a path to find required modules, which we put in the /root/dev/modules directory.

    76 

    77 These informations must be provided to Vagrant so it knows that we will set up our VM with puppet. If you open the VagrantFile, you will see these lines around the end of the file:

    78 

    79     config.vm.provision :puppet do |puppet|

    80         puppet.manifests_path = "manifests"

    81         puppet.manifest_file  = "site.pp"

    82         puppet.module_path    = "modules"

    83         puppet.options        = "--hiera_config /vagrant/hiera.yaml"

    84         puppet.facter = {

    85           "vagrant_base_path" => File.dirname(__FILE__)

    86     }

    87     end

    88     

    89 This bit of code tells Vagrant to use Puppet to provision the machine, and specify the different informations we talked about just above. The last one we need to describe is the puppet.option line, which tells puppet to check the hiera.yaml file. This file is used to tell puppet where to find the files containing the definitions of the global configuration variables: basically it tells puppet to check our config.yaml file and from there set the appropriate global variables.

    90 

    91 ## A look at site.pp

    92 

    93 Here is our site.pp file:

    94 

    95     exec {

    96     'apt_update_site':

    97       command     => '/usr/bin/apt-get update',

    98       timeout     => 2400,

    99       returns     => [ 0, 100 ];

   100       #refreshonly => true;

   101     

   102     }

   103     

   104     Exec["apt_update_site"] -> Package <| |>

   105     

   106     # upgrade system

   107     class { 'sysconfig::sys_upgrade': }->

   108     

   109     # params

   110     class { 'sysconfig::params': }->

   111     

   112     # install packages

   113     class { 'sysconfig::packages': }->

   114     

   115     # install postgres

   116     class { 'sysconfig::postgresql': }->

   117     

   118     # set up the virtualenv

   119     class { 'sysconfig::virtualenv': }->

   120     

   121     # write config

   122     class { 'sysconfig::config': }->

   123     

   124     # write django_init

   125     class { 'sysconfig::django_init': }->

   126     

   127     exec { 'shared_folder_fix_OracleVM_4.3.12':

   128        command  => 'sudo /etc/init.d/vboxadd setup'

   129     }

   130     

   131 The first exec block runs the apt-get update command. Just below this we have the Exec["apt_update_site"] -> Package <| |> which is meant to tell Puppet "do the "apt_update_site" command before installing any package resource". 

   132 

   133 Then we have a number of class declarations, which are all subclasses of the "sysconfig" class from the "sysconfig" submodule, which is the custom module for puppet provisioning we are building. Each class has a corresponding .pp file with the relevant instructions in, located in /root/dev/modules/sysconfig/manifests/. This way we can properly identify what each file do and most importantly when. You will notice the -> at the end of class declarations, this is used to tell Puppet in which order it must process the different classes we have declared.

   134 

   135 At the end of the file is an exec command that will be executed last, meant to fix the shared folders problems we can encounter on Windows.

   136 

   137 ### The Sysconfig submodule

   138 

   139 The Sysconfig submodule is our main module. All the .pp manifests can be found in the /root/dev/modules/sysconfig/manifest folder. A notable one is init.pp which is a mandatory manifest in any Puppet submodule containing the sysconfig class definition. All the other manifests are the subclasses of the sysconfig class that are called in site.pp.

   140 

   141 * sys_upgrade class

   142 

   143 This class executes commands such as apt-get upgrade and apt-get dist upgrade, as well as adding the apt postgresql repository and key to the sources.list.d folder so the postgresql module will get it from there when installing postgresql.

   144 

   145 * params class

   146 

   147 This class has a number of hiera(var_name, default_value) calls, which Puppet uses to set a the variable named var_name to the corresponding value from the config file mentioned in the hiera.yaml file. If there is no config.yaml file or it can't find your config.yaml file (it can be because you didn't make one), it will set the variable to default_value. 

   148 

   149 This is the class where you can find all the default values for the variables used in the puppet provisioning.

   150 

   151 * packages class

   152 

   153 A straightforward class that will install all the required packages for your provisioning that you would normally install via apt-get install command if you didn't use puppet. It use the puppet Package resource for this. The package list is not generated so if you need a new package at some point, you'll have to add it here for it to be installed automatically on new VMs.

   154 

   155 * postgres class

   156 

   157 This class will install and setup Postgresql according to the parameters specified in config.yaml. 

   158 

   159 * virtualenv class

   160 

   161 This class will set up the virtual env in the directory mentioned in config.yaml and install all python packages into it, using the create_python_env.py script and the resulting project-boot.py script.

   162 

   163 * config class

   164  

   165 The config class generates the config.py file using a template you can find in /root/dev/modules/sysconfig/template. To do this we use the File resource of Puppet with the attribute "ensure => 'present' " and "content => template(templatepath)"

   166 

   167 * django_init class

   168  

   169 Finally, this class initialize Django, by doing the syncb and superuser creation. Additionally for spel, it does the manip with south commenting for migrations by calling a Python script (stored in root/dev/files) that parse the file and replace "'south'," by "#'south'," and vice-versa.