1 #Elasticsearch Puppet module

     2 

     3 ####Table of Contents

     4 

     5 1. [Overview](#overview)

     6 2. [Module description - What the module does and why it is useful](#module-description)

     7 3. [Setup - The basics of getting started with Elasticsearch](#setup)

     8   * [The module manages the following](#the-module-manages-the-following)

     9   * [Requirements](#requirements)

    10 4. [Usage - Configuration options and additional functionality](#usage)

    11 5. [Advanced features - Extra information on advanced usage](#advanced-features)

    12 6. [Limitations - OS compatibility, etc.](#limitations)

    13 7. [Development - Guide for contributing to the module](#development)

    14 8. [Support - When you need help with this module](#support)

    15 

    16 

    17 

    18 ##Overview

    19 

    20 This module manages Elasticsearch (http://www.elasticsearch.org/overview/elasticsearch/)

    21 

    22 ##Module description

    23 

    24 The elasticsearch module sets up Elasticsearch instances and can manage plugins and templates.

    25 

    26 This module has been tested against all versions of ES 1.x and 2.x

    27 

    28 ##Setup

    29 

    30 ###The module manages the following

    31 

    32 * Elasticsearch repository files.

    33 * Elasticsearch package.

    34 * Elasticsearch configuration file.

    35 * Elasticsearch service.

    36 * Elasticsearch plugins.

    37 * Elasticsearch templates.

    38 

    39 ###Requirements

    40 

    41 * The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) Puppet library.

    42 * [ceritsc/yum](https://forge.puppetlabs.com/ceritsc/yum) For yum version lock.

    43 * [richardc/datacat](https://forge.puppetlabs.com/richardc/datacat)

    44 * [Augeas](http://augeas.net/)

    45 

    46 #### Repository management

    47 When using the repository management you will need the following dependency modules:

    48 

    49 * Debian/Ubuntu: [Puppetlabs/apt](http://forge.puppetlabs.com/puppetlabs/apt)

    50 * OpenSuSE: [Darin/zypprepo](https://forge.puppetlabs.com/darin/zypprepo)

    51 

    52 ##Usage

    53 

    54 ###Main class

    55 

    56 ####Install a specific version

    57 

    58 ```puppet

    59 class { 'elasticsearch':

    60   version => '1.4.2'

    61 }

    62 ```

    63 

    64 Note: This will only work when using the repository.

    65 

    66 ####Automatic upgrade of the software ( default set to false )

    67 ```puppet

    68 class { 'elasticsearch':

    69   autoupgrade => true

    70 }

    71 ```

    72 

    73 ####Removal/decommissioning

    74 ```puppet

    75 class { 'elasticsearch':

    76   ensure => 'absent'

    77 }

    78 ```

    79 

    80 ####Install everything but disable service(s) afterwards

    81 ```puppet

    82 class { 'elasticsearch':

    83   status => 'disabled'

    84 }

    85 ```

    86 

    87 ###Instances

    88 

    89 This module works with the concept of instances. For service to start you need to specify at least one instance.

    90 

    91 ####Quick setup

    92 ```puppet

    93 elasticsearch::instance { 'es-01': }

    94 ```

    95 

    96 This will set up its own data directory and set the node name to `$hostname-$instance_name`

    97 

    98 ####Advanced options

    99 

   100 Instance specific options can be given:

   101 

   102 ```puppet

   103 elasticsearch::instance { 'es-01':

   104   config => { },        # Configuration hash

   105   init_defaults => { }, # Init defaults hash

   106   datadir => [ ],       # Data directory

   107 }

   108 ```

   109 

   110 See [Advanced features](#advanced-features) for more information

   111 

   112 ###Plug-ins

   113 

   114 Install [a variety of plugins](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html#known-plugins). Note that `module_dir` is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.

   115 

   116 ####From official repository

   117 ```puppet

   118 elasticsearch::plugin{'lmenezes/elasticsearch-kopf':

   119   instances  => 'instance_name'

   120 }

   121 ```

   122 ####From custom url

   123 ```puppet

   124 elasticsearch::plugin{ 'jetty':

   125   url        => 'https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip',

   126   instances  => 'instance_name'

   127 }

   128 ```

   129 

   130 

   131 ####Using a proxy

   132 You can also use a proxy if required by setting the `proxy_host` and `proxy_port` options:

   133 ```puppet

   134 elasticsearch::plugin { 'lmenezes/elasticsearch-kopf',

   135   instances  => 'instance_name',

   136   proxy_host => 'proxy.host.com',

   137   proxy_port => 3128

   138 }

   139 ```

   140 

   141 #####Plugin name could be:

   142 * `elasticsearch/plugin/version` for official elasticsearch plugins (download from download.elasticsearch.org)

   143 * `groupId/artifactId/version`   for community plugins (download from maven central or oss sonatype)

   144 * `username/repository`          for site plugins (download from github master)

   145 

   146 ####Upgrading plugins

   147 When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.

   148 

   149 ```puppet

   150 elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1':

   151 }

   152 ```

   153 

   154 And to upgrade, you would simply change it to

   155 

   156 ```puppet

   157 elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1':

   158 }

   159 ```

   160 

   161 Please note that this does not work when you specify 'latest' as a version number.

   162 

   163 ####ES 2.x official plugins

   164 For the Elasticsearch commercial plugins you can refer them to the simple name.

   165 

   166 See the [Plugin installation](https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html) for more details.

   167 

   168 ###Scripts

   169 

   170 Install [scripts](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) to be used by Elasticsearch.

   171 These scripts are shared across all defined instances on the same host.

   172 

   173 ```puppet

   174 elasticsearch::script { 'myscript':

   175   ensure => 'present',

   176   source => 'puppet:///path/to/my/script.groovy'

   177 }

   178 ```

   179 

   180 ###Templates

   181 

   182 #### Add a new template using a file

   183 

   184 This will install and/or replace the template in Elasticsearch:

   185 

   186 ```puppet

   187 elasticsearch::template { 'templatename':

   188   file => 'puppet:///path/to/template.json'

   189 }

   190 ```

   191 

   192 #### Add a new template using content

   193 

   194 This will install and/or replace the template in Elasticsearch:

   195 

   196 ```puppet

   197 elasticsearch::template { 'templatename':

   198   content => '{"template":"*","settings":{"number_of_replicas":0}}'

   199 }

   200 ```

   201 

   202 #### Delete a template

   203 

   204 ```puppet

   205 elasticsearch::template { 'templatename':

   206   ensure => 'absent'

   207 }

   208 ```

   209 

   210 #### Host

   211 

   212 By default it uses localhost:9200 as host. you can change this with the `host` and `port` variables

   213 

   214 ```puppet

   215 elasticsearch::template { 'templatename':

   216   host => $::ipaddress,

   217   port => 9200

   218 }

   219 ```

   220 

   221 ###Bindings / Clients

   222 

   223 Install a variety of [clients/bindings](http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/clients.html):

   224 

   225 ####Python

   226 

   227 ```puppet

   228 elasticsearch::python { 'rawes': }

   229 ```

   230 

   231 ####Ruby

   232 ```puppet

   233 elasticsearch::ruby { 'elasticsearch': }

   234 ```

   235 

   236 ###Connection Validator

   237 

   238 This module offers a way to make sure an instance has been started and is up and running before

   239 doing a next action. This is done via the use of the `es_instance_conn_validator` resource.

   240 ```puppet

   241 es_instance_conn_validator { 'myinstance' :

   242   server => 'es.example.com',

   243   port   => '9200',

   244 }

   245 ```

   246 

   247 A common use would be for example :

   248 

   249 ```puppet

   250 class { 'kibana4' :

   251   require => Es_Instance_Conn_Validator['myinstance'],

   252 }

   253 ```

   254 

   255 ###Package installation

   256 

   257 There are 2 different ways of installing the software

   258 

   259 ####Repository

   260 

   261 This option allows you to use an existing repository for package installation.

   262 The `repo_version` corresponds with the major version of Elasticsearch.

   263 

   264 ```puppet

   265 class { 'elasticsearch':

   266   manage_repo  => true,

   267   repo_version => '1.4',

   268 }

   269 ```

   270 

   271 ####Remote package source

   272 

   273 When a repository is not available or preferred you can install the packages from a remote source:

   274 

   275 #####http/https/ftp

   276 ```puppet

   277 class { 'elasticsearch':

   278   package_url       => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',

   279   proxy_url         => 'http://proxy.example.com:8080/',

   280 }

   281 ```

   282 Setting proxy_url to a location will enable download using the provided proxy

   283 server. This parameter is also used by elasticsearch::plugin. Setting the port

   284 in the proxy_url is mandatory. proxy_url defaults to undef (proxy disabled). 

   285 

   286 #####puppet://

   287 ```puppet

   288 class { 'elasticsearch':

   289   package_url => 'puppet:///path/to/elasticsearch-1.4.2.deb'

   290 }

   291 ```

   292 

   293 #####Local file

   294 ```puppet

   295 class { 'elasticsearch':

   296   package_url => 'file:/path/to/elasticsearch-1.4.2.deb'

   297 }

   298 ```

   299 

   300 ###Java installation

   301 

   302 Most sites will manage Java separately; however, this module can attempt to install Java as well.

   303 This is done by using the [puppetlabs-java](https://forge.puppetlabs.com/puppetlabs/java) module.

   304 

   305 ```puppet

   306 class { 'elasticsearch':

   307   java_install => true

   308 }

   309 ```

   310 

   311 Specify a particular Java package/version to be installed:

   312 

   313 ```puppet

   314 class { 'elasticsearch':

   315   java_install => true,

   316   java_package => 'packagename'

   317 }

   318 ```

   319 

   320 ###Service management

   321 

   322 Currently only the basic SysV-style [init](https://en.wikipedia.org/wiki/Init) and [Systemd](http://en.wikipedia.org/wiki/Systemd) service providers are supported, but other systems could be implemented as necessary (pull requests welcome).

   323 

   324 

   325 ####Defaults File

   326 

   327 The *defaults* file (`/etc/defaults/elasticsearch` or `/etc/sysconfig/elasticsearch`) for the Elasticsearch service can be populated as necessary. This can either be a static file resource or a simple key value-style  [hash](http://docs.puppetlabs.com/puppet/latest/reference/lang_datatypes.html#hashes) object, the latter being particularly well-suited to pulling out of a data source such as Hiera.

   328 

   329 #####file source

   330 ```puppet

   331 class { 'elasticsearch':

   332   init_defaults_file => 'puppet:///path/to/defaults'

   333 }

   334 ```

   335 #####hash representation

   336 ```puppet

   337 $config_hash = {

   338   'ES_USER' => 'elasticsearch',

   339   'ES_GROUP' => 'elasticsearch',

   340 }

   341 

   342 class { 'elasticsearch':

   343   init_defaults => $config_hash

   344 }

   345 ```

   346 

   347 Note: `init_defaults` hash can be passed to the main class and to the instance.

   348 

   349 ##Advanced features

   350 

   351 ###Package version pinning

   352 

   353 The module supports pinning the package version to avoid accidental upgrades that are not done by Puppet.

   354 To enable this feature:

   355 

   356 ```puppet

   357 class { 'elasticsearch':

   358   package_pin => true,

   359   version     => '1.5.2',

   360 }

   361 ```

   362 

   363 In this example we pin the package version to 1.5.2.

   364 

   365 

   366 ###Data directories

   367 

   368 There are 4 different ways of setting data directories for Elasticsearch.

   369 In every case the required configuration options are placed in the `elasticsearch.yml` file.

   370 

   371 ####Default

   372 By default we use:

   373 

   374 `/usr/share/elasticsearch/data/$instance_name`

   375 

   376 Which provides a data directory per instance.

   377 

   378 

   379 ####Single global data directory

   380 

   381 ```puppet

   382 class { 'elasticsearch':

   383   datadir => '/var/lib/elasticsearch-data'

   384 }

   385 ```

   386 Creates the following for each instance:

   387 

   388 `/var/lib/elasticsearch-data/$instance_name`

   389 

   390 ####Multiple Global data directories

   391 

   392 ```puppet

   393 class { 'elasticsearch':

   394   datadir => [ '/var/lib/es-data1', '/var/lib/es-data2']

   395 }

   396 ```

   397 Creates the following for each instance:

   398 `/var/lib/es-data1/$instance_name`

   399 and

   400 `/var/lib/es-data2/$instance_name`

   401 

   402 

   403 ####Single instance data directory

   404 

   405 ```puppet

   406 class { 'elasticsearch': }

   407 

   408 elasticsearch::instance { 'es-01':

   409   datadir => '/var/lib/es-data-es01'

   410 }

   411 ```

   412 Creates the following for this instance:

   413 `/var/lib/es-data-es01`

   414 

   415 ####Multiple instance data directories

   416 

   417 ```puppet

   418 class { 'elasticsearch': }

   419 

   420 elasticsearch::instance { 'es-01':

   421   datadir => ['/var/lib/es-data1-es01', '/var/lib/es-data2-es01']

   422 }

   423 ```

   424 Creates the following for this instance:

   425 `/var/lib/es-data1-es01`

   426 and

   427 `/var/lib/es-data2-es01`

   428 

   429 

   430 ###Main and instance configurations

   431 

   432 The `config` option in both the main class and the instances can be configured to work together.

   433 

   434 The options in the `instance` config hash will merged with the ones from the main class and override any duplicates.

   435 

   436 #### Simple merging

   437 

   438 ```puppet

   439 class { 'elasticsearch':

   440   config => { 'cluster.name' => 'clustername' }

   441 }

   442 

   443 elasticsearch::instance { 'es-01':

   444   config => { 'node.name' => 'nodename' }

   445 }

   446 elasticsearch::instance { 'es-02':

   447   config => { 'node.name' => 'nodename2' }

   448 }

   449 

   450 ```

   451 

   452 This example merges the `cluster.name` together with the `node.name` option.

   453 

   454 #### Overriding

   455 

   456 When duplicate options are provided, the option in the instance config overrides the ones from the main class.

   457 

   458 ```puppet

   459 class { 'elasticsearch':

   460   config => { 'cluster.name' => 'clustername' }

   461 }

   462 

   463 elasticsearch::instance { 'es-01':

   464   config => { 'node.name' => 'nodename', 'cluster.name' => 'otherclustername' }

   465 }

   466 

   467 elasticsearch::instance { 'es-02':

   468   config => { 'node.name' => 'nodename2' }

   469 }

   470 ```

   471 

   472 This will set the cluster name to `otherclustername` for the instance `es-01` but will keep it to `clustername` for instance `es-02`

   473 

   474 ####Configuration writeup

   475 

   476 The `config` hash can be written in 2 different ways:

   477 

   478 ##### Full hash writeup

   479 

   480 Instead of writing the full hash representation:

   481 ```puppet

   482 class { 'elasticsearch':

   483   config                 => {

   484    'cluster'             => {

   485      'name'              => 'ClusterName',

   486      'routing'           => {

   487         'allocation'     => {

   488           'awareness'    => {

   489             'attributes' => 'rack'

   490           }

   491         }

   492       }

   493     }

   494   }

   495 }

   496 ```

   497 ##### Short hash writeup

   498 ```puppet

   499 class { 'elasticsearch':

   500   config => {

   501     'cluster' => {

   502       'name' => 'ClusterName',

   503       'routing.allocation.awareness.attributes' => 'rack'

   504     }

   505   }

   506 }

   507 ```

   508 

   509 

   510 ##Limitations

   511 

   512 This module has been built on and tested against Puppet 3.2 and higher.

   513 

   514 The module has been tested on:

   515 

   516 * Debian 6/7/8

   517 * CentOS 6/7

   518 * Ubuntu 12.04, 14.04

   519 * OpenSuSE 13.x

   520 

   521 Other distro's that have been reported to work:

   522 

   523 * RHEL 6

   524 * OracleLinux 6

   525 * Scientific 6

   526 

   527 Testing on other platforms has been light and cannot be guaranteed.

   528 

   529 ##Development

   530 

   531 

   532 ##Support

   533 

   534 Need help? Join us in [#elasticsearch](https://webchat.freenode.net?channels=%23elasticsearch) on Freenode IRC or on the [discussion forum](https://discuss.elastic.co/).