# Elasticsearch Puppet Module

[![Build Status](https://travis-ci.org/elastic/puppet-elasticsearch.svg?branch=master)](https://travis-ci.org/elastic/puppet-elasticsearch)

[![Puppet Forge endorsed](https://img.shields.io/puppetforge/e/elasticsearch/elasticsearch.svg)](https://forge.puppetlabs.com/elasticsearch/elasticsearch)

[![Puppet Forge Version](https://img.shields.io/puppetforge/v/elasticsearch/elasticsearch.svg)](https://forge.puppetlabs.com/elasticsearch/elasticsearch)

[![Puppet Forge Downloads](https://img.shields.io/puppetforge/dt/elasticsearch/elasticsearch.svg)](https://forge.puppetlabs.com/elasticsearch/elasticsearch)

#### Table of Contents

1. [Module description - What the module does and why it is useful](#module-description)

2. [Setup - The basics of getting started with Elasticsearch](#setup)

  * [The module manages the following](#the-module-manages-the-following)

  * [Requirements](#requirements)

3. [Usage - Configuration options and additional functionality](#usage)

4. [Advanced features - Extra information on advanced usage](#advanced-features)

5. [Reference - An under-the-hood peek at what the module is doing and how](#reference)

6. [Limitations - OS compatibility, etc.](#limitations)

7. [Development - Guide for contributing to the module](#development)

8. [Support - When you need help with this module](#support)

## Module description

This module sets up [Elasticsearch](https://www.elastic.co/overview/elasticsearch/) instances with additional resource for plugins, templates, and more.

This module has been tested against all versions of ES 1.x and 2.x.

## Setup

### The module manages the following

* Elasticsearch repository files.

* Elasticsearch package.

* Elasticsearch configuration file.

* Elasticsearch service.

* Elasticsearch plugins.

* Elasticsearch templates.

* Elasticsearch Shield users, roles, and certificates.

### Requirements

* The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) Puppet library.

* [ceritsc/yum](https://forge.puppetlabs.com/ceritsc/yum) For yum version lock.

* [richardc/datacat](https://forge.puppetlabs.com/richardc/datacat)

* [Augeas](http://augeas.net/)

* [puppetlabs-java](https://forge.puppetlabs.com/puppetlabs/java) for Java installation (optional).

* [puppetlabs-java_ks](https://forge.puppetlabs.com/puppetlabs/java_ks) for Shield certificate management (optional).

#### Repository management

When using the repository management, the following module dependencies are required:

* Debian/Ubuntu: [Puppetlabs/apt](http://forge.puppetlabs.com/puppetlabs/apt)

* OpenSuSE/SLES: [Darin/zypprepo](https://forge.puppetlabs.com/darin/zypprepo)

### Beginning with Elasticsearch

Declare the top-level `elasticsearch` class (managing repositories) and set up an instance:

```puppet

class { 'elasticsearch':

  java_install => true,

  manage_repo  => true,

  repo_version => '2.x',

}

elasticsearch::instance { 'es-01': }

```

## Usage

### Main class

Most top-level parameters in the `elasticsearch` class are set to reasonable defaults.

The following are some parameters that may be useful to override:

#### Install a specific version

```puppet

class { 'elasticsearch':

  version => '1.4.2'

}

```

Note: This will only work when using the repository.

#### Automatically restarting the service (default set to false)

By default, the module will not restart Elasticsearch when the configuration file, package, or plugins change.

This can be overridden globally with the following option:

```puppet

class { 'elasticsearch':

  restart_on_change => true

}

```

Or controlled with the more granular options: `restart_config_change`, `restart_package_change`, and `restart_plugin_change.`

#### Automatic upgrades (default set to false)

```puppet

class { 'elasticsearch':

  autoupgrade => true

}

```

#### Removal/Decommissioning

```puppet

class { 'elasticsearch':

  ensure => 'absent'

}

```

#### Install everything but disable service(s) afterwards

```puppet

class { 'elasticsearch':

  status => 'disabled'

}

```

#### API Settings

Some resources, such as `elasticsearch::template`, require communicating with the Elasticsearch REST API.

By default, these API settings are set to:

```puppet

class { 'elasticsearch':

  api_protocol            => 'http',

  api_host                => 'localhost',

  api_port                => 9200,

  api_timeout             => 10,

  api_basic_auth_username => undef,

  api_basic_auth_password => undef,

  api_ca_file             => undef,

  api_ca_path             => undef,

  validate_tls            => true,

}

```

Each of these can be set at the top-level `elasticsearch` class and inherited for each resource or overridden on a per-resource basis.

### Instances

This module works with the concept of instances. For service to start you need to specify at least one instance.

#### Quick setup

```puppet

elasticsearch::instance { 'es-01': }

```

This will set up its own data directory and set the node name to `$hostname-$instance_name`

#### Advanced options

Instance specific options can be given:

```puppet

elasticsearch::instance { 'es-01':

  config        => { }, # Configuration hash

  init_defaults => { }, # Init defaults hash

  datadir       => [ ], # Data directory

}

```

See [Advanced features](#advanced-features) for more information.

### Plugins

This module can help manage [a variety of plugins](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html#known-plugins).

Note that `module_dir` is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.

#### From an official repository

```puppet

elasticsearch::plugin { 'lmenezes/elasticsearch-kopf':

  instances => 'instance_name'

}

```

#### From a custom url

```puppet

elasticsearch::plugin { 'jetty':

  url        => 'https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip',

  instances  => 'instance_name'

}

```

#### Using a proxy

You can also use a proxy if required by setting the `proxy_host` and `proxy_port` options:

```puppet

elasticsearch::plugin { 'lmenezes/elasticsearch-kopf',

  instances  => 'instance_name',

  proxy_host => 'proxy.host.com',

  proxy_port => 3128

}

```

Proxies that require usernames and passwords are similarly supported with the `proxy_username` and `proxy_password` parameters.

Plugin name formats that are supported include:

* `elasticsearch/plugin/version` (for official elasticsearch plugins downloaded from download.elastic.co)

* `groupId/artifactId/version` (for community plugins downloaded from maven central or OSS Sonatype)

* `username/repository` (for site plugins downloaded from github master)

#### Upgrading plugins

When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.

```puppet

elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1': }

```

And to upgrade, you would simply change it to

```puppet

elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1': }

```

Please note that this does not work when you specify 'latest' as a version number.

#### ES 2.x official plugins

For the Elasticsearch commercial plugins you can refer them to the simple name.

See [Plugin installation](https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html) for more details.

### Scripts

Installs [scripts](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) to be used by Elasticsearch.

These scripts are shared across all defined instances on the same host.

```puppet

elasticsearch::script { 'myscript':

  ensure => 'present',

  source => 'puppet:///path/to/my/script.groovy'

}

```

### Templates

By default templates use the top-level `elasticsearch::api_*` settings to communicate with Elasticsearch.

The following is an example of how to override these settings:

```puppet

elasticsearch::template { 'templatename':

  api_protocol            => 'https',

  api_host                => $::ipaddress,

  api_port                => 9201,

  api_timeout             => 60,

  api_basic_auth_username => 'admin',

  api_basic_auth_password => 'adminpassword',

  api_ca_file             => '/etc/ssl/certs',

  api_ca_path             => '/etc/pki/certs',

  validate_tls            => false,

  source                  => 'puppet:///path/to/template.json',

}

```

#### Add a new template using a file

This will install and/or replace the template in Elasticsearch:

```puppet

elasticsearch::template { 'templatename':

  source => 'puppet:///path/to/template.json',

}

```

#### Add a new template using content

This will install and/or replace the template in Elasticsearch:

```puppet

elasticsearch::template { 'templatename':

  content => {

    'template' => "*",

    'settings' => {

      'number_of_replicas' => 0

    }

  }

}

```

Plain JSON strings are also supported.

```puppet

elasticsearch::template { 'templatename':

  content => '{"template":"*","settings":{"number_of_replicas":0}}'

}

```

#### Delete a template

```puppet

elasticsearch::template { 'templatename':

  ensure => 'absent'

}

```

### Bindings/Clients

Install a variety of [clients/bindings](http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/clients.html):

#### Python

```puppet

elasticsearch::python { 'rawes': }

```

#### Ruby

```puppet

elasticsearch::ruby { 'elasticsearch': }

```

### Connection Validator

This module offers a way to make sure an instance has been started and is up and running before

doing a next action. This is done via the use of the `es_instance_conn_validator` resource.

```puppet

es_instance_conn_validator { 'myinstance' :

  server => 'es.example.com',

  port   => '9200',

}

```

A common use would be for example :

```puppet

class { 'kibana4' :

  require => Es_Instance_Conn_Validator['myinstance'],

}

```

### Package installation

There are two different ways of installing Elasticsearch:

#### Repository

This option allows you to use an existing repository for package installation.

The `repo_version` corresponds with the `major.minor` version of Elasticsearch for versions before 2.x.

```puppet

class { 'elasticsearch':

  manage_repo  => true,

  repo_version => '1.4',

}

```

For 2.x versions of Elasticsearch, use `repo_version => '2.x'`.

```puppet

class { 'elasticsearch':

  manage_repo  => true,

  repo_version => '2.x',

}

```

#### Remote package source

When a repository is not available or preferred you can install the packages from a remote source:

##### http/https/ftp

```puppet

class { 'elasticsearch':

  package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',

  proxy_url   => 'http://proxy.example.com:8080/',

}

```

Setting `proxy_url` to a location will enable download using the provided proxy

server.

This parameter is also used by `elasticsearch::plugin`.

Setting the port in the `proxy_url` is mandatory.

`proxy_url` defaults to `undef` (proxy disabled).

##### puppet://

```puppet

class { 'elasticsearch':

  package_url => 'puppet:///path/to/elasticsearch-1.4.2.deb'

}

```

##### Local file

```puppet

class { 'elasticsearch':

  package_url => 'file:/path/to/elasticsearch-1.4.2.deb'

}

```

### Java installation

Most sites will manage Java separately; however, this module can attempt to install Java as well.

This is done by using the [puppetlabs-java](https://forge.puppetlabs.com/puppetlabs/java) module.

```puppet

class { 'elasticsearch':

  java_install => true

}

```

Specify a particular Java package/version to be installed:

```puppet

class { 'elasticsearch':

  java_install => true,

  java_package => 'packagename'

}

```

### Service management

Currently only the basic SysV-style [init](https://en.wikipedia.org/wiki/Init) and [Systemd](http://en.wikipedia.org/wiki/Systemd) service providers are supported, but other systems could be implemented as necessary (pull requests welcome).

#### Defaults File

The *defaults* file (`/etc/defaults/elasticsearch` or `/etc/sysconfig/elasticsearch`) for the Elasticsearch service can be populated as necessary.

This can either be a static file resource or a simple key value-style  [hash](http://docs.puppetlabs.com/puppet/latest/reference/lang_datatypes.html#hashes) object, the latter being particularly well-suited to pulling out of a data source such as Hiera.

##### File source

```puppet

class { 'elasticsearch':

  init_defaults_file => 'puppet:///path/to/defaults'

}

```

##### Hash representation

```puppet

$config_hash = {

  'ES_HEAP_SIZE' => '30g',

}

class { 'elasticsearch':

  init_defaults => $config_hash