--- a/dev/provisioning/modules/elasticsearch/README.md Tue Nov 08 18:23:01 2016 +0100
+++ b/dev/provisioning/modules/elasticsearch/README.md Wed Nov 09 15:05:41 2016 +0100
@@ -1,33 +1,32 @@
-#Elasticsearch Puppet module
-
-####Table of Contents
+# Elasticsearch Puppet Module
-1. [Overview](#overview)
-2. [Module description - What the module does and why it is useful](#module-description)
-3. [Setup - The basics of getting started with Elasticsearch](#setup)
+[](https://travis-ci.org/elastic/puppet-elasticsearch)
+[](https://forge.puppetlabs.com/elasticsearch/elasticsearch)
+[](https://forge.puppetlabs.com/elasticsearch/elasticsearch)
+[](https://forge.puppetlabs.com/elasticsearch/elasticsearch)
+
+#### Table of Contents
+
+1. [Module description - What the module does and why it is useful](#module-description)
+2. [Setup - The basics of getting started with Elasticsearch](#setup)
* [The module manages the following](#the-module-manages-the-following)
* [Requirements](#requirements)
-4. [Usage - Configuration options and additional functionality](#usage)
-5. [Advanced features - Extra information on advanced usage](#advanced-features)
+3. [Usage - Configuration options and additional functionality](#usage)
+4. [Advanced features - Extra information on advanced usage](#advanced-features)
+5. [Reference - An under-the-hood peek at what the module is doing and how](#reference)
6. [Limitations - OS compatibility, etc.](#limitations)
7. [Development - Guide for contributing to the module](#development)
8. [Support - When you need help with this module](#support)
-
-
-##Overview
+## Module description
-This module manages Elasticsearch (http://www.elasticsearch.org/overview/elasticsearch/)
-
-##Module description
+This module sets up [Elasticsearch](https://www.elastic.co/overview/elasticsearch/) instances with additional resource for plugins, templates, and more.
-The elasticsearch module sets up Elasticsearch instances and can manage plugins and templates.
-
-This module has been tested against all versions of ES 1.x and 2.x
+This module has been tested against all versions of ES 1.x and 2.x.
-##Setup
+## Setup
-###The module manages the following
+### The module manages the following
* Elasticsearch repository files.
* Elasticsearch package.
@@ -35,25 +34,46 @@
* Elasticsearch service.
* Elasticsearch plugins.
* Elasticsearch templates.
+* Elasticsearch Shield users, roles, and certificates.
-###Requirements
+### Requirements
* The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) Puppet library.
* [ceritsc/yum](https://forge.puppetlabs.com/ceritsc/yum) For yum version lock.
* [richardc/datacat](https://forge.puppetlabs.com/richardc/datacat)
* [Augeas](http://augeas.net/)
+* [puppetlabs-java](https://forge.puppetlabs.com/puppetlabs/java) for Java installation (optional).
+* [puppetlabs-java_ks](https://forge.puppetlabs.com/puppetlabs/java_ks) for Shield certificate management (optional).
#### Repository management
-When using the repository management you will need the following dependency modules:
+
+When using the repository management, the following module dependencies are required:
* Debian/Ubuntu: [Puppetlabs/apt](http://forge.puppetlabs.com/puppetlabs/apt)
-* OpenSuSE: [Darin/zypprepo](https://forge.puppetlabs.com/darin/zypprepo)
+* OpenSuSE/SLES: [Darin/zypprepo](https://forge.puppetlabs.com/darin/zypprepo)
-##Usage
+### Beginning with Elasticsearch
+
+Declare the top-level `elasticsearch` class (managing repositories) and set up an instance:
-###Main class
+```puppet
+class { 'elasticsearch':
+ java_install => true,
+ manage_repo => true,
+ repo_version => '2.x',
+}
-####Install a specific version
+elasticsearch::instance { 'es-01': }
+```
+
+## Usage
+
+### Main class
+
+Most top-level parameters in the `elasticsearch` class are set to reasonable defaults.
+The following are some parameters that may be useful to override:
+
+#### Install a specific version
```puppet
class { 'elasticsearch':
@@ -63,72 +83,114 @@
Note: This will only work when using the repository.
-####Automatic upgrade of the software ( default set to false )
+#### Automatically restarting the service (default set to false)
+
+By default, the module will not restart Elasticsearch when the configuration file, package, or plugins change.
+This can be overridden globally with the following option:
+
+```puppet
+class { 'elasticsearch':
+ restart_on_change => true
+}
+```
+
+Or controlled with the more granular options: `restart_config_change`, `restart_package_change`, and `restart_plugin_change.`
+
+#### Automatic upgrades (default set to false)
+
```puppet
class { 'elasticsearch':
autoupgrade => true
}
```
-####Removal/decommissioning
+#### Removal/Decommissioning
+
```puppet
class { 'elasticsearch':
ensure => 'absent'
}
```
-####Install everything but disable service(s) afterwards
+#### Install everything but disable service(s) afterwards
+
```puppet
class { 'elasticsearch':
status => 'disabled'
}
```
-###Instances
+#### API Settings
+
+Some resources, such as `elasticsearch::template`, require communicating with the Elasticsearch REST API.
+By default, these API settings are set to:
+
+```puppet
+class { 'elasticsearch':
+ api_protocol => 'http',
+ api_host => 'localhost',
+ api_port => 9200,
+ api_timeout => 10,
+ api_basic_auth_username => undef,
+ api_basic_auth_password => undef,
+ api_ca_file => undef,
+ api_ca_path => undef,
+ validate_tls => true,
+}
+```
+
+Each of these can be set at the top-level `elasticsearch` class and inherited for each resource or overridden on a per-resource basis.
+
+### Instances
This module works with the concept of instances. For service to start you need to specify at least one instance.
-####Quick setup
+#### Quick setup
+
```puppet
elasticsearch::instance { 'es-01': }
```
This will set up its own data directory and set the node name to `$hostname-$instance_name`
-####Advanced options
+#### Advanced options
Instance specific options can be given:
```puppet
elasticsearch::instance { 'es-01':
- config => { }, # Configuration hash
+ config => { }, # Configuration hash
init_defaults => { }, # Init defaults hash
- datadir => [ ], # Data directory
+ datadir => [ ], # Data directory
}
```
-See [Advanced features](#advanced-features) for more information
+See [Advanced features](#advanced-features) for more information.
-###Plug-ins
+### Plugins
-Install [a variety of plugins](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html#known-plugins). Note that `module_dir` is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.
+This module can help manage [a variety of plugins](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html#known-plugins).
+Note that `module_dir` is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.
-####From official repository
+#### From an official repository
+
```puppet
-elasticsearch::plugin{'lmenezes/elasticsearch-kopf':
- instances => 'instance_name'
+elasticsearch::plugin { 'lmenezes/elasticsearch-kopf':
+ instances => 'instance_name'
}
```
-####From custom url
+
+#### From a custom url
+
```puppet
-elasticsearch::plugin{ 'jetty':
+elasticsearch::plugin { 'jetty':
url => 'https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip',
instances => 'instance_name'
}
```
+#### Using a proxy
-####Using a proxy
You can also use a proxy if required by setting the `proxy_host` and `proxy_port` options:
```puppet
elasticsearch::plugin { 'lmenezes/elasticsearch-kopf',
@@ -138,36 +200,38 @@
}
```
-#####Plugin name could be:
-* `elasticsearch/plugin/version` for official elasticsearch plugins (download from download.elasticsearch.org)
-* `groupId/artifactId/version` for community plugins (download from maven central or oss sonatype)
-* `username/repository` for site plugins (download from github master)
+Proxies that require usernames and passwords are similarly supported with the `proxy_username` and `proxy_password` parameters.
+
+Plugin name formats that are supported include:
-####Upgrading plugins
+* `elasticsearch/plugin/version` (for official elasticsearch plugins downloaded from download.elastic.co)
+* `groupId/artifactId/version` (for community plugins downloaded from maven central or OSS Sonatype)
+* `username/repository` (for site plugins downloaded from github master)
+
+#### Upgrading plugins
+
When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.
```puppet
-elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1':
-}
+elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1': }
```
And to upgrade, you would simply change it to
```puppet
-elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1':
-}
+elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1': }
```
Please note that this does not work when you specify 'latest' as a version number.
-####ES 2.x official plugins
+#### ES 2.x official plugins
For the Elasticsearch commercial plugins you can refer them to the simple name.
-See the [Plugin installation](https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html) for more details.
+See [Plugin installation](https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html) for more details.
-###Scripts
+### Scripts
-Install [scripts](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) to be used by Elasticsearch.
+Installs [scripts](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) to be used by Elasticsearch.
These scripts are shared across all defined instances on the same host.
```puppet
@@ -177,7 +241,25 @@
}
```
-###Templates
+### Templates
+
+By default templates use the top-level `elasticsearch::api_*` settings to communicate with Elasticsearch.
+The following is an example of how to override these settings:
+
+```puppet
+elasticsearch::template { 'templatename':
+ api_protocol => 'https',
+ api_host => $::ipaddress,
+ api_port => 9201,
+ api_timeout => 60,
+ api_basic_auth_username => 'admin',
+ api_basic_auth_password => 'adminpassword',
+ api_ca_file => '/etc/ssl/certs',
+ api_ca_path => '/etc/pki/certs',
+ validate_tls => false,
+ source => 'puppet:///path/to/template.json',
+}
+```
#### Add a new template using a file
@@ -185,7 +267,7 @@
```puppet
elasticsearch::template { 'templatename':
- file => 'puppet:///path/to/template.json'
+ source => 'puppet:///path/to/template.json',
}
```
@@ -195,6 +277,19 @@
```puppet
elasticsearch::template { 'templatename':
+ content => {
+ 'template' => "*",
+ 'settings' => {
+ 'number_of_replicas' => 0
+ }
+ }
+}
+```
+
+Plain JSON strings are also supported.
+
+```puppet
+elasticsearch::template { 'templatename':
content => '{"template":"*","settings":{"number_of_replicas":0}}'
}
```
@@ -207,33 +302,23 @@
}
```
-#### Host
-
-By default it uses localhost:9200 as host. you can change this with the `host` and `port` variables
-
-```puppet
-elasticsearch::template { 'templatename':
- host => $::ipaddress,
- port => 9200
-}
-```
-
-###Bindings / Clients
+### Bindings/Clients
Install a variety of [clients/bindings](http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/clients.html):
-####Python
+#### Python
```puppet
elasticsearch::python { 'rawes': }
```
-####Ruby
+#### Ruby
+
```puppet
elasticsearch::ruby { 'elasticsearch': }
```
-###Connection Validator
+### Connection Validator
This module offers a way to make sure an instance has been started and is up and running before
doing a next action. This is done via the use of the `es_instance_conn_validator` resource.
@@ -252,14 +337,14 @@
}
```
-###Package installation
+### Package installation
-There are 2 different ways of installing the software
+There are two different ways of installing Elasticsearch:
-####Repository
+#### Repository
This option allows you to use an existing repository for package installation.
-The `repo_version` corresponds with the major version of Elasticsearch.
+The `repo_version` corresponds with the `major.minor` version of Elasticsearch for versions before 2.x.
```puppet
class { 'elasticsearch':
@@ -268,36 +353,50 @@
}
```
-####Remote package source
+For 2.x versions of Elasticsearch, use `repo_version => '2.x'`.
+
+```puppet
+class { 'elasticsearch':
+ manage_repo => true,
+ repo_version => '2.x',
+}
+```
+
+#### Remote package source
When a repository is not available or preferred you can install the packages from a remote source:
-#####http/https/ftp
+##### http/https/ftp
+
```puppet
class { 'elasticsearch':
- package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',
- proxy_url => 'http://proxy.example.com:8080/',
+ package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',
+ proxy_url => 'http://proxy.example.com:8080/',
}
```
-Setting proxy_url to a location will enable download using the provided proxy
-server. This parameter is also used by elasticsearch::plugin. Setting the port
-in the proxy_url is mandatory. proxy_url defaults to undef (proxy disabled).
-#####puppet://
+Setting `proxy_url` to a location will enable download using the provided proxy
+server.
+This parameter is also used by `elasticsearch::plugin`.
+Setting the port in the `proxy_url` is mandatory.
+`proxy_url` defaults to `undef` (proxy disabled).
+
+##### puppet://
```puppet
class { 'elasticsearch':
package_url => 'puppet:///path/to/elasticsearch-1.4.2.deb'
}
```
-#####Local file
+##### Local file
+
```puppet
class { 'elasticsearch':
package_url => 'file:/path/to/elasticsearch-1.4.2.deb'
}
```
-###Java installation
+### Java installation
Most sites will manage Java separately; however, this module can attempt to install Java as well.
This is done by using the [puppetlabs-java](https://forge.puppetlabs.com/puppetlabs/java) module.
@@ -317,26 +416,27 @@
}
```
-###Service management
+### Service management
Currently only the basic SysV-style [init](https://en.wikipedia.org/wiki/Init) and [Systemd](http://en.wikipedia.org/wiki/Systemd) service providers are supported, but other systems could be implemented as necessary (pull requests welcome).
-
-####Defaults File
+#### Defaults File
-The *defaults* file (`/etc/defaults/elasticsearch` or `/etc/sysconfig/elasticsearch`) for the Elasticsearch service can be populated as necessary. This can either be a static file resource or a simple key value-style [hash](http://docs.puppetlabs.com/puppet/latest/reference/lang_datatypes.html#hashes) object, the latter being particularly well-suited to pulling out of a data source such as Hiera.
+The *defaults* file (`/etc/defaults/elasticsearch` or `/etc/sysconfig/elasticsearch`) for the Elasticsearch service can be populated as necessary.
+This can either be a static file resource or a simple key value-style [hash](http://docs.puppetlabs.com/puppet/latest/reference/lang_datatypes.html#hashes) object, the latter being particularly well-suited to pulling out of a data source such as Hiera.
-#####file source
+##### File source
+
```puppet
class { 'elasticsearch':
init_defaults_file => 'puppet:///path/to/defaults'
}
```
-#####hash representation
+##### Hash representation
+
```puppet
$config_hash = {
- 'ES_USER' => 'elasticsearch',
- 'ES_GROUP' => 'elasticsearch',
+ 'ES_HEAP_SIZE' => '30g',
}
class { 'elasticsearch':
@@ -346,9 +446,169 @@
Note: `init_defaults` hash can be passed to the main class and to the instance.
-##Advanced features
+## Advanced features
+
+### Shield
+
+[Shield](https://www.elastic.co/products/shield) users, roles, and certificates can be managed by this module.
+
+**Note**: If you are planning to use these features, it is *highly recommended* you read the following documentation to understand the caveats and extent of the resources available to you.
+
+#### Getting Started
+
+Although this module can handle several types of Shield resources, you are expected to manage the plugin installation and versions for your deployment.
+For example, the following manifest will install Elasticseach with a single instance running shield:
+
+```puppet
+class { 'elasticsearch':
+ java_install => true,
+ manage_repo => true,
+ repo_version => '1.7',
+}
+
+elasticsearch::instance { 'es-01': }
+
+Elasticsearch::Plugin { instances => ['es-01'], }
+elasticsearch::plugin { 'elasticsearch/license/latest': }
+elasticsearch::plugin { 'elasticsearch/shield/latest': }
+```
+
+The following examples will assume the preceding resources are part of your puppet manifest.
+
+#### Roles
+
+Roles in the `esusers` realm can be managed using the `elasticsearch::shield::role` type.
+For example, to create a role called `myrole`, you could use the following resource:
+
+```puppet
+elasticsearch::shield::role { 'myrole':
+ privileges => {
+ 'cluster' => 'monitor',
+ 'indices' => {
+ '*' => 'read'
+ }
+ }
+}
+```
+
+This role would grant users access to cluster monitoring and read access to all indices.
+See the [Shield documentation](https://www.elastic.co/guide/en/shield/index.html) for your version to determine what `privileges` to use and how to format them (the Puppet hash representation will simply be translated into yaml.)
+
+**Note**: The Puppet provider for `esusers` has fine-grained control over the `roles.yml` file and thus will leave the default roles Shield installs in-place.
+If you would like to explicitly purge the default roles (leaving only roles managed by puppet), you can do so by including the following in your manifest:
+
+```puppet
+resources { 'elasticsearch_shield_role':
+ purge => true,
+}
+```
+
+##### Mappings
+
+Associating mappings with a role is done by passing an array of strings to the `mappings` parameter of the `elasticsearch::shield::role` type.
+For example, to define a role with mappings using Shield >= 2.3.x style role definitions:
+
+```puppet
+elasticsearch::shield::role { 'logstash':
+ mappings => [
+ 'cn=group,ou=devteam',
+ ],
+ privileges => {
+ 'cluster' => 'manage_index_templates',
+ 'indices' => [{
+ 'names' => ['logstash-*'],
+ 'privileges' => [
+ 'write',
+ 'delete',
+ 'create_index',
+ ],
+ }],
+ },
+}
+```
-###Package version pinning
+**Note**: Observe the brackets around `indices` in the preceding role definition; which is an array of hashes per the format in Shield 2.3.x. Follow the documentation to determine the correct formatting for your version of Shield.
+
+If you'd like to keep the mappings file purged of entries not under Puppet's control, you should use the following `resources` declaration because mappings are a separate low-level type:
+
+```puppet
+resources { 'elasticsearch_shield_role_mapping':
+ purge => true,
+}
+```
+
+#### Users
+
+Users can be managed using the `elasticsearch::shield::user` type.
+For example, to create a user `mysuser` with membership in `myrole`:
+
+```puppet
+elasticsearch::shield::user { 'myuser':
+ password => 'mypassword',
+ roles => ['myrole'],
+}
+```
+
+The `password` parameter will also accept password hashes generated from the `esusers` utility and ensure the password is kept in-sync with the Shield `users` file for all Elasticsearch instances.
+
+```puppet
+elasticsearch::shield::user { 'myuser':
+ password => '$2a$10$IZMnq6DF4DtQ9c4sVovgDubCbdeH62XncmcyD1sZ4WClzFuAdqspy',
+ roles => ['myrole'],
+}
+```
+
+**Note**: When using the `esusers` provider (the default for plaintext passwords), Puppet has no way to determine whether the given password is in-sync with the password hashed by Shield.
+In order to work around this, the `elasticsearch::shield::user` resource has been designed to accept refresh events in order to update password values.
+This is not ideal, but allows you to instruct the resource to change the password when needed.
+For example, to update the aforementioned user's password, you could include the following your manifest:
+
+```puppet
+notify { 'update password': } ~>
+elasticsearch::shield::user { 'myuser':
+ password => 'mynewpassword',
+ roles => ['myrole'],
+}
+```
+
+#### Certificates
+
+SSL/TLS can be enabled by providing an `elasticsearch::instance` type with paths to the certificate and private key files, and a password for the keystore.
+
+```puppet
+elasticsearch::instance { 'es-01':
+ ssl => true,
+ ca_certificate => '/path/to/ca.pem',
+ certificate => '/path/to/cert.pem',
+ private_key => '/path/to/key.pem',
+ keystore_password => 'keystorepassword',
+}
+```
+
+**Note**: Setting up a proper CA and certificate infrastructure is outside the scope of this documentation, see the aforementioned Shield guide for more information regarding the generation of these certificate files.
+
+The module will set up a keystore file for the node to use and set the relevant options in `elasticsearch.yml` to enable TLS/SSL using the certificates and key provided.
+
+#### System Keys
+
+Shield system keys can be passed to the module, where they will be placed into individual instance configuration directories.
+This can be set at the `elasticsearch` class and inherited across all instances:
+
+```puppet
+class { 'elasticsearch':
+ system_key => 'puppet:///path/to/key',
+}
+```
+
+Or set on a per-instance basis:
+
+```puppet
+elasticsearch::instance { 'es-01':
+ system_key => '/local/path/to/key',
+}
+```
+
+### Package version pinning
The module supports pinning the package version to avoid accidental upgrades that are not done by Puppet.
To enable this feature:
@@ -362,32 +622,31 @@
In this example we pin the package version to 1.5.2.
-
-###Data directories
+### Data directories
There are 4 different ways of setting data directories for Elasticsearch.
In every case the required configuration options are placed in the `elasticsearch.yml` file.
-####Default
+#### Default
By default we use:
-`/usr/share/elasticsearch/data/$instance_name`
+ /usr/share/elasticsearch/data/$instance_name
Which provides a data directory per instance.
-
-####Single global data directory
+#### Single global data directory
```puppet
class { 'elasticsearch':
datadir => '/var/lib/elasticsearch-data'
}
```
+
Creates the following for each instance:
-`/var/lib/elasticsearch-data/$instance_name`
+ /var/lib/elasticsearch-data/$instance_name
-####Multiple Global data directories
+#### Multiple Global data directories
```puppet
class { 'elasticsearch':
@@ -397,10 +656,9 @@
Creates the following for each instance:
`/var/lib/es-data1/$instance_name`
and
-`/var/lib/es-data2/$instance_name`
+`/var/lib/es-data2/$instance_name`.
-
-####Single instance data directory
+#### Single instance data directory
```puppet
class { 'elasticsearch': }
@@ -409,10 +667,12 @@
datadir => '/var/lib/es-data-es01'
}
```
+
Creates the following for this instance:
-`/var/lib/es-data-es01`
-####Multiple instance data directories
+ /var/lib/es-data-es01
+
+#### Multiple instance data directories
```puppet
class { 'elasticsearch': }
@@ -421,13 +681,14 @@
datadir => ['/var/lib/es-data1-es01', '/var/lib/es-data2-es01']
}
```
+
Creates the following for this instance:
`/var/lib/es-data1-es01`
and
-`/var/lib/es-data2-es01`
+`/var/lib/es-data2-es01`.
-###Main and instance configurations
+### Main and instance configurations
The `config` option in both the main class and the instances can be configured to work together.
@@ -446,7 +707,6 @@
elasticsearch::instance { 'es-02':
config => { 'node.name' => 'nodename2' }
}
-
```
This example merges the `cluster.name` together with the `node.name` option.
@@ -471,13 +731,14 @@
This will set the cluster name to `otherclustername` for the instance `es-01` but will keep it to `clustername` for instance `es-02`
-####Configuration writeup
+#### Configuration writeup
The `config` hash can be written in 2 different ways:
##### Full hash writeup
Instead of writing the full hash representation:
+
```puppet
class { 'elasticsearch':
config => {
@@ -494,7 +755,9 @@
}
}
```
+
##### Short hash writeup
+
```puppet
class { 'elasticsearch':
config => {
@@ -506,8 +769,7 @@
}
```
-
-##Limitations
+## Limitations
This module has been built on and tested against Puppet 3.2 and higher.
@@ -515,20 +777,22 @@
* Debian 6/7/8
* CentOS 6/7
+* OracleLinux 6/7
* Ubuntu 12.04, 14.04
* OpenSuSE 13.x
+* SLES 12
Other distro's that have been reported to work:
* RHEL 6
-* OracleLinux 6
* Scientific 6
Testing on other platforms has been light and cannot be guaranteed.
-##Development
+## Development
+Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for instructions regarding development environments and testing.
-##Support
+## Support
Need help? Join us in [#elasticsearch](https://webchat.freenode.net?channels=%23elasticsearch) on Freenode IRC or on the [discussion forum](https://discuss.elastic.co/).