MK

Salt Reclass with Remote FS

reclass is an “external node classifier” (ENC) as can be used with automation tools, such as Puppet, Salt, and Ansible. It is also a stand-alone tool for merging data sources recursively.

The purpose of an ENC is to allow a system administrator to maintain an inventory of nodes to be managed, completely separately from the configuration of the automation tool. Usually, the external node classifier completely replaces the tool-specific inventory (such as site.pp for Puppet, ext_pillar/master_tops for Salt, or /etc/ansible/hosts).

With respect to the configuration management tool, the ENC then fulfills two jobs:

  • it provides information about groups of nodes and group memberships
  • it gives access to node-specific information, such as variables

reclass allows you to define your nodes through class inheritance, while always able to override details further up the tree (i.e. in more specific nodes). Think of classes as feature sets, as commonalities between nodes, or as tags. Add to that the ability to nest classes (multiple inheritance is allowed, well-defined, and encouraged), and you can assemble your infrastructure from smaller bits, eliminating duplication and exposing all important parameters to a single location, logically organised. And if that isn’t enough, reclass lets you reference other parameters in the very hierarchy you are currently assembling.

from http://reclass.pantsfullofunix.net/

Remote FS

I make experiment with remote storage for reclass, also exist better ways how you can store this data, but this is just simple tool for storing reclass in database with ability to rendering each template with custom queries. That means you can make one template from reclass metada and then render it with different data. This is very useful in some cases.

Install remote_fs storage from https://github.com/michaelkuty/reclass/

Django Backend

Just standard Django Model. All templates are stored in dbtemplates.Template and then are connected to ReclassTemplates which is ServiceTemplate model entity. All ServiceTemplates has context which is standard yaml with some metada. When Salt wants metadata for minion then are called API which render your templates with given context and Jinja2 engine. 

Motivation

When you manage large infrastructure with tens of nodes and hundereds of services then you need some tools for better sleeping. If you are familiar with Salt or other Configuration Management tools like a Puppet or Ansible maybe you listen something about Reclass which is Recursive external node classification which makes your pillars more clear and keeps DRY. But if you have more than one infrastructure then you maybe want to split your reclasses into separate units because customers or users of reclass can ready all data. If you more than one reclass then you need sync some parts which are same or blend together.

Model in this app supports storing Reclass files as templates in db and then could be rendered with given context and saved for update by hands. With this models is possible to do

  • Import your Reclass Metadata
  • Connect nodes and classes to Reclass Model and restrict access
  • Share some parts between Reclasses
  • Render Reclass Templates with Jinja2
  • Process your metada before are sent into your Configuration Management tool
  • Programmatically manage your metadata
  • Post your metadata via Rest API

Import reclass

If you have huge reclass you could use import_reclass function.

python manage.py import_reclass --name=salt-model -path=/home/user/reclass/

Test your instalation

http 10.10.10.166/reclass/mynode.cz 'Authorization:  Token d224f170701754225faa3c33d7d56ca0e9679482'

note: If you haven't token, login into admin and create new one.


http 10.10.10.166/reclass/leonardo-multi.webapp.dev.mjk.robotice.cz 'Authorization:  Token d224f170701754225faa3c33d7d56ca0e9679482'

HTTP/1.0 200 OK
Allow: GET, HEAD, OPTIONS
Content-Language: en
Content-Length: 556
Content-Type: application/json
Date: Sat, 09 Jan 2016 16:55:55 GMT
Server: WSGIServer/0.1 Python/2.7.6
Set-Cookie:  frontend_editing=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
Vary: Accept, Accept-Language, Cookie
X-Frame-Options: SAMEORIGIN

{
    "context": null, 
    "extra": null, 
    "id": 6807, 
    "label": ".leonardo-multi.webapp.dev.mjk.robotice.cz", 
    "modified": null, 
    "path": "/home/majklk/reclass9/nodes/_generated/leonardo-multi.webapp.dev.mjk.robotice.cz.yml", 
    "polymorphic_ctype": 202, 
    "rendered": "{'classes': ['system.linux.system.virtualbox', 'system.leonardo.server.multi', 'system.leonardo.server.app.steakhousepisek'], 'parameters': {'_param': {'salt_master_host': '10.10.10.1'}, 'linux': {'system': {'domain': 'webapp.dev.mjk.robotice.cz', 'name': 'leonardo-multi'}}}}", 
    "sync": null, 
    "template": 4453, 
    "user": null
}

If you call standard reclass-salt then you can see on server side

[09/Jan/2016 16:59:42] "GET /reclass/leonardo-multi.webapp.dev.mjk.robotice.cz HTTP/1.1" 200 556
[09/Jan/2016 16:59:42] "GET /reclass/system.linux.system.virtualbox HTTP/1.1" 200 3066
[09/Jan/2016 16:59:42] "GET /reclass/service.linux.system HTTP/1.1" 200 593
[09/Jan/2016 16:59:43] "GET /reclass/service.ntp.client HTTP/1.1" 200 356
[09/Jan/2016 16:59:43] "GET /reclass/service.openssh.server HTTP/1.1" 200 353
[09/Jan/2016 16:59:43] "GET /reclass/service.salt.minion.master HTTP/1.1" 200 371
[09/Jan/2016 16:59:43] "GET /reclass/system.openssh.server.team.robotice HTTP/1.1" 200 2323
[09/Jan/2016 16:59:43] "GET /reclass/system.leonardo.server.multi HTTP/1.1" 200 544
[09/Jan/2016 16:59:43] "GET /reclass/service.git.client HTTP/1.1" 200 313
[09/Jan/2016 16:59:43] "GET /reclass/service.postgresql.server.local HTTP/1.1" 200 436
[09/Jan/2016 16:59:43] "GET /reclass/service.memcached.server.local HTTP/1.1" 200 407
...

note: Don't worry about these gets, because if you have configured memcached proxy then is no call to API.

Configure Reclass

vim /etc/reclass/reclass-config.yml

storage_type: remote_fs
pretty_print: True
output: yaml
inventory_base_uri: /srv/salt/reclass
remote_fs_host_url: http://10.10.10.166/reclass/
remote_fs_token: d224f170701754225faa3c33d7d56ca0e9679482


python reclass/adapters/salt.py  --pillar leonardo-multi.webapp.dev.mjk.robotice.cz
__reclass__:
  applications:
  - linux
  - ntp
  - openssh
  - salt
  - git
  - postgresql
  - memcached
  - python
  - leonardo
  - nginx
  - supervisor
  classes:
  - service.linux.system
  - service.ntp.client
  - service.openssh.server
  - service.salt.minion.master
  - system.openssh.server.team.robotice
  - service.git.client
  - service.postgresql.server.local
  - service.memcached.server.local
  - service.python.environment
  - service.leonardo.server.multi
  - service.nginx.server.single
  - service.supervisor.server.single
  - system.leonardo.server.common
  - system.linux.system.virtualbox
  - system.leonardo.server.multi
  - system.leonardo.server.app.steakhousepisek

Using multimple backends

Use multi_fs which tries to find in given order, for example try to find in yaml_fs and then if is not present here call remote_fs.

storage_type: multi_fs
pretty_print: True
output: yaml
inventory_base_uri: /srv/salt/reclass
multi_fs: ['yaml_fs', 'remote_fs']
remote_fs_url: http://10.10.10.166/reclass/
remote_fs_token: d224f170701754225faa3c33d7d56ca0e9679482


(venv)root@samsung:/home/majklk/repa/reclass# python reclass/adapters/salt.py  --pillar .ubuntu1404.vanilla.robotice.cz
Node .ubuntu1404.vanilla.robotice.cz not found in yaml_fs

Show me

With app you could process your metada before passing it to Salt or other configuration management tools. All context keys are passed into Jinja2 rendering.

For example if you have dbtemplate.Template

{%- if 'store' in plugins %}

include store plugin..

{%- endif %}

and then create ReclassTemplate with this dbtemplate.Template and someting like this in the context:

plugins: ['store', 'blog']

 

Next steps

Read More