Tools¶
Tools for bootstrapping, installing, and managing change in systems.
Objective: Maximize Tool Value (Output / Input)
Distro Packages¶
Operating Systems Packaging
Source and/or binary packages to install from a standard archive with a signed manifest containing file signatures of package files.
RPM Package¶
https://en.wikipedia.org/wiki/RPM_Package_Manager
Installable with yum, {...}
Build with TODO: rpmbuild
Python: build with bdist_rpm, {...}
List contents:
# with lesspipe less ~/path/to/local.rpm
Package Repositories (yum):
- Local: directories of packages and metadata
- Network: HTTP, HTTPS, RSYNC, FTP
DEB Package¶
https://en.wikipedia.org/wiki/Deb_(file_format)
Installable with apt-get, aptitutde,
Build with dpkg
List contents:
# with lesspipe less ~/path/to/local.deb
Package Repositories (apt):
- Local: directories of packages and metadata
- Network: HTTP, HTTPS, RSYNC, FTP (apt transports)
Linux/Mac/Windows: Yes / Fink / No
Homebrew¶
https://en.wikipedia.org/wiki/Homebrew_(package_management_software)
- Linux/Mac/Windows: No / Yes / No
- Package Recipe Repositories (brew):
- Local:
- Network: HTTP, HTTPS
NuGet¶
https://en.wikipedia.org/wiki/NuGet
- Package Repositories (chocolatey):
- Linux/Mac/Windows: No / No / Yes
Portage¶
https://en.wikipedia.org/wiki/Portage_(software)
- Build recipes with flag sets
- Package Repositories (portage)
Port Tree¶
Sources and Makefiles designed to compile software packages for particular distributions’ kernel and standard libraries on a particular platform.
CoreOS Docker Images¶
CoreOS schedules redundant docker images and configuration over etcd, a key-value store with a D-Bus interface.
- Create high availability zone clusters with fleet
- Systemd init files
Apt¶
APT is the Debian package management system.
APT retrieves packages over FTP, HTTP, HTTPS, and RSYNC.
man apt-get
man sources.list
echo 'deb repo_URL distribution component1' >> /etc/apt/sources.list
apt-get update
apt-cache show bash
apt-get install bash
apt-get upgrade
apt-get dist-upgrade
Bash¶
Bash, the Bourne-again shell.
type bash
bash --help
help help
help type
apropos bash
info bash
man bash
Designed to work with unix command outputs and return codes
Functions
Portability: sh (sh, bash, dash, zsh) shell scripts are mostly compatible
Logging:
set -x # print commands and arguments set -v # print source
Bash Configuration:
/etc/profile
/etc/bash.bashrc
/etc/profile.d/*.sh
${HOME}/.profile /etc/skel/.profile # PATH=+$HOME/bin # umask
${HOME}/.bash_profile # empty. preempts .profile
Linux/Mac/Windows: Almost Always / Bash 3.2 / Cygwin/Mingwin
Dpkg¶
Lower-level package management scripts for creating and working with .DEB Debian packages.
Docker¶
Docker is an OS virtualization project which utilizes Linux LXC Containers to partition process workloads all running under one kernel.
Limitations
- Writing to /etc/hosts: https://github.com/dotcloud/docker/issues/2267
- Apt-get upgrade: https://github.com/dotcloud/docker/issues/3934
Docutils¶
Docutils is a text processing system which ‘parses” ReStructuredText lightweight markup language into a doctree which it serializes into HTML, LaTeX, man-pages, Open Document files, XML, and a number of other formats.
Filesystem Hierarchy Standard¶
The Filesystem Hierarchy Standard is a well-worn industry-supported system file naming structure.
Ubuntu and Virtualenv implement a Filesystem Hierarchy.
Docker layers filesystem hierarchies with aufs and now also btrfs subvolumes.
Git¶
Git is a distributed version control system for tracking a branching and merging repository of file revisions.
Go¶
Go is a relatively new statically-typed C-based language.
Libcloud¶
Apache Libcloud is a Python library which abstracts and unifies a large number of Cloud APIs for Compute Resources, Object Storage, Load Balancing, and DNS.
Libvirt¶
Libvirt is a system for platform virtualization with various Linux hypervisors.
- KVM/QEMU
- Xen
- LXC
- OpenVZ
- VirtualBox
Linux¶
A free and open source operating system kernel written in C.
uname -a
Make¶
GNU Make is a classic, ubiquitous software build tool designed for file-based source code compilation.
Bash, Python, and the GNU/Linux kernel are all built with Make.
Make build task chains are represented in a Makefile.
Pros
- Simple, easy to read syntax
- Designed to build files on disk
- Nesting: make -C <path> <taskname>
- Variable Syntax: $(VARIABLE_NAME)
- Bash completion: make <tab>
- Python: Parseable with disutils.text_file Text File
- Logging: command names and values to stdout
Cons
- Platform Portability: make is not installed everywhere
- Global Variables: Parametrization with shell scripts
- Linux/Mac/Windows: Usually / brew / executable
MessagePack¶
MessagePack is a data interchange format with implementations in many languages.
Packer¶
Packer generates machine images for multiple platforms, clouds, and hypervisors from a parameterizable template.
- Packer Artifact
- Build products: machine image and manifest
- Packer Template
- JSON build definitions with optional variables and templating
- Packer Build
- A task defined by a JSON file containing build steps which produce a machine image
- Packer Builder
Packer components which produce machine images for one of many platforms:
- VirtualBox
- Docker
- OpenStack
- GCE
- EC2
- VMware
- QEMU (KVM, Xen)
- http://www.packer.io/docs/templates/builders.html
- Packer Provisioner
Packer components for provisioning machine images at build time
- Shell scripts
- File uploads
- ansible
- chef
- solo
- puppet
- salt
- Packer Post-Processor
- Packer components for compressing and uploading built machine images
Perl¶
Perl is a dynamically typed, C-based scripting language.
Many of the Debian system management tools are or were originally written in Perl.
Python¶
Python is a dynamically-typed, C-based scripting language.
Many of the RedHat system management tools are or were originally written in Python.
Pip, Sphinx, Salt, Tox, Virtualenv, and Virtualenvwrapper are all written in Python.
Python Package¶
Archive of source and/or binary files containing a setup.py.
A setup.py calls a distutils.setup or setuptools.setup function with package metadata fields like name, version, maintainer name, maintainer email, and home page; as well as package requirements: lists of package names and version specifiers in install_requires and tests_require, and a dict for any extras_require such that ‘easy_install setup.py, python setup.py install, and pip install --upgrade pip can all retrieve versions of packages which it depends on.
- Distutils is in the Python standard library
- Setuptools is widely implemented: easy_install
- Setuptools can be installed with python ez_setup.py
- Setuptools can be installed with a system package manager (apt, yum)
- Python packages are tested and repackaged by package maintainers
- Python packages are served from a package index
- PyPi is the Python Community package home
- Packages are released to PyPi
- Package Repositories (setup.py -> pypi)
- Package Repositories (conda)
- Package Repositories (enpkg)
- Package Repositories (deb/apt, rpm/yum)
- Build RPM and DEB packages from Python packages with setuptools
- python setup.py bdist_rpm --help
- python setup.py --command-packages=stdeb.command bdist_deb --help
Pip¶
Pip is a tool for working with Python packages.
pip help
pip help install
pip --version
sudo apt-get install python-pip
pip install --upgrade pip
pip install libcloud
pip install -r requirements.txt
pip uninstall libcloud
- Pip retrieves and installs packages from package indexes
- Pip can do uninstall and upgrade
- Pip builds upon distutils and setuptools
- Pip can install from version control repository URLs
- Pip configuration is in ${HOME}/.pip/pip.conf.
- Pip can maintain a local cache of downloaded packages
Note
With Python 2, pip is preferable to easy_install because Pip installs backports.ssl_match_hostname.
- Pip Requirements File
Plaintext list of packages and package URIs to install.
Requirements files may contain version specifiers (pip >= 1.5)
Pip installs Pip Requirement Files:
pip install -r requirements.txt pip install --upgrade -r requirements.txt pip install --upgrade --user --force-reinstall -r requirements.txt
An example requirements.txt file:
# install pip from the default index (PyPi) pip --index=https://pypi.python.org/simple --upgrade pip # Install pip 1.5 or greater from PyPi pip >= 1.5 # Git clone and install pip as an editable develop egg -e git+https://github.com/pypa/pip@1.5.X#egg=pip # Install a source distribution release from PyPi # and check the MD5 checksum in the URL https://pypi.python.org/packages/source/p/pip/pip-1.5.5.tar.gz#md5=7520581ba0687dec1ce85bd15496537b # Install a source distribution release from Warehouse https://warehouse.python.org/packages/source/p/pip/pip-1.5.5.tar.gz # Install an additional requirements.txt file -r requirements/more-requirements.txt
ReStructuredText¶
ReStructuredText (RST, ReST) is a plaintext lightweight markup language commonly used for narrative documentation and Python docstrings.
Sphinx is built on Docutils, which is the primary implementation of ReStructuredText.
Pandoc also supports a form of ReStructuredText.
- ReStructuredText Directive
Actionable blocks of ReStructuredText
.. include:: goals.rst .. contents:: Table of Contents :depth: 3 .. include:: LICENSE
- ReStructuredText Role
RestructuredText role extensions
.. _anchor-name: :ref:`Anchor <anchor-name>`
Salt¶
Salt is an open source configuration management system for managing one or more physical and virtual machines running various operating systems.
- Salt Top File
- Root of a Salt Environment (top.sls)
- Salt Environment
- Folder of Salt States with a top.sls top file.
- Salt Bootstrap
- Installer for salt master and/or salt minion
- Salt Minion
Daemon process which executes Salt States on the local machine.
Can run as a background daemon. Can retrieve and execute states from a salt master
Can execute local states in a standalone minion setup:
salt-call --local grains.items
- Salt Minion ID
Machine ID value uniquely identifying a minion instance to a Salt Master.
By default the minion ID is set to the FQDN
python -c 'import socket; print(socket.getfqdn())'
The minion ID can be set explicitly in two ways:
/etc/salt/minion.conf:
id: devserver-123.example.org
/etc/salt/minion_id:
$ hostname -f > /etc/salt/minion_id $ cat /etc/salt/minion_id devserver-123.example.org
- Salt Master
Server daemon which compiles pillar data for and executes commands on Salt Minions:
salt '*' grains.items
- Salt SSH
Execute salt commands and states over SSH without a minion process:
salt-ssh '*' grains.items
- Salt Grains
Static system information keys and values
- hostname
- operating system
- ip address
- interfaces
Show grains on the local system:
salt-call --local grains.items
- Salt Modules
Remote execution functions for files, packages, services, commands.
Can be called with salt-call
- Salt States
Graphs of nodes and attributes which are templated and compiled into ordered sequences of system configuration steps.
Naturally stored in .sls YAML files parsed by salt.states.<state>.py.
Salt States files are processed as Jinja templates (by default) they can access system-specific grains and pillar data at compile time.
- Salt Renderers
- Templating engines (by default: Jinja) for processing templated states and configuration files.
- Salt Pillar
Key Value data interface for storing and making available global and host-specific values for minions: values like hostnames, usernames, and keys.
Pillar configuration must be kept separate from states (e.g. users, keys) but works the same way.
In a master/minion configuration, minions do not have access to the whole pillar.
- Salt Cloud
Salt Cloud can provision cloud image, instance, and networking services with various cloud providers (libcloud):
- Google Compute Engine (GCE) [KVM]
- Amazon EC2 [Xen]
- Rackspace Cloud [KVM]
- OpenStack [https://wiki.openstack.org/wiki/HypervisorSupportMatrix]
- Linux LXC (Cgroups)
- KVM
Sphinx¶
Sphinx is a tool for working with ReStructuredText documentation trees and rendering them into HTML, PDF, LaTeX, ePub, and a number of other formats.
Sphinx extends Docutils with a number of useful markup behaviors which are not supported by other ReStructuredText parsers.
Most other ReStructuredText parsers do not support Sphinx directives; so, for example,
GitHub and BitBucket do not support Sphinx but do support ReStructuredText so README.rst containing Sphinx tags renders in plaintext or raises errors.
For example, the index page of this Sphinx documentation set is generated from a file named index.rst and referenced by docs/conf.py.
- Sphinx Builder
Render Sphinx ReStructuredText into various forms:
- HTML
- LaTeX
- ePub
See: Sphinx Builders
- Sphinx ReStructuredText
- Sphinx extends ReStructuredText with roles and directives which only work with Sphinx.
- Sphinx Directive
Sphinx extensions of Docutils ReStructuredText directives.
Most other ReStructuredText parsers do not support Sphinx directives.
.. toctree:: readme installation usage
See: Sphinx Directives
- Sphinx Role
Sphinx extensions of Docutils ReStructuredText roles
Most other ReStructured
.. _anchor-name: :ref:`Anchor <anchor-name>`
Ruby¶
Ruby is a dynamically-typed programming language.
Vagrant is written in Ruby.
Tox¶
Tox is a build automation tool designed to build and test Python projects with multiple language versions and environments in separate virtualenvs.
Run the py27 environment:
tox -v -e py27
tox --help
Ubuntu¶
Vagrant¶
Vagrant is a tool for creating and managing virtual machine instances with CPU, RAM, Storage, and Networking.
- Vagrant:
- provides helpful commandline porcelain on top of VirtualBox VboxManage
vagrant help
vagrant status
vagrant init ubuntu/trusty64
vagrant up
vagrant ssh
$EDITOR Vagrantfile
vagrant provision
vagrant halt
vagrant destroy
- Vagrantfile
Vagrant script defining a team of one or more virtual machines and networks.
Create a Vagrantfile:
vagrant init [basebox] cat Vagrantfile
Start virtual machines and networks defined in the Vagrantfile:
vagrant status vagrant up
- Vagrant Box
Vagrant base machine virtual machine image.
There are many baseboxes for various operating systems.
Essentially a virtual disk plus CPU, RAM, Storage, and Networking metadata.
Locally-stored and cached vagrant boxes can be listed with:
vagrant help box vagrant box list
A running vagrant environment can be packaged into a new box with:
vagrant package
Packer generates VirtualBox Vagrant Boxes with a Post-Processor.
- Vagrant Cloud
Vagrant-hosted public Vagrant Box storage.
Install a box from Vagrant cloud:
vagrant init ubuntu/trusty64 vagrant up vagrant ssh
- Vagrant Provider
A driver for running Vagrant Boxes with a hypervisor or in a cloud.
The Vagrant VirtualBox Provider is well-supported.
With Plugins: https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins
See also: Libcloud.
- Vagrant Provisioner
Set of hooks to install and run shell scripts and configuration managment tools over vagrant ssh.
Vagrant up runs vagrant provision on first invocation of vagrant up.
vagrant provision
Note
Vagrant configures a default NFS share mounted at /vagrant.
Note
Vagrant adds a default NAT Adapter as eth0; presumably for DNS, the default route, and to ensure vagrant ssh connectivity.
VirtualBox¶
Oracle VirtualBox is a platform virtualization package for running one or more guest VMs (virtual machines) within a host system.
VirtualBox:
- runs on many platforms: Linux, OSX, Windows
- has support for full platform NX/AMD-v virtualization
- requires matching kernel modules
Vagrant scripts VirtualBox.
Virtualenv¶
Virtualenv is a tool for creating reproducible Python environments.
Virtualenv sets the shell environment variable $VIRTUAL_ENV when active.
Paths within a virtualenv are more-or-less FSH standard paths, making virtualenv structure very useful for building chroot and container overlays.
A standard virtual environment:
bin/ # pip, easy_install, console_scripts
bin/activate # source bin/activate to work on a virtualenv
include/ # (symlinks to) dev headers (python-dev/python-devel)
lib/ # libraries
lib/python2.7/site-packages/ # pip and easy_installed packages
local/ # symlinks to bin, include, and lib
src/ # pip installs editable requirements here
# also useful
etc/ # configuration
var/log # logs
var/run # sockets, PID files
tmp/ # mkstemp temporary files with permission bits
srv/ # local data
Virtualenvwrapper wraps virtualenv. In the following code shell example, comments with ## are virtualenvwrapper
# Print Python site settings
python -m site
# Create a virtualenv
cd $WORKON_HOME
virtualenv example
source ./example/bin/activate
## mkvirtualenv example
## workon example
# Review virtualenv Python site settings
python -m site
# List files in site-packages
ls -altr $VIRTUAL_ENV/lib/python*/site-packages/**
## (cdsitepackages && ls -altr **)
## lssitepackages -altr **
Virtualenvwrapper¶
Virtualenvwrapper is a tool which extends virtualenvwrapper.
Virtualenvwrapper provides a number of useful shell commands and python functions for working with and within virtualenvs, as well as project event scripts (e.g. postactivate, postmkvirtualenv) and two filesystem configuration variables useful for structuring development projects of any language within virtualenvs: $PROJECT_HOME and $WORKON_HOME.
Virtualenvwrapper is sourced into the shell:
# pip install --user --upgrade virtualenvwrapper
source ~/.local/bin/virtualenvwrapper.sh
# sudo apt-get install virtualenvwrapper
source /etc/bash_completion.d/virtualenvwrapper
echo $PROJECT_HOME; echo ~/wrk # default: ~/workspace
echo $WORKON_HOME; echo ~/wrk/.ve # default: ~/.virtualenvs
mkvirtualenv example
workon example
cdvirtualenv ; ls
mkdir src ; cd src/
cdsitepackages
lssitepackages
deactivate
rmvirtualenv example