Automation for DBA - Vagrant part 2

Hello,

In the last post I presented steps to create Virtual Machine using Vagrant with Virtual Box. It is a native combination but there are other possibilities as well. This post will cover an integration of Vagrant tool with KVM hypervisor and AWS EC2 cloud.

In both cases, an additional plugins has to be installed:
-    vagrant-libvirt plugin for KVM - https://github.com/pradels/vagrant-libvirt
-    vagrant-aws plugin for EC2 - https://github.com/mitchellh/vagrant-aws

If there is a requirement to use a Vagrant boxes defined for other hypervisors, there is a plugin called vagrant-mutate, which help you to convert it from one format to another.

KVM installation steps:

Host configuration:
-    Fedora 21
-    QEMU emulator version 2.1.


List of plugins is maintain on this page - Available-Vagrant-Plugins

Vagrant has very easy way to manage plugins. If plugin is listed on the official page installation process is limited to single command.

[pioro@piorovm]$ vagrant plugin install vagrant-mutate
Installing the 'vagrant-mutate' plugin. This can take a few minutes...
Installed the plugin 'vagrant-mutate (1.0.1)'!

[pioro@piorovm]$ vagrant plugin install vagrant-libvirt
Installing the 'vagrant-libvirt' plugin. This can take a few minutes...
Installed the plugin 'vagrant-libvirt (0.0.30)'!

Displaying a list of installed plugins with version.

[pioro@piorovm]$ vagrant plugin list
vagrant-libvirt (0.0.30)
vagrant-mutate (1.0.1)
vagrant-share (1.1.4)

As in the previous post a prebuild box racattack/oracle65 will be used. This box has been defined for Virtual Box so it has to be downloaded and migrated into KVM format. There is one caveat – after conversion new box will be still using a SATA drivers and not a native virtio drivers. If performance of Vagrant box is important, I will recommend creating a new native version of box for KVM configured with virtio.

Box racattack has two versions in official repository and mutate plugin was unable to use it directly. To workaround that issue a box has to be downloaded first using ex. wget. 

[pioro@piorovm ~]$ wget https://atlas.hashicorp.com/racattack/boxes/oracle65/versions/14.11.01/providers/virtualbox.box
--2015-08-19 23:33:57--  https://atlas.hashicorp.com/racattack/boxes/oracle65/versions/14.11.01/providers/virtualbox.box
Resolving atlas.hashicorp.com (atlas.hashicorp.com)... 107.23.224.212, 54.175.82.169
Connecting to atlas.hashicorp.com (atlas.hashicorp.com)|107.23.224.212|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://dl.dropbox.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0 [following]
--2015-08-19 23:33:58--  https://dl.dropbox.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0
Resolving dl.dropbox.com (dl.dropbox.com)... 54.243.87.222
Connecting to dl.dropbox.com (dl.dropbox.com)|54.243.87.222|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://dl.dropboxusercontent.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0 [following]
--2015-08-19 23:33:58--  https://dl.dropboxusercontent.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0
Resolving dl.dropboxusercontent.com (dl.dropboxusercontent.com)... 107.21.214.189, 107.22.170.202, 107.22.227.179, ...
Connecting to dl.dropboxusercontent.com (dl.dropboxusercontent.com)|107.21.214.189|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1185789020 (1.1G) [application/octet-stream]
Saving to: ‘virtualbox.box’

In the next steps it will be renamed, converted and added to box's repository.

[pioro@piorovm ~]$ mv virtualbox.box racattack.box

[pioro@piorovm ~]$ vagrant mutate racattack.box libvirt
Extracting box file to a temporary directory.
Converting racattack from virtualbox to libvirt.
    (100.00/100%)
Cleaning up temporary files.
The box racattack (libvirt) is now ready to use.

[pioro@piorovm ~]$ vagrant box list
homeora65 (libvirt, 0)
precise32 (virtualbox, 0)
racattack (libvirt, 0)

Vagrant configuration is ready and a new Virtual Machine can be created. Like in the previous host a new directory will be created and a Vagrant will be initialized. In the next step a VM will be started.

[pioro@piorovm ~]$ cd test
[pioro@piorovm test]$ vagrant init racattack
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
[pioro@piorovm test]$ vagrant up
Bringing machine 'default' up with 'libvirt' provider...
==> default: Uploading base box image as volume into libvirt storage...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              test_default
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              1
==> default:  -- Memory:            512M
==> default:  -- Base box:          racattack
==> default:  -- Storage pool:      default
==> default:  -- Image:             /VM/d1/images/test_default.img
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:
==> default:  -- Initrd:
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     5900
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        9216
==> default:  -- Keymap:            en-us
==> default:  -- Command line :
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if its present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
==> default: Rsyncing folder: /home/pioro/test/ => /vagrant
[pioro@piorovm test]$

There is a difference how files are synchronized between Vagrant and a VM machine in VirtualBox and other providers. Virtual Box is using a shared directory feature and KVM and EC2 are using rsync. Rsync process is started only once during VM start, and it to synchronize files after that “vagrant rsync” command has to be run

Like I mentioned in previous post, there are differences in Vagrant configuration between hypervisors. 

The following example is showing a configuration which will add two disks:
 - 10 GB none shared disk and 
 - 20 GB disk which can be shared between hosts. 

Memory size is set to 4 GB and number of vCPUs is set to 2. 
VM will have two network interfaces - one bridged to host network card (192.168.1.x) and one private KVM network (10.10.10.x)

Vagrant.configure(2) do |config|
config.vm.box = "racattack "

  config.vm.provider :libvirt do |libvirt|
 libvirt.storage_pool_name = "pool_d1"
        libvirt.memory=4096
        libvirt.cpus=2
        libvirt.storage :file, :size => '20G', :format => 'qcow2
        libvirt.storage :file, :size => '10G', :type => 'raw', :allow_existing => 'true', :bus=> 'scsi', :device=>'sda', :path=>'asmdisk1_2'
  end

  config.vm.define :host do |host|
         host.vm.hostname = "targetkvm"
         host.vm.network :public_network, :dev => "br0", :mode => "bridge", :type => "bridge", :ip => "192.168.1.152"
         host.vm.network :private_network, :libvirt__network_name => "private", :ip => "10.10.10.152"
  end

end

This is all about KVM for now but I believe this is a good starting point for experiments.

EC2 configuration

AWS integration with Vagrnat is based on plugin as well. The main difference is that Vagrant box is limited to a file definition and VM is based on AMI delivered by AWS.
This AMI image can be official AWS build, community one or the one created by you.

Installation of the additional plugin:

$ vagrant plugin install vagrant-aws
Installing the 'vagrant-aws' plugin. This can take a few minutes...
Installed the plugin 'vagrant-aws (0.6.0)'!

List of all plugins

$ vagrant plugin list
vagrant-aws (0.6.0)
vagrant-share (1.1.4, system)

Let's create a new directory and configure an EC2 container to start

$ mkdir awstest
$ cd awstest/
$ vagrant init

In the next step a VM has to be configured. This is example Vagrantfile

Vagrant.configure("2") do |config|

  # for AWS use dummy box 
  config.vm.box = "dummy"            

  
  config.vm.provider :aws do |aws, override|
    aws.access_key_id = "KEY_ID"
    aws.secret_access_key = "SECRET_ACCESS_KEY"
    aws.instance_type = "t1.micro"
    aws.security_groups = "quick-start-1"
    aws.keypair_name = "mykeys"
    aws.region = "eu-west-1"

    # this is official CentOS AMI
    # agree to CentOS license - http://wiki.centos.org/Cloud/AWS
    aws.ami = "ami-42718735"

    override.ssh.username = "root"
    override.ssh.private_key_path = "/tmp/mykey.pem"

    aws.block_device_mapping = [{ 'DeviceName' => '/dev/sdb1', 'Ebs.VolumeSize' => 50 }]
  end
end

Next step is easy "vagrant up" will start your EC2 instance

$ vagrant up
Bringing machine 'default' up with 'aws' provider...
==> default: Warning! The AWS provider doesn't support any of the Vagrant
==> default: high-level network configurations (`config.vm.network`). They
==> default: will be silently ignored.
==> default: Launching an instance with the following settings...
==> default:  -- Type: t1.micro
==> default:  -- AMI: ami-42718735
==> default:  -- Region: eu-west-1
==> default:  -- Keypair: mykeys
==> default:  -- Security Groups: ["quick-start-1"]
==> default:  -- Block Device Mapping: [{"DeviceName"=>"/dev/sdb1", "Ebs.VolumeSize"=>50}]
==> default:  -- Terminate On Shutdown: false
==> default:  -- Monitoring: false
==> default:  -- EBS optimized: false
==> default:  -- Assigning a public IP address in a VPC: false
==> default: Waiting for instance to become "ready"...
==> default: Waiting for SSH to become available...
==> default: Machine is booted and ready for use!
==> default: Rsyncing folder: /tmp/awstest/ => /vagrant
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

mkdir -p '/vagrant'

Stdout from the command:



Stderr from the command:

sudo: sorry, you must have a tty to run sudo

Instance is up and running. There was a problem with a rsync as there is problem with SUDO configuration in this AMI. It can be resolved but this is not an issue for this post. Let's try to ssh to EC2 VM now

$ vagrant ssh
[root@ip-10-105-167-95 ~]# ls -l /dev/x*
brw-rw----. 1 root disk 202, 64 Sep 20 10:43 /dev/xvde
brw-rw----. 1 root disk 202, 81 Sep 20 10:43 /dev/xvdf1

SSH to EC2 with Vagrant is very simple. You don't need to remember that is a public IP of EC2 instance, "vagrant ssh" will do a trick. 
To stop EC2 instance use "vagrant halt" and to delete EC2 instance use "vagrant destroy"

This is all I would like to share in this post. In the next one, I will use Vagrant with VirtualBox and I will show how to add an automatic Oracle installation to your VM.

regards,
Marcin