Skip to content

Configurable tools #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
damphyr wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Configurable tools #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
79fcc6d
YAML configuration with updated tool versions
Oct 20, 2016
e04749c
updated gem versions
Oct 20, 2016
489c518
download and unpack uses the YAML config
Oct 20, 2016
ec60108
chef, atom and vagrant plugins read plugin list and versions from the…
Oct 20, 2016
3153bae
pass the tool config
Oct 20, 2016
3463e76
not an Array of Hashes, just a Hash with name=<version
Oct 20, 2016
31a2807
correct some paths and restore the original logic in move_chefdk
tknerr Oct 25, 2016
57fae34
adapt tool name to chef_dk as expected in the Rakefile
tknerr Oct 25, 2016
33da03d
restore the original tool names so they match the paths defined in se…
tknerr Oct 25, 2016
c797e58
fix vagrant plugin installation
tknerr Oct 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
source 'http://rubygems.org'

# gems for building the devpack
gem 'bundler', '>= 1.1.1'
gem 'rake', '>= 0.9.2'
gem 'redcarpet', '~>2.1.1'
gem 'bundler', '~> 1.13.5'
gem 'rake','~>11.3.0'
gem 'redcarpet', '~>3.3.4'
gem 'albino', '~>1.3.3'

gem 'rspec', '~> 2.14.1'
17 changes: 10 additions & 7 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ GEM
albino (1.3.3)
posix-spawn (>= 0.3.6)
diff-lcs (1.2.5)
posix-spawn (0.3.8)
rake (10.4.2)
redcarpet (2.1.1)
posix-spawn (0.3.11)
rake (11.3.0)
redcarpet (3.3.4)
rspec (2.14.1)
rspec-core (~> 2.14.0)
rspec-expectations (~> 2.14.0)
rspec-mocks (~> 2.14.0)
rspec-core (2.14.7)
rspec-core (2.14.8)
rspec-expectations (2.14.5)
diff-lcs (>= 1.1.3, < 2.0)
rspec-mocks (2.14.6)
Expand All @@ -22,7 +22,10 @@ PLATFORMS

DEPENDENCIES
albino (~> 1.3.3)
bundler (>= 1.1.1)
rake (>= 0.9.2)
redcarpet (~> 2.1.1)
bundler (~> 1.13.5)
rake (~> 11.3.0)
redcarpet (~> 3.3.4)
rspec (~> 2.14.1)

BUNDLED WITH
1.13.5
173 changes: 94 additions & 79 deletions Rakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,76 @@
%w{ bundler/setup rubygems fileutils uri net/https tmpdir digest/md5 ./doc/markit }.each do |file|
%w{ bundler/setup yaml uri net/https tmpdir digest/md5 ./doc/markit }.each do |file|
require file
end

# Immediately sync all stdout so that it's immediately visible, e.g. on appveyor
$stdout.sync = true
$stderr.sync = true

VERSION = '3.1-SNAPSHOT'
BASE_DIR = File.expand_path('.', File.dirname(__FILE__))
TARGET_DIR = "#{BASE_DIR}/target"
BUILD_DIR = "#{BASE_DIR}/target/build"
CACHE_DIR = "#{BASE_DIR}/target/cache"
ZIP_EXE = 'C:\Program Files\7-Zip\7z.exe'
module EnvironmentOptions
BASE_DIR = File.expand_path('.', File.dirname(__FILE__))
VERSION = '3.1-SNAPSHOT'
def version
VERSION
end
def base_dir
return BASE_DIR
end
def target_dir
ddir=ENV["TARGET_DIR"]
ddir||=File.join(base_dir,"target")
File.expand_path(ddir)
Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about File.expand_path(ENV["TARGET_DIR"] || File.join(base_dir, "target") instead?

for my own taste a bit more readable as a one-liner, but not being picky here

end
def build_dir
ddir=ENV["BUILD_DIR"]
ddir||=File.join(target_dir,"build")
File.expand_path(ddir)
end
def cache_dir
ddir=ENV["CACHE_DIR"]
ddir||=File.join(target_dir,"cache")
File.expand_path(ddir)
end
def zip_exe
zip=ENV["ZIP_EXE"]
zip||='C:\Program Files\7-Zip\7z.exe'
return File.expand_path(zip)
end
end

include EnvironmentOptions

desc 'cleans the build output directory'
task :clean do
purge_atom_plugins_with_insanely_long_path
FileUtils.rm_rf BUILD_DIR, secure: true
FileUtils.rm_rf build_dir, secure: true
end

desc 'wipes all output and cache directories'
task :wipe do
purge_atom_plugins_with_insanely_long_path
FileUtils.rm_rf TARGET_DIR, secure: true
FileUtils.rm_rf target_dir, secure: true
end

desc 'downloads required resources and builds the devpack binary'
task :build => :clean do
tools_config=YAML.load(File.read("#{base_dir}/config/tools.yaml"))
Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I like that! :-)

recreate_dirs
download_tools
move_chefdk
fix_chefdk
download_tools(tools_config)
if tools_config.keys.include?("chefdk")
Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wonder if we could make this a bit more generic and pluggable. not necessarily for bills-kitchen here, but if we want to use that as a minimal template for future devpacks

Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the ifs in the :build task I mean...

move_chefdk
fix_chefdk
end
copy_files
generate_docs
install_knife_plugins
install_vagrant_plugins
install_atom_plugins
if tools_config.keys.include?("chefdk")
install_knife_plugins(tools_config.fetch("chefdk",{}))
end
if tools_config.keys.include?("atom")
install_atom_plugins(tools_config.fetch("atom",{}))
end
if tools_config.keys.include?("vagrant")
install_vagrant_plugins(tools_config.fetch("vagrant",{}))
end
run_integration_tests
end

Expand Down Expand Up @@ -69,9 +103,9 @@ end

def run_acceptance_tests(provider)
Bundler.with_clean_env do
FileUtils.rm_rf "#{BUILD_DIR}/repo/vagrant-workflow-tests"
command = "#{BUILD_DIR}/set-env.bat \
&& cd #{BUILD_DIR}/repo \
FileUtils.rm_rf "#{build_dir}/repo/vagrant-workflow-tests"
command = "#{build_dir}/set-env.bat \
&& cd #{build_dir}/repo \
&& git clone https://github.com/tknerr/vagrant-workflow-tests \
&& cd vagrant-workflow-tests \
&& #{acceptance_test_run_cmd(provider)}"
Expand All @@ -92,16 +126,16 @@ end

def recreate_dirs
%w{ home repo tools }.each do |dir|
FileUtils.mkdir_p "#{BUILD_DIR}/#{dir}"
FileUtils.mkdir_p "#{build_dir}/#{dir}"
end
FileUtils.mkdir_p CACHE_DIR
FileUtils.mkdir_p cache_dir
end

# use Windows builtin robocopy command to purge overly long paths,
# see https://blog.bertvanlangen.com/articles/path-too-long-use-robocopy/
def purge_atom_plugins_with_insanely_long_path
empty_dir = "#{TARGET_DIR}/empty"
atom_packages_dir = "#{BUILD_DIR}/home/.atom/packages"
empty_dir = "#{target_dir}/empty"
atom_packages_dir = "#{build_dir}/home/.atom/packages"
if File.exist?(atom_packages_dir)
FileUtils.rm_rf empty_dir
FileUtils.mkdir_p empty_dir
Expand All @@ -110,114 +144,95 @@ def purge_atom_plugins_with_insanely_long_path
end

def copy_files
FileUtils.cp_r Dir.glob("#{BASE_DIR}/files/*"), "#{BUILD_DIR}"
FileUtils.cp_r Dir.glob("#{base_dir}/files/*"), "#{build_dir}"
end

def generate_docs
Dir.glob("#{BASE_DIR}/*.md").each do |md_file|
Dir.glob("#{base_dir}/*.md").each do |md_file|
html = MarkIt.to_html(IO.read(md_file))
outfile = "#{BUILD_DIR}/_#{File.basename(md_file, '.md')}.html"
outfile = "#{build_dir}/_#{File.basename(md_file, '.md')}.html"
File.open(outfile, 'w') {|f| f.write(html) }
end
end

def download_tools
[
%w{ github.com/boot2docker/boot2docker-cli/releases/download/v1.7.1/boot2docker-v1.7.1-windows-amd64.exe docker/boot2docker.exe },
%w{ get.docker.com/builds/Windows/x86_64/docker-1.7.1.exe docker/docker.exe },
%w{ github.com/Maximus5/ConEmu/releases/download/v16.03.01/ConEmuPack.160301.7z conemu },
%w{ github.com/mridgers/clink/releases/download/0.4.4/clink_0.4.4_setup.exe clink },
%w{ github.com/atom/atom/releases/download/v1.7.3/atom-windows.zip atom },
%w{ github.com/git-for-windows/git/releases/download/v2.8.2.windows.1/PortableGit-2.8.2-64-bit.7z.exe portablegit },
%w{ cdn.rubyinstaller.org/archives/devkits/DevKit-mingw64-32-4.7.2-20130224-1151-sfx.exe devkit },
%w{ downloads.sourceforge.net/project/kdiff3/kdiff3/0.9.96/KDiff3Setup_0.9.96.exe kdiff3
kdiff3.exe },
%w{ the.earth.li/~sgtatham/putty/0.63/x86/putty.zip putty },
%w{ www.itefix.net/dl/cwRsync_5.4.1_x86_Free.zip cwrsync },
%w{ releases.hashicorp.com/vagrant/1.8.1/vagrant_1.8.1.msi vagrant },
%w{ releases.hashicorp.com/terraform/0.6.16/terraform_0.6.16_windows_amd64.zip terraform },
%w{ releases.hashicorp.com/packer/0.10.1/packer_0.10.1_windows_amd64.zip packer },
%w{ releases.hashicorp.com/consul/0.6.4/consul_0.6.4_windows_amd64.zip consul },
%w{ packages.chef.io/stable/windows/2008r2/chefdk-0.13.21-1-x86.msi cdk }
]
.each do |host_and_path, target_dir, includes = ''|
download_and_unpack "http://#{host_and_path}", "#{BUILD_DIR}/tools/#{target_dir}", includes.split('|')
def download_tools tools_config
tools_config.each do |tname,cfg|
download_and_unpack(cfg["url"], File.join(build_dir,'tools',tname), [])
end
end

# move chef-dk to a shorter path to reduce the likeliness that a gem fails to install due to max path length
def move_chefdk
FileUtils.mv "#{BUILD_DIR}/tools/cdk/opscode/chefdk", "#{BUILD_DIR}/tools/chefdk"
# chefdk requires a two step install
unpack "#{BUILD_DIR}/tools/cdk/opscode/chefdk.zip", "#{BUILD_DIR}/tools/chefdk"
FileUtils.rm_rf "#{BUILD_DIR}/tools/cdk"
FileUtils.mv "#{build_dir}/tools/chef_dk/opscode/chefdk", "#{build_dir}/tools/chefdk"
#chefdk install package contains a zip file
unpack("#{build_dir}/tools/chef_dk/opscode/chefdk.zip", "#{build_dir}/tools/chefdk")
FileUtils.rm_rf "#{build_dir}/tools/chef_dk"
end

# ensure omnibus / chef-dk use the embedded ruby, see opscode/chef#1512
def fix_chefdk
Dir.glob("#{BUILD_DIR}/tools/chefdk/bin/*").each do |file|
Dir.glob("#{build_dir}/tools/chefdk/bin/*").each do |file|
if File.extname(file).empty? && File.exist?("#{file}.bat") # do this only for the extensionless .bat counterparts
File.write(file, File.read(file).gsub('#!C:/opscode/chefdk/embedded/bin/ruby.exe', '#!/usr/bin/env ruby'))
end
end
Dir.glob("#{BUILD_DIR}/tools/chefdk/embedded/bin/*").each do |file|
Dir.glob("#{build_dir}/tools/chefdk/embedded/bin/*").each do |file|
if File.extname(file).empty? && File.exist?("#{file}.bat") # do this only for the extensionless .bat counterparts
File.write(file, File.read(file).gsub('#!C:/opscode/chefdk/embedded/bin/ruby.exe', '#!/usr/bin/env ruby'))
end
end
Dir.glob("#{BUILD_DIR}/tools/chefdk/embedded/bin/*.bat").each do |file|
Dir.glob("#{build_dir}/tools/chefdk/embedded/bin/*.bat").each do |file|
File.write(file, File.read(file).gsub('@"C:\opscode\chefdk\embedded\bin\ruby.exe" "%~dpn0" %*', '@"%~dp0ruby.exe" "%~dpn0" %*'))
end
Dir.glob("#{BUILD_DIR}/tools/chefdk/embedded/lib/ruby/gems/2.1.0/bin/*.bat").each do |file|
Dir.glob("#{build_dir}/tools/chefdk/embedded/lib/ruby/gems/2.1.0/bin/*.bat").each do |file|
File.write(file, File.read(file).gsub('@"C:\opscode\chefdk\embedded\bin\ruby.exe" "%~dpn0" %*', '@"%~dp0\..\..\..\..\..\bin\ruby.exe" "%~dpn0" %*'))
end
end

def install_knife_plugins
def install_knife_plugins tool_config
Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice one

commands=["#{build_dir}/set-env.bat"]
tool_config.fetch("plugins",{}).each do |plug,ver|
commands<<"chef gem install #{plug} -v #{ver} --no-ri --no-rdoc"
end
Bundler.with_clean_env do
command = "#{BUILD_DIR}/set-env.bat \
&& chef gem install knife-audit -v 0.2.0 --no-ri --no-rdoc \
&& chef gem install knife-server -v 1.1.0 --no-ri --no-rdoc"
command = commands.join(" && ")
fail "knife plugin installation failed" unless system(command)
end
end

def install_vagrant_plugins
def install_vagrant_plugins tool_config
Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice one

commands=["#{build_dir}/set-env.bat"]
tool_config.fetch("plugins",{}).each do |plug,ver|
commands<<"vagrant plugin install #{plug} --plugin-version #{ver}"
end
Bundler.with_clean_env do
command = "#{BUILD_DIR}/set-env.bat \
&& vagrant plugin install vagrant-toplevel-cookbooks --plugin-version 0.2.4 \
&& vagrant plugin install vagrant-omnibus --plugin-version 1.4.1 \
&& vagrant plugin install vagrant-cachier --plugin-version 1.2.1 \
&& vagrant plugin install vagrant-proxyconf --plugin-version 1.5.2 \
&& vagrant plugin install vagrant-berkshelf --plugin-version 4.1.0 \
&& vagrant plugin install vagrant-winrm --plugin-version 0.7.0"
command = commands.join(" && ")
fail "vagrant plugin installation failed" unless system(command)
end
end

def install_atom_plugins
def install_atom_plugins tool_config
Copy link
Owner

@tknerr tknerr Oct 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice one, too.

commands=["#{build_dir}/set-env.bat"]
commands+=tool_config.fetch("plugins",{}).keys.map do |plug|
"apm install #{plug}"
end
Bundler.with_clean_env do
command = "#{BUILD_DIR}/set-env.bat \
&& apm install atom-beautify \
&& apm install minimap \
&& apm install line-ending-converter \
&& apm install language-chef \
&& apm install language-batchfile"
command = commands.join(" && ")
fail "atom plugins installation failed" unless system(command)
end
end

def reset_git_user
Bundler.with_clean_env do
command = "#{BUILD_DIR}/set-env.bat \
command = "#{build_dir}/set-env.bat \
&& git config --global --unset user.name \
&& git config --global --unset user.email"
fail "resetting dummy git user failed" unless system(command)
end
end

def pre_packaging_checks
chefdk_gem_bindir = "#{BUILD_DIR}/home/.chefdk/gem/ruby/2.1.0/bin"
chefdk_gem_bindir = "#{build_dir}/home/.chefdk/gem/ruby/2.1.0/bin"
unless Dir.glob("#{chefdk_gem_bindir}/*").empty?
raise "beware: gem binaries in '#{chefdk_gem_bindir}' might use an absolute path to ruby.exe! Use `gem pristine` to fix it."
end
Expand All @@ -226,7 +241,7 @@ end
def assemble_kitchen
pre_packaging_checks
reset_git_user
pack BUILD_DIR, "#{TARGET_DIR}/bills-kitchen-#{VERSION}.7z"
pack build_dir, "#{target_dir}/bills-kitchen-#{VERSION}.7z"
end

def download_and_unpack(url, target_dir, includes = [])
Expand All @@ -245,7 +260,7 @@ end
def download(url, outfile)
puts "checking cache for '#{url}'"
url_hash = Digest::MD5.hexdigest(url)
cached_file = "#{CACHE_DIR}/#{url_hash}"
cached_file = "#{cache_dir}/#{url_hash}"
if File.exist? cached_file
puts "cache-hit: read from '#{url_hash}'"
FileUtils.cp cached_file, outfile
Expand Down Expand Up @@ -300,7 +315,7 @@ def unpack(archive, target_dir, includes = [])
puts "extracting '#{archive}' to '#{target_dir}'"
case File.extname(archive)
when '.zip', '.7z', '.exe'
system("\"#{ZIP_EXE}\" x -o\"#{target_dir}\" -y \"#{archive}\" -r #{includes.join(' ')} 1> NUL")
system("\"#{zip_exe}\" x -o\"#{target_dir}\" -y \"#{archive}\" -r #{includes.join(' ')} 1> NUL")
when '.msi'
system("start /wait msiexec /a \"#{archive.gsub('/', '\\')}\" /qb TARGETDIR=\"#{target_dir.gsub('/', '\\')}\"")
else
Expand All @@ -311,7 +326,7 @@ end
def pack(target_dir, archive)
puts "packing '#{target_dir}' into '#{archive}'"
Dir.chdir(target_dir) do
system("\"#{ZIP_EXE}\" a -t7z -y \"#{archive}\" \".\" 1> NUL")
system("\"#{zip_exe}\" a -t7z -y \"#{archive}\" \".\" 1> NUL")
end
end

Expand Down
Loading