Files
mruby-mruby/lib/mruby/build/command.rb
T
KOBAYASHI Shuji 456878ba06 Improve source scanning for presym
The accuracy is greatly improved by using the C preprocessor to scan C
sources for presym. C preprocessor can perfectly interpret all comments and
preprocessor directives, so it can detect all symbols defined, for example
`mrbgems/mruby-socket/src/const.cstub`.

Also, as described later, this change will greatly improve the accuracy of
presym detection from Ruby sources.

## Result

The number of lines in the `presym` file for all gems is as follows:

  ```console
  Previous:   999 (false positive = 89, undetected = 297)
  New:       1207
  ```

## Build process

The new build process (with presym) is as follows:

1. Build `mrbc` without presym (more on building without presym later).
2. Compile Ruby sources to C struct format with the `mrbc` created in
   step 1, and create` mrblib.c` and `gem_init.c`. Note that the symbols
   in the created files are output as `MRB_SYM` family macros or
   `mrb_intern_lit` instead of IDs (details will be described later).
3. C preprocessor processes C sources including the created files of
   step 2 and outputs them as `.i` files. In these files, for example,
   `MRB_IVSYM(foo)` is converted to `<@! "@" "foo" !@>` and
   `mrb_define_module(mrb, "Foo")` is converted to `<@! "Foo" !@>`.
4. Scan the files created in step 3 and create `presym` and` presym.inc`
   files.

The files created in step 2 should output all static symbols defined in Ruby
sources, including local variables, so we can detect all presyms by just
scanning C sources without scanning Ruby sources directly.

Further, by this process, the files to be scanned becomes the same as the
files to be compiled, so that there is no excess or deficiency.

## Related changes

The following changes have been made in relation to realizing this feature.

### Allow build without presym

It enables build without presym to achieve the "Build process: 1". This
incorporates #5202, see its issue for details.

Note that when presym is enabled, even adding a local variable to a Ruby
source may change contents of presym and require recompilation of almost
all C sources. This is inconvenient, especially during trial and error in
development, but this feature is also useful because it does not cause
this problem if presym is disabled.

### Automatically create build target for `mrbc` without presym

The `mrbc` used in the "Build process: 1" will be built by automatically
creating a build target for it. The build name is `SOURCE_BUILD_NAME/mrbc`.

### Constantize output of C struct format by `mrbc`

To realizing the "Build process: 2", as mentioned above, symbol IDs are not
output directly in C struct format output by `mrbc`. As a result, the output
becomes constant regardless of the state of presym at the time of `mrbc`
build, and it is possible to detect symbols of Ruby sources in the same way
as other C sources.

Note that `mrb_intern_lit` is used for symbols that do not become presym,
but in this state, the corresponding element in the symbol array cannot be
statically initialized, so it is initialized at run time (therefore, in this
case, the `const` qualifier is not added to the symbol array).

### Specify arbitrary `mrbc` file

To realizing the "Build process: 2", enabled to specify `mrbc` created by
another build target or pre-built` mrbc`. Use `MRuby::Build#mrbcfile =` to
specify it explicitly. You can omit the "Build process: 1" by specifying
pre-built `mrbc`, and you can always use an optimized build to compile Ruby
sources faster. I think changes that affect the output of `mrbc` are rare,
so in many cases it helps to improve efficiency.

With presym, the build will be a little slower due to more build steps, but
this feature will improve it a bit.

### Create presym files for each build target

This feature was proposed at #5194 and merged once, but was reverted in
5c205e6e due to problems especially with cross-compilation. It has been
introduced again because this change solves the problem.

The presym files will be created below.

* `build/NAME/presym`
* `build/NAME/include/mruby/presym.inc`

### Other changes

* Because presym detection accuracy is greatly improved as mentioned above,
  `MRuby::Gem::Specification#cdump?` is set to true by default, and
  `disable_cdump` is added instead of `enable_cdump`. Also, support for gem
  specific presym files has been discontinued (https://github.com/mruby/mruby/issues/5151#issuecomment-730967232).
* Previously, `mrbc` was automatically created for the `host` build, but it
  will not be created if the build target for `mrbc` mentioned above is
  automatically created. At this time, `mrbc` file of the `mrbc` build is
  copied to` bin/`.
* Two types of `.d` files will be created, `.o.d` and `.i.d`. oThis is
  because if `.i` depends on `presym.inc`, the dependency will circulate, so
  the `.d` file cannot be shared.
* Changed file created with `enable_cxx_exception` to `X-cxx.cxx` from
  `X.cxx` to use the mruby standard Rake rule.

### Note

Almost all C sources will need to be recompiled if there are any changes to
`persym.inc` (if not recompiled properly, it will often result in run-time
error). If `gcc` toolchain is used, dependencies are resolved by the `.d`
file, so it become automatically recompile target, but if not (e.g. MSVC),
it is necessary to manually make it recompile target.

Also, even if `gcc` toolchain is used, it may not become recompile target if
external gems does not use the mruby standard Rake rule. In particular, if
the standard rule is overwritten, such as
https://github.com/mruby/mruby/pull/5112/files, `.d` file will not be read,
so be careful.
2020-12-13 15:27:53 +09:00

366 lines
12 KiB
Ruby

require 'forwardable'
module MRuby
class Command
include Rake::DSL
extend Forwardable
def_delegators :@build, :filename, :objfile, :libfile, :exefile
attr_accessor :build, :command
def initialize(build)
@build = build
end
# clone is deep clone without @build
def clone
target = super
excepts = %w(@build)
instance_variables.each do |attr|
unless excepts.include?(attr.to_s)
val = Marshal::load(Marshal.dump(instance_variable_get(attr))) # deep clone
target.instance_variable_set(attr, val)
end
end
target
end
def shellquote(s)
if ENV['OS'] == 'Windows_NT'
"\"#{s}\""
else
"#{s}"
end
end
private
def _run(options, params={})
sh "#{build.filename(command)} #{options % params}"
end
end
class Command::Compiler < Command
attr_accessor :label, :flags, :include_paths, :defines, :source_exts
attr_accessor :compile_options, :option_define, :option_include_path, :out_ext
attr_accessor :cxx_compile_flag, :cxx_exception_flag, :cxx_invalid_flags
attr_writer :preprocess_options
def initialize(build, source_exts=[], label: "CC")
super(build)
@command = ENV['CC'] || 'cc'
@label = label
@flags = [ENV['CFLAGS'] || []]
@source_exts = source_exts
@include_paths = ["#{MRUBY_ROOT}/include"]
@defines = %w()
@option_include_path = %q[-I"%s"]
@option_define = %q[-D"%s"]
@compile_options = %q[%{flags} -o "%{outfile}" -c "%{infile}"]
@cxx_invalid_flags = []
@out_ext = build.exts.object
end
alias header_search_paths include_paths
def preprocess_options
@preprocess_options ||= @compile_options.sub(/(?:\A|\s)\K-c(?=\s)/, "-E -P")
end
def search_header_path(name)
header_search_paths.find do |v|
File.exist? build.filename("#{v}/#{name}").sub(/^"(.*)"$/, '\1')
end
end
def search_header(name)
path = search_header_path name
path && build.filename("#{path}/#{name}").sub(/^"(.*)"$/, '\1')
end
def all_flags(_defines=[], _include_paths=[], _flags=[])
define_flags = [defines, _defines].flatten.map{ |d| option_define % d }
include_path_flags = [include_paths, _include_paths].flatten.map do |f|
option_include_path % filename(f)
end
[flags, define_flags, include_path_flags, _flags].flatten.join(' ')
end
def run(outfile, infile, _defines=[], _include_paths=[], _flags=[])
mkdir_p File.dirname(outfile)
flags = all_flags(_defines, _include_paths, _flags)
if File.extname(outfile) == build.exts.object
label = @label
opts = compile_options
else
label = "CPP"
opts = preprocess_options
flags << " -DMRB_PRESYM_SCANNING"
end
_pp label, infile.relative_path, outfile.relative_path
_run opts, flags: flags, infile: filename(infile), outfile: filename(outfile)
end
def define_rules(build_dir, source_dir='', out_ext=build.exts.object)
gemrake = File.join(source_dir, "mrbgem.rake")
rakedep = File.exist?(gemrake) ? [ gemrake ] : []
if build_dir.include? "mrbgems/"
generated_file_matcher = Regexp.new("^#{Regexp.escape build_dir}/(.*)#{Regexp.escape out_ext}$")
else
generated_file_matcher = Regexp.new("^#{Regexp.escape build_dir}/(?!mrbgems/.+/)(.*)#{Regexp.escape out_ext}$")
end
source_exts.each do |ext, compile|
rule generated_file_matcher => [
proc { |file|
file.sub(generated_file_matcher, "#{source_dir}/\\1#{ext}")
},
proc { |file|
get_dependencies(file) + rakedep
}
] do |t|
run t.name, t.prerequisites.first
end
rule generated_file_matcher => [
proc { |file|
file.sub(generated_file_matcher, "#{build_dir}/\\1#{ext}")
},
proc { |file|
get_dependencies(file) + rakedep
}
] do |t|
run t.name, t.prerequisites.first
end
end
end
private
#
# === Example of +.d+ file
#
# ==== Without <tt>-MP</tt> compiler flag
#
# /build/host/src/array.o: /src/array.c \
# /include/mruby/common.h /include/mruby/value.h \
# /src/value_array.h
#
# ==== With <tt>-MP</tt> compiler flag
#
# /build/host/src/array.o: /src/array.c \
# /include/mruby/common.h /include/mruby/value.h \
# /src/value_array.h
#
# /include/mruby/common.h:
#
# /include/mruby/value.h:
#
# /src/value_array.h:
#
def get_dependencies(file)
dep_file = "#{file}.d"
return [MRUBY_CONFIG] unless File.exist?(dep_file)
deps = File.read(dep_file).gsub("\\\n ", "").split("\n").map do |dep_line|
# dep_line:
# - "/build/host/src/array.o: /src/array.c /include/mruby/common.h ..."
# - ""
# - "/include/mruby/common.h:"
dep_line.scan(/^\S+:\s+(.+)$/).flatten.map { |s| s.split(' ') }.flatten
# => ["/src/array.c", "/include/mruby/common.h" , ...]
# []
# []
end.flatten.uniq
deps << MRUBY_CONFIG
end
end
class Command::Linker < Command
attr_accessor :flags, :library_paths, :flags_before_libraries, :libraries, :flags_after_libraries
attr_accessor :link_options, :option_library, :option_library_path
def initialize(build)
super
@command = ENV['LD'] || 'ld'
@flags = (ENV['LDFLAGS'] || [])
@flags_before_libraries, @flags_after_libraries = [], []
@libraries = []
@library_paths = []
@option_library = %q[-l"%s"]
@option_library_path = %q[-L"%s"]
@link_options = %Q[%{flags} -o "%{outfile}" %{objs} %{flags_before_libraries} %{libs} %{flags_after_libraries}]
end
def all_flags(_library_paths=[], _flags=[])
library_path_flags = [library_paths, _library_paths].flatten.map do |f|
option_library_path % filename(f)
end
[flags, library_path_flags, _flags].flatten.join(' ')
end
def library_flags(_libraries)
[libraries, _libraries].flatten.map{ |d| option_library % d }.join(' ')
end
def run_attrs
[@libraries, @library_paths, @flags, @flags_before_libraries, @flags_after_libraries]
end
def run(outfile, objfiles, _libraries=[], _library_paths=[], _flags=[], _flags_before_libraries=[], _flags_after_libraries=[])
mkdir_p File.dirname(outfile)
library_flags = [libraries, _libraries].flatten.map { |d| option_library % d }
_pp "LD", outfile.relative_path
_run link_options, { :flags => all_flags(_library_paths, _flags),
:outfile => filename(outfile) , :objs => filename(objfiles).map{|f| %Q["#{f}"]}.join(' '),
:flags_before_libraries => [flags_before_libraries, _flags_before_libraries].flatten.join(' '),
:flags_after_libraries => [flags_after_libraries, _flags_after_libraries].flatten.join(' '),
:libs => library_flags.join(' ') }
end
end
class Command::Archiver < Command
attr_accessor :archive_options
def initialize(build)
super
@command = ENV['AR'] || 'ar'
@archive_options = 'rs "%{outfile}" %{objs}'
end
def run(outfile, objfiles)
mkdir_p File.dirname(outfile)
_pp "AR", outfile.relative_path
_run archive_options, { :outfile => filename(outfile), :objs => filename(objfiles).map{|f| %Q["#{f}"]}.join(' ') }
end
end
class Command::Yacc < Command
attr_accessor :compile_options
def initialize(build)
super
@command = 'bison'
@compile_options = %q[-o "%{outfile}" "%{infile}"]
end
def run(outfile, infile)
mkdir_p File.dirname(outfile)
_pp "YACC", infile.relative_path, outfile.relative_path
_run compile_options, { :outfile => filename(outfile) , :infile => filename(infile) }
end
end
class Command::Gperf < Command
attr_accessor :compile_options
def initialize(build)
super
@command = 'gperf'
@compile_options = %q[-L ANSI-C -C -p -j1 -i 1 -g -o -t -N mrb_reserved_word -k"1,3,$" "%{infile}" > "%{outfile}"]
end
def run(outfile, infile)
mkdir_p File.dirname(outfile)
_pp "GPERF", infile.relative_path, outfile.relative_path
_run compile_options, { :outfile => filename(outfile) , :infile => filename(infile) }
end
end
class Command::Git < Command
attr_accessor :flags
attr_accessor :clone_options, :pull_options, :checkout_options, :checkout_detach_options, :reset_options
def initialize(build)
super
@command = 'git'
@flags = %w[]
@clone_options = "clone %{flags} %{url} %{dir}"
@pull_options = "--git-dir %{repo_dir}/.git --work-tree %{repo_dir} pull"
@checkout_options = "--git-dir %{repo_dir}/.git --work-tree %{repo_dir} checkout %{checksum_hash}"
@checkout_detach_options = "--git-dir %{repo_dir}/.git --work-tree %{repo_dir} checkout --detach %{checksum_hash}"
@reset_options = "--git-dir %{repo_dir}/.git --work-tree %{repo_dir} reset %{checksum_hash}"
end
def run_clone(dir, url, _flags = [])
_pp "GIT", url, dir.relative_path
_run clone_options, { :flags => [flags, _flags].flatten.join(' '), :url => shellquote(url), :dir => shellquote(filename(dir)) }
end
def run_pull(dir, url)
_pp "GIT PULL", url, dir.relative_path
_run pull_options, { :repo_dir => shellquote(dir) }
end
def run_checkout(dir, checksum_hash)
_pp "GIT CHECKOUT", dir, checksum_hash
_run checkout_options, { :checksum_hash => checksum_hash, :repo_dir => shellquote(dir) }
end
def run_checkout_detach(dir, checksum_hash)
_pp "GIT CHECKOUT DETACH", dir, checksum_hash
_run checkout_detach_options, { :checksum_hash => checksum_hash, :repo_dir => shellquote(dir) }
end
def run_reset_hard(dir, checksum_hash)
_pp "GIT RESET", dir, checksum_hash
_run reset_options, { :checksum_hash => checksum_hash, :repo_dir => shellquote(dir) }
end
def commit_hash(dir)
`#{@command} --git-dir #{shellquote(dir +'/.git')} --work-tree #{shellquote(dir)} rev-parse --verify HEAD`.strip
end
def current_branch(dir)
`#{@command} --git-dir #{shellquote(dir + '/.git')} --work-tree #{shellquote(dir)} rev-parse --abbrev-ref HEAD`.strip
end
end
class Command::Mrbc < Command
attr_accessor :compile_options
def initialize(build)
super
@command = nil
@compile_options = "-B%{funcname} -o-"
end
def run(out, infiles, funcname, cdump = true)
@command ||= @build.mrbcfile
infiles = [infiles].flatten
infiles.each_with_index do |f, i|
_pp i == 0 ? "MRBC" : "", f.relative_path, indent: 2
end
cmd = %Q["#{filename @command}" #{cdump ? "-S" : ""} #{@compile_options % {:funcname => funcname}} #{filename(infiles).map{|f| %Q["#{f}"]}.join(' ')}]
puts cmd if Rake.verbose
IO.popen(cmd, 'r+') do |io|
out.puts io.read
end
# if mrbc execution fail, drop the file
unless $?.success?
rm_f out.path
fail "Command failed with status (#{$?.exitstatus}): [#{cmd[0,42]}...]"
end
end
end
class Command::CrossTestRunner < Command
attr_accessor :runner_options
attr_accessor :verbose_flag
attr_accessor :flags
def initialize(build)
super
@command = nil
@runner_options = '%{flags} %{infile}'
@verbose_flag = ''
@flags = []
end
def run(testbinfile)
puts "TEST for " + @build.name
_run runner_options, { :flags => [flags, verbose_flag].flatten.join(' '), :infile => testbinfile }
end
end
end