Since presym is now always enabled, the @enable_presym flag,
presym_enabled? method, and all conditional branches guarding
presym-specific code paths are dead code. Remove them and
simplify the affected build logic.
Co-authored-by: Claude <noreply@anthropic.com>
Headers in mrbgems are now categorized into three types:
- src/*.h: gem internal only
- include/*.h: inter-gem use (visible to dependent gems)
- include/export/*.h: external API (exported via mruby-config --cflags)
This prevents internal headers like *_hal.h from being exposed to
external users while maintaining inter-gem header accessibility.
Co-authored-by: Claude <noreply@anthropic.com>
The issue resolved by the preceding patch was solely the C++ exception task within the mruby core.
This patch aims to resolve a similar sequencing issue that also exists in GEMS.
In practice, `mruby-compiler` is sometimes loaded via dependencies rather than being explicitly specified in the build configuration file.
In such cases, when `mruby-compiler/mrbgem.rake` is loaded, it is not yet determined whether C++ exceptions will be used. Consequently, even if it later becomes clear that `core/codegen-cxx.cxx` and `core/y.tab-cxx.cxx` are required, the system could not handle this.
To resolve this issue, we introduce the `MRuby::Gem::Specification#build_settings` method as a mechanism for lazily evaluating build setup.
However, for backward compatibility, the commands are cloned twice in `gem.setup` and `gem.setup_build`.
This is because many existing GEMS configure commands directly within the setup block.
ref. https://github.com/mruby/mruby/issues/6615
Until now, GEMs dependent on GEMs described in the build configuration file were loaded and set up after mruby core tasks were defined.
This caused an issue where, if C++ exceptions were enabled later by a dependent GEM, the necessary tasks for mruby core were not defined.
fixed https://github.com/mruby/mruby/issues/6615
```console
% find -s lib -type f -name '*.rb' -exec ruby -cw {} \;
lib/mruby/build/command.rb:320: warning: `+' after local variable or literal is interpreted as binary operator
lib/mruby/build/command.rb:320: warning: even though it seems like unary operator
Syntax OK
Syntax OK
Syntax OK
Syntax OK
Syntax OK
lib/mruby/gem.rb:469: warning: `&' interpreted as argument prefix
Syntax OK
Syntax OK
Syntax OK
Syntax OK
```
Make the `mrb_init_mrbgems()` function do the cleanup and error handling.
Instead, simplify processing in the `GENERATED_TMP_mrb_XXX_gem_init()` function.
Previously, `mrb_load_irep()` or `mrb_load_proc()` would kill the process when an exception occurred.
With this patch, it is now caught by `mrb_core_init_protect()`.
The `REnv` object is difficult to deal with, and it would be ideal if the user did not have to manipulate it directly.
In some previous situations, it was necessary to call `mrb_env_unshare()`, a non-API function, after `mrb_load_string()` or similar.
With this patch, it is no longer necessary for users to use `mrb_env_unshare()` directly, as it is now handled internally simply by using the `mrb_vm_ci_env_clear()` function.
Also, `mrb_vm_ci_env_set()` is demoted from the `MRB_API` function for the same reason.
ref. commit 1ab3da6f08
This is so that the build directory can be regarded as a temporary installation directory to work in.
Directories set in `MRuby::Gem::Specification#export_include_paths` are used as they are if they are subdirectories placed under `gem.dir`.
Files are placed under `<build-dir>/include/mruby/gems/<gem-name>/<gem-include_paths>` to separate each GEM.
When GEM is compiled by building `libmruby.a`, the same `include_paths` will be set as before, so it is expected that no unintended references will be made.
Exception raising can now be controlled by the caller.
The main purpose on this patch is:
- Suppress exceptions from `obj_free()` in `src/gc.c` with `mrb_env_unshare()`.
- Consider the possibility that calls to `mrb_malloc()` may cause `e` objects to be subject to GC.
When control is returned to `mrb_env_unshare()`, `struct free_obj::next` in the same offset as `struct REnv::stack` is rewritten.
Unexpected results then occur when the object is reused.
Also, if `mrb_heap_page` containing an `e` object is freed, it may cause `SIGSEGV` at that point.
- Protects the value of the stack on `callinfo` that just exits if GC occurs inside `mrb_env_unshare()`.
```ruby
def m
b = -> { b }
end
p m.call
# => print block object, not nil
```
This patch does not raise a `NoMemoryError` exception in `mrb_env_unshare()` and can detect that error.
Thus, the problem fixed in # 3087 is not resurrected.
Also, it may seem that this patch should suppress exceptions raised by `cipop()` during `mrb_protect_error()` and `mrb_vm_exec()` unwinds.
However, `mrb_callinfo::u.env` by `CINFO_DIRECT` is not seen to be set.
So in that case `mrb_env_unshare()` is assumed to be originally exception-free.
When `disable_cdump` is declared on a Mrbgem spec, the procedure for
loading MRuby code from the gem's `mrblib` directory is slightly
different; instead of loading the mrblib code as a Proc built from the
compiled irep, the compiled irep is loaded directly. The header files
`mruby.h` and `mruby/proc.h` are needed only when the irep is loaded
directly. They are currently included only when presym is disabled, they
should be included whenever either presym is disabled *or* when
`disable_cdump` is called. The `cdump?` predicate method happens to
return true in either case (presym disabled *or* cdump disabled), so
this change should be safe.
With this change, the test code will not be built unless `rake test` is
run, so there will be almost no side effects even if `enable_test` is
always set (but, gems specified by `add_test_dependency` are included
in `libmruby.a`).
Also added are `test: build` task, which only builds the test code
(including the main code), and `test: run` task, which only runs tests
independent of build. Therefore, the idiom for building in parallel and
not running tests in parallel is `rake -m test:build && rake test:run`.
The accuracy is greatly improved by using the C preprocessor to scan C
sources for presym. C preprocessor can perfectly interpret all comments and
preprocessor directives, so it can detect all symbols defined, for example
`mrbgems/mruby-socket/src/const.cstub`.
Also, as described later, this change will greatly improve the accuracy of
presym detection from Ruby sources.
## Result
The number of lines in the `presym` file for all gems is as follows:
```console
Previous: 999 (false positive = 89, undetected = 297)
New: 1207
```
## Build process
The new build process (with presym) is as follows:
1. Build `mrbc` without presym (more on building without presym later).
2. Compile Ruby sources to C struct format with the `mrbc` created in
step 1, and create` mrblib.c` and `gem_init.c`. Note that the symbols
in the created files are output as `MRB_SYM` family macros or
`mrb_intern_lit` instead of IDs (details will be described later).
3. C preprocessor processes C sources including the created files of
step 2 and outputs them as `.i` files. In these files, for example,
`MRB_IVSYM(foo)` is converted to `<@! "@" "foo" !@>` and
`mrb_define_module(mrb, "Foo")` is converted to `<@! "Foo" !@>`.
4. Scan the files created in step 3 and create `presym` and` presym.inc`
files.
The files created in step 2 should output all static symbols defined in Ruby
sources, including local variables, so we can detect all presyms by just
scanning C sources without scanning Ruby sources directly.
Further, by this process, the files to be scanned becomes the same as the
files to be compiled, so that there is no excess or deficiency.
## Related changes
The following changes have been made in relation to realizing this feature.
### Allow build without presym
It enables build without presym to achieve the "Build process: 1". This
incorporates #5202, see its issue for details.
Note that when presym is enabled, even adding a local variable to a Ruby
source may change contents of presym and require recompilation of almost
all C sources. This is inconvenient, especially during trial and error in
development, but this feature is also useful because it does not cause
this problem if presym is disabled.
### Automatically create build target for `mrbc` without presym
The `mrbc` used in the "Build process: 1" will be built by automatically
creating a build target for it. The build name is `SOURCE_BUILD_NAME/mrbc`.
### Constantize output of C struct format by `mrbc`
To realizing the "Build process: 2", as mentioned above, symbol IDs are not
output directly in C struct format output by `mrbc`. As a result, the output
becomes constant regardless of the state of presym at the time of `mrbc`
build, and it is possible to detect symbols of Ruby sources in the same way
as other C sources.
Note that `mrb_intern_lit` is used for symbols that do not become presym,
but in this state, the corresponding element in the symbol array cannot be
statically initialized, so it is initialized at run time (therefore, in this
case, the `const` qualifier is not added to the symbol array).
### Specify arbitrary `mrbc` file
To realizing the "Build process: 2", enabled to specify `mrbc` created by
another build target or pre-built` mrbc`. Use `MRuby::Build#mrbcfile =` to
specify it explicitly. You can omit the "Build process: 1" by specifying
pre-built `mrbc`, and you can always use an optimized build to compile Ruby
sources faster. I think changes that affect the output of `mrbc` are rare,
so in many cases it helps to improve efficiency.
With presym, the build will be a little slower due to more build steps, but
this feature will improve it a bit.
### Create presym files for each build target
This feature was proposed at #5194 and merged once, but was reverted in
5c205e6e due to problems especially with cross-compilation. It has been
introduced again because this change solves the problem.
The presym files will be created below.
* `build/NAME/presym`
* `build/NAME/include/mruby/presym.inc`
### Other changes
* Because presym detection accuracy is greatly improved as mentioned above,
`MRuby::Gem::Specification#cdump?` is set to true by default, and
`disable_cdump` is added instead of `enable_cdump`. Also, support for gem
specific presym files has been discontinued (https://github.com/mruby/mruby/issues/5151#issuecomment-730967232).
* Previously, `mrbc` was automatically created for the `host` build, but it
will not be created if the build target for `mrbc` mentioned above is
automatically created. At this time, `mrbc` file of the `mrbc` build is
copied to` bin/`.
* Two types of `.d` files will be created, `.o.d` and `.i.d`. oThis is
because if `.i` depends on `presym.inc`, the dependency will circulate, so
the `.d` file cannot be shared.
* Changed file created with `enable_cxx_exception` to `X-cxx.cxx` from
`X.cxx` to use the mruby standard Rake rule.
### Note
Almost all C sources will need to be recompiled if there are any changes to
`persym.inc` (if not recompiled properly, it will often result in run-time
error). If `gcc` toolchain is used, dependencies are resolved by the `.d`
file, so it become automatically recompile target, but if not (e.g. MSVC),
it is necessary to manually make it recompile target.
Also, even if `gcc` toolchain is used, it may not become recompile target if
external gems does not use the mruby standard Rake rule. In particular, if
the standard rule is overwritten, such as
https://github.com/mruby/mruby/pull/5112/files, `.d` file will not be read,
so be careful.
Previously, `libmruby.a` was created even if only `mruby-bin-mrbc` or`
mruby-compiler` was specified for gem, but by specifying `disable_libmruby`,
the creation of `libmruby.a` can be suppressed.
### Note
The https://github.com/mruby/mruby/pull/5084#issuecomment-723521971
incompatibility seems to be difficult for users to avoid, so the original
behavior has been restored. Therefore, if we need `mrbc` other than the
`host` build, we need to explicitly specify `mruby-bin-mrbc` gem.
Due to the above changes, `build_config/boxing.rb` etc. will not work, but I
have added` mruby-bin-mrbc` to `default.gembox` to fix it. I don't think
this change is a big deal because originally `mruby-compiler` was included.
* Output `GEN` log for generated files
* `MRBC` log is outputted one for each `mrbc` execution
#### Before this patch:
```console
CC src/array.c -> build/host/src/array.o
(snip)
GEN mrblib/*.rb -> build/host/mrblib/mrblib.c
MRBC mrblib/00class.rb
MRBC mrblib/10error.rb
(snip)
CC mrbgems/mruby-time/src/time.c -> build/host/mrbgems/mruby-time/src/time.o
MRBC mrbgems/mruby-time/mrblib/time.rb
(snip)
CC mrbgems/mruby-socket/test/sockettest.c -> build/host/mrbgems/mruby-socket/test/sockettest.o
MRBC mrbgems/mruby-socket/test/addrinfo.rb
MRBC mrbgems/mruby-socket/test/basicsocket.rb
(snip)
```
#### After this patch:
```console
GEN build/presym
GEN build/presym.inc
CC src/array.c -> build/host/src/array.o
(snip)
GEN mrblib/*.rb -> build/host/mrblib/mrblib.c
MRBC mrblib/00class.rb
mrblib/10error.rb
(snip)
CC mrbgems/mruby-time/src/time.c -> build/host/mrbgems/mruby-time/src/time.o
GEN build/host/mrbgems/mruby-time/gem_init.c
MRBC mrbgems/mruby-time/mrblib/time.rb
(snip)
CC mrbgems/mruby-socket/test/sockettest.c -> build/host/mrbgems/mruby-socket/test/sockettest.o
GEN build/host/mrbgems/mruby-socket/gem_test.c
MRBC mrbgems/mruby-socket/test/addrinfo.rb
MRBC mrbgems/mruby-socket/test/basicsocket.rb
(snip)
```
If you confirm the gem is cdump safe, add `spec.enable_cdump` in
`mrbgem.rake` file. Some external gems e.g. `mruby-mgem-template` do
not work with cdump.
Yukihiro "Matz" Matsumoto