tools/lrama: bundle Lrama 0.6.9 (copied from CRuby source)

This commit is contained in:
Yukihiro "Matz" Matsumoto
2024-06-07 10:49:00 +09:00
parent d6da2d4760
commit fbe21a0321
80 changed files with 13465 additions and 3148 deletions
+1
View File
@@ -34,3 +34,4 @@ mruby-source-*.gem
node_modules
tags
!Gemfile.lock
y.tab.c
+1 -1
View File
@@ -249,7 +249,7 @@ module MRuby
def initialize(build)
super
@command = 'bison'
@command = "ruby #{MRUBY_ROOT}/tools/lrama/exe/lrama"
@compile_options = %q[-o "%{outfile}" "%{infile}"]
end
File diff suppressed because it is too large Load Diff
-5
View File
@@ -33,11 +33,6 @@ MRuby::Toolchain.new(:visualcpp) do |conf, _params|
archiver.archive_options = '/nologo /OUT:"%{outfile}" %{objs}'
end
conf.yacc do |yacc|
yacc.command = ENV['YACC'] || 'bison.exe'
yacc.compile_options = %q[-o "%{outfile}" "%{infile}"]
end
conf.gperf do |gperf|
gperf.command = 'gperf.exe'
gperf.compile_options = %q[-L ANSI-C -C -p -j1 -i 1 -g -o -t -N mrb_reserved_word -k"1,3,$" "%{infile}" > "%{outfile}"]
+12
View File
@@ -0,0 +1,12 @@
# LEGAL NOTICE INFORMATION
All the files in this distribution are covered under the MIT License except some files
mentioned below.
## GNU General Public License version 3
These files are licensed under the GNU General Public License version 3 or later. See these files for more information.
- template/bison/\_yacc.h
- template/bison/yacc.c
- template/bison/yacc.h
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2023 Yuichiro Kaneko
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+381
View File
@@ -0,0 +1,381 @@
# NEWS for Lrama
## Lrama 0.6.5 (2024-03-25)
### Typed Midrule Actions
User can specify the type of mid rule action by tag (`<bar>`) instead of specifying it with in an action.
```
primary: k_case expr_value terms?
{
$<val>$ = p->case_labels;
p->case_labels = Qnil;
}
case_body
k_end
{
...
}
```
can be written as
```
primary: k_case expr_value terms?
{
$$ = p->case_labels;
p->case_labels = Qnil;
}<val>
case_body
k_end
{
...
}
```
`%destructor` for midrule action is invoked only when tag is specified by Typed Midrule Actions.
Difference from Bison's Typed Midrule Actions is that tag is postposed in Lrama however it's preposed in Bison.
Bison supports this feature from 3.1.
## Lrama 0.6.4 (2024-03-22)
### Parameterizing rules (preceded, terminated, delimited)
Support `preceded`, `terminated` and `delimited` rules.
```
program: preceded(opening, X)
// Expanded to
program: preceded_opening_X
preceded_opening_X: opening X
```
```
program: terminated(X, closing)
// Expanded to
program: terminated_X_closing
terminated_X_closing: X closing
```
```
program: delimited(opening, X, closing)
// Expanded to
program: delimited_opening_X_closing
delimited_opening_X_closing: opening X closing
```
https://github.com/ruby/lrama/pull/382
### Support `%destructor` declaration
User can set codes for freeing semantic value resources by using `%destructor`.
In general, these resources are freed by actions or after parsing.
However if syntax error happens in parsing, these codes may not be executed.
Codes associated to `%destructor` are executed when semantic value is popped from the stack by an error.
```
%token <val1> NUM
%type <val2> expr2
%type <val3> expr
%destructor {
printf("destructor for val1: %d\n", $$);
} <val1> // printer for TAG
%destructor {
printf("destructor for val2: %d\n", $$);
} <val2>
%destructor {
printf("destructor for expr: %d\n", $$);
} expr // printer for symbol
```
Bison supports this feature from 1.75b.
https://github.com/ruby/lrama/pull/385
## Lrama 0.6.3 (2024-02-15)
### Bring Your Own Stack
Provide functionalities for Bring Your Own Stack.
Rubys Ripper library requires their own semantic value stack to manage Ruby Objects returned by user defined callback method. Currently Ripper uses semantic value stack (`yyvsa`) which is used by parser to manage Node. This hack introduces some limitation on Ripper. For example, Ripper can not execute semantic analysis depending on Node structure.
Lrama introduces two features to support another semantic value stack by parser generator users.
1. Callback entry points
User can emulate semantic value stack by these callbacks.
Lrama provides these five callbacks. Registered functions are called when each event happen. For example %after-shift function is called when shift happens on original semantic value stack.
- `%after-shift` function_name
- `%before-reduce` function_name
- `%after-reduce` function_name
- `%after-shift-error-token` function_name
- `%after-pop-stack` function_name
2. `$:n` variable to access index of each grammar symbols
User also needs to access semantic value of their stack in grammar action. `$:n` provides the way to access to it. `$:n` is translated to the minus index from the top of the stack.
For example
```
primary: k_if expr_value then compstmt if_tail k_end
{
/*% ripper: if!($:2, $:4, $:5) %*/
/* $:2 = -5, $:4 = -3, $:5 = -2. */
}
```
https://github.com/ruby/lrama/pull/367
## Lrama 0.6.2 (2024-01-27)
### %no-stdlib directive
If `%no-stdlib` directive is set, Lrama doesn't load Lrama standard library for
parameterizing rules, stdlib.y.
https://github.com/ruby/lrama/pull/344
## Lrama 0.6.1 (2024-01-13)
### Nested parameterizing rules
Allow to pass an instantiated rule to other parameterizing rules.
```
%rule constant(X) : X
;
%rule option(Y) : /* empty */
| Y
;
%%
program : option(constant(number)) // Nested rule
;
%%
```
Allow to use nested parameterizing rules when define parameterizing rules.
```
%rule option(x) : /* empty */
| X
;
%rule double(Y) : Y Y
;
%rule double_opt(A) : option(double(A)) // Nested rule
;
%%
program : double_opt(number)
;
%%
```
https://github.com/ruby/lrama/pull/337
## Lrama 0.6.0 (2023-12-25)
### User defined parameterizing rules
Allow to define parameterizing rule by `%rule` directive.
```
%rule pair(X, Y): X Y { $$ = $1 + $2; }
;
%%
program: stmt
;
stmt: pair(ODD, EVEN) <num>
| pair(EVEN, ODD) <num>
;
```
https://github.com/ruby/lrama/pull/285
## Lrama 0.5.11 (2023-12-02)
### Type specification of parameterizing rules
Allow to specify type of rules by specifying tag, `<i>` in below example.
Tag is post-modification style.
```
%union {
int i;
}
%%
program : option(number) <i>
| number_alias? <i>
;
```
https://github.com/ruby/lrama/pull/272
## Lrama 0.5.10 (2023-11-18)
### Parameterizing rules (option, nonempty_list, list)
Support function call style parameterizing rules for `option`, `nonempty_list` and `list`.
https://github.com/ruby/lrama/pull/197
### Parameterizing rules (separated_list)
Support `separated_list` and `separated_nonempty_list` parameterizing rules.
```
program: separated_list(',', number)
// Expanded to
program: separated_list_number
separated_list_number: ε
separated_list_number: separated_nonempty_list_number
separated_nonempty_list_number: number
separated_nonempty_list_number: separated_nonempty_list_number ',' number
```
```
program: separated_nonempty_list(',', number)
// Expanded to
program: separated_nonempty_list_number
separated_nonempty_list_number: number
separated_nonempty_list_number: separated_nonempty_list_number ',' number
```
https://github.com/ruby/lrama/pull/204
## Lrama 0.5.9 (2023-11-05)
### Parameterizing rules (suffix)
Parameterizing rules are template of rules.
It's very common pattern to write "list" grammar rule like:
```
opt_args: /* none */
| args
;
args: arg
| args arg
```
Lrama supports these suffixes:
- `?`: option
- `+`: nonempty list
- `*`: list
Idea of Parameterizing rules comes from Menhir LR(1) parser generator (https://gallium.inria.fr/~fpottier/menhir/manual.html#sec32).
https://github.com/ruby/lrama/pull/181
## Lrama 0.5.7 (2023-10-23)
### Racc parser
Replace Lrama's parser from hand written parser to LR parser generated by Racc.
Lrama uses `--embedded` option to generate LR parser because Racc is changed from default gem to bundled gem by Ruby 3.3 (https://github.com/ruby/lrama/pull/132).
https://github.com/ruby/lrama/pull/62
## Lrama 0.5.4 (2023-08-17)
### Runtime configuration for error recovery
Meke error recovery function configurable on runtime by two new macros.
- `YYMAXREPAIR`: Expected to return max length of repair operations. `%parse-param` is passed to this function.
- `YYERROR_RECOVERY_ENABLED`: Expected to return bool value to determine error recovery is enabled or not. `%parse-param` is passed to this function.
https://github.com/ruby/lrama/pull/74
## Lrama 0.5.3 (2023-08-05)
### Error Recovery
Support token insert base Error Recovery.
`-e` option is needed to generate parser with error recovery functions.
https://github.com/ruby/lrama/pull/44
## Lrama 0.5.2 (2023-06-14)
### Named References
Instead of positional references like `$1` or `$$`,
named references allow to access to symbol by name.
```
primary: k_class cpath superclass bodystmt k_end
{
$primary = new_class($cpath, $bodystmt, $superclass);
}
```
Alias name can be declared.
```
expr[result]: expr[ex-left] '+' expr[ex.right]
{
$result = $[ex-left] + $[ex.right];
}
```
Bison supports this feature from 2.5.
### Add parse params to some macros and functions
`%parse-param` are added to these macros and functions to remove ytab.sed hack from Ruby.
- `YY_LOCATION_PRINT`
- `YY_SYMBOL_PRINT`
- `yy_stack_print`
- `YY_STACK_PRINT`
- `YY_REDUCE_PRINT`
- `yysyntax_error`
https://github.com/ruby/lrama/pull/40
See also: https://github.com/ruby/ruby/pull/7807
## Lrama 0.5.0 (2023-05-17)
### stdin mode
When `-` is given as grammar file name, reads the grammar source from STDIN, and takes the next argument as the input file name. This mode helps pre-process a grammar source.
https://github.com/ruby/lrama/pull/8
## Lrama 0.4.0 (2023-05-13)
This is the first version migrated to Ruby.
This version generates "parse.c" compatible with Bison 3.8.2.
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env ruby
$LOAD_PATH << File.join(__dir__, "../lib")
require "lrama"
Lrama::Command.new.run(ARGV.dup)
+17
View File
@@ -0,0 +1,17 @@
require "lrama/bitmap"
require "lrama/command"
require "lrama/context"
require "lrama/counterexamples"
require "lrama/digraph"
require "lrama/grammar"
require "lrama/lexer"
require "lrama/option_parser"
require "lrama/options"
require "lrama/output"
require "lrama/parser"
require "lrama/report"
require "lrama/state"
require "lrama/states"
require "lrama/states_reporter"
require "lrama/version"
require "lrama/warning"
+29
View File
@@ -0,0 +1,29 @@
module Lrama
module Bitmap
def self.from_array(ary)
bit = 0
ary.each do |int|
bit |= (1 << int)
end
bit
end
def self.to_array(int)
a = []
i = 0
while int > 0 do
if int & 1 == 1
a << i
end
i += 1
int >>= 1
end
a
end
end
end
+73
View File
@@ -0,0 +1,73 @@
module Lrama
class Command
LRAMA_LIB = File.realpath(File.join(File.dirname(__FILE__)))
STDLIB_FILE_PATH = File.join(LRAMA_LIB, 'grammar', 'stdlib.y')
def run(argv)
begin
options = OptionParser.new.parse(argv)
rescue => e
message = e.message
message = message.gsub(/.+/, "\e[1m\\&\e[m") if Exception.to_tty?
abort message
end
Report::Duration.enable if options.trace_opts[:time]
warning = Lrama::Warning.new
text = options.y.read
options.y.close if options.y != STDIN
begin
grammar = Lrama::Parser.new(text, options.grammar_file, options.debug).parse
unless grammar.no_stdlib
stdlib_grammar = Lrama::Parser.new(File.read(STDLIB_FILE_PATH), STDLIB_FILE_PATH, options.debug).parse
grammar.insert_before_parameterizing_rules(stdlib_grammar.parameterizing_rules)
end
grammar.prepare
grammar.validate!
rescue => e
raise e if options.debug
message = e.message
message = message.gsub(/.+/, "\e[1m\\&\e[m") if Exception.to_tty?
abort message
end
states = Lrama::States.new(grammar, warning, trace_state: (options.trace_opts[:automaton] || options.trace_opts[:closure]))
states.compute
context = Lrama::Context.new(states)
if options.report_file
reporter = Lrama::StatesReporter.new(states)
File.open(options.report_file, "w+") do |f|
reporter.report(f, **options.report_opts)
end
end
if options.trace_opts && options.trace_opts[:rules]
puts "Grammar rules:"
puts grammar.rules
end
if options.trace_opts && options.trace_opts[:actions]
puts "Grammar rules with actions:"
grammar.rules.each { |rule| puts rule.with_actions }
end
File.open(options.outfile, "w+") do |f|
Lrama::Output.new(
out: f,
output_file_path: options.outfile,
template_name: options.skeleton,
grammar_file_path: options.grammar_file,
header_file_path: options.header_file,
context: context,
grammar: grammar,
error_recovery: options.error_recovery,
).render
end
if warning.has_error?
exit false
end
end
end
end
+497
View File
@@ -0,0 +1,497 @@
require "lrama/report/duration"
module Lrama
# This is passed to a template
class Context
include Report::Duration
ErrorActionNumber = -Float::INFINITY
BaseMin = -Float::INFINITY
# TODO: It might be better to pass `states` to Output directly?
attr_reader :states, :yylast, :yypact_ninf, :yytable_ninf, :yydefact, :yydefgoto
def initialize(states)
@states = states
@yydefact = nil
@yydefgoto = nil
# Array of array
@_actions = []
compute_tables
end
# enum yytokentype
def yytokentype
@states.terms.reject do |term|
0 < term.token_id && term.token_id < 128
end.map do |term|
[term.id.s_value, term.token_id, term.display_name]
end.unshift(["YYEMPTY", -2, nil])
end
# enum yysymbol_kind_t
def yysymbol_kind_t
@states.symbols.map do |sym|
[sym.enum_name, sym.number, sym.comment]
end.unshift(["YYSYMBOL_YYEMPTY", -2, nil])
end
# State number of final (accepted) state
def yyfinal
@states.states.find do |state|
state.items.find do |item|
item.lhs.accept_symbol? && item.end_of_rule?
end
end.id
end
# Number of terms
def yyntokens
@states.terms.count
end
# Number of nterms
def yynnts
@states.nterms.count
end
# Number of rules
def yynrules
@states.rules.count
end
# Number of states
def yynstates
@states.states.count
end
# Last token number
def yymaxutok
@states.terms.map(&:token_id).max
end
# YYTRANSLATE
#
# yytranslate is a mapping from token id to symbol number
def yytranslate
# 2 is YYSYMBOL_YYUNDEF
a = Array.new(yymaxutok, 2)
@states.terms.each do |term|
a[term.token_id] = term.number
end
return a
end
def yytranslate_inverted
a = Array.new(@states.symbols.count, @states.undef_symbol.token_id)
@states.terms.each do |term|
a[term.number] = term.token_id
end
return a
end
# Mapping from rule number to line number of the rule is defined.
# Dummy rule is appended as the first element whose value is 0
# because 0 means error in yydefact.
def yyrline
a = [0]
@states.rules.each do |rule|
a << rule.lineno
end
return a
end
# Mapping from symbol number to its name
def yytname
@states.symbols.sort_by(&:number).map do |sym|
sym.display_name
end
end
def yypact
@base[0...yynstates]
end
def yypgoto
@base[yynstates..-1]
end
def yytable
@table
end
def yycheck
@check
end
def yystos
@states.states.map do |state|
state.accessing_symbol.number
end
end
# Mapping from rule number to symbol number of LHS.
# Dummy rule is appended as the first element whose value is 0
# because 0 means error in yydefact.
def yyr1
a = [0]
@states.rules.each do |rule|
a << rule.lhs.number
end
return a
end
# Mapping from rule number to length of RHS.
# Dummy rule is appended as the first element whose value is 0
# because 0 means error in yydefact.
def yyr2
a = [0]
@states.rules.each do |rule|
a << rule.rhs.count
end
return a
end
private
# Compute these
#
# See also: "src/tables.c" of Bison.
#
# * yydefact
# * yydefgoto
# * yypact and yypgoto
# * yytable
# * yycheck
# * yypact_ninf
# * yytable_ninf
def compute_tables
report_duration(:compute_yydefact) { compute_yydefact }
report_duration(:compute_yydefgoto) { compute_yydefgoto }
report_duration(:sort_actions) { sort_actions }
# debug_sorted_actions
report_duration(:compute_packed_table) { compute_packed_table }
end
def vectors_count
@states.states.count + @states.nterms.count
end
# In compressed table, rule 0 is appended as an error case
# and reduce is represented as minus number.
def rule_id_to_action_number(rule_id)
(rule_id + 1) * -1
end
# Symbol number is assigned to term first then nterm.
# This method calculates sequence_number for nterm.
def nterm_number_to_sequence_number(nterm_number)
nterm_number - @states.terms.count
end
# Vector is states + nterms
def nterm_number_to_vector_number(nterm_number)
@states.states.count + (nterm_number - @states.terms.count)
end
def compute_yydefact
# Default action (shift/reduce/error) for each state.
# Index is state id, value is `rule id + 1` of a default reduction.
@yydefact = Array.new(@states.states.count, 0)
@states.states.each do |state|
# Action number means
#
# * number = 0, default action
# * number = -Float::INFINITY, error by %nonassoc
# * number > 0, shift then move to state "number"
# * number < 0, reduce by "-number" rule. Rule "number" is already added by 1.
actions = Array.new(@states.terms.count, 0)
if state.reduces.map(&:selected_look_ahead).any? {|la| !la.empty? }
# Iterate reduces with reverse order so that first rule is used.
state.reduces.reverse_each do |reduce|
reduce.look_ahead.each do |term|
actions[term.number] = rule_id_to_action_number(reduce.rule.id)
end
end
end
# Shift is selected when S/R conflict exists.
state.selected_term_transitions.each do |shift, next_state|
actions[shift.next_sym.number] = next_state.id
end
state.resolved_conflicts.select do |conflict|
conflict.which == :error
end.each do |conflict|
actions[conflict.symbol.number] = ErrorActionNumber
end
# If default_reduction_rule, replace default_reduction_rule in
# actions with zero.
if state.default_reduction_rule
actions.map! do |e|
if e == rule_id_to_action_number(state.default_reduction_rule.id)
0
else
e
end
end
end
# If no default_reduction_rule, default behavior is an
# error then replace ErrorActionNumber with zero.
if !state.default_reduction_rule
actions.map! do |e|
if e == ErrorActionNumber
0
else
e
end
end
end
s = actions.each_with_index.map do |n, i|
[i, n]
end.reject do |i, n|
# Remove default_reduction_rule entries
n == 0
end
if s.count != 0
# Entry of @_actions is an array of
#
# * State id
# * Array of tuple, [from, to] where from is term number and to is action.
# * The number of "Array of tuple" used by sort_actions
# * "width" used by sort_actions
@_actions << [state.id, s, s.count, s.last[0] - s.first[0] + 1]
end
@yydefact[state.id] = state.default_reduction_rule ? state.default_reduction_rule.id + 1 : 0
end
end
def compute_yydefgoto
# Default GOTO (nterm transition) for each nterm.
# Index is sequence number of nterm, value is state id
# of a default nterm transition destination.
@yydefgoto = Array.new(@states.nterms.count, 0)
# Mapping from nterm to next_states
nterm_to_next_states = {}
@states.states.each do |state|
state.nterm_transitions.each do |shift, next_state|
key = shift.next_sym
nterm_to_next_states[key] ||= []
nterm_to_next_states[key] << [state, next_state] # [from_state, to_state]
end
end
@states.nterms.each do |nterm|
if !(states = nterm_to_next_states[nterm])
default_goto = 0
not_default_gotos = []
else
default_state = states.map(&:last).group_by {|s| s }.max_by {|_, v| v.count }.first
default_goto = default_state.id
not_default_gotos = []
states.each do |from_state, to_state|
next if to_state.id == default_goto
not_default_gotos << [from_state.id, to_state.id]
end
end
k = nterm_number_to_sequence_number(nterm.number)
@yydefgoto[k] = default_goto
if not_default_gotos.count != 0
v = nterm_number_to_vector_number(nterm.number)
# Entry of @_actions is an array of
#
# * Nterm number as vector number
# * Array of tuple, [from, to] where from is state number and to is state number.
# * The number of "Array of tuple" used by sort_actions
# * "width" used by sort_actions
@_actions << [v, not_default_gotos, not_default_gotos.count, not_default_gotos.last[0] - not_default_gotos.first[0] + 1]
end
end
end
def sort_actions
# This is not same with #sort_actions
#
# @sorted_actions = @_actions.sort_by do |_, _, count, width|
# [-width, -count]
# end
@sorted_actions = []
@_actions.each do |action|
if @sorted_actions.empty?
@sorted_actions << action
next
end
j = @sorted_actions.count - 1
_state_id, _froms_and_tos, count, width = action
while (j >= 0) do
case
when @sorted_actions[j][3] < width
j -= 1
when @sorted_actions[j][3] == width && @sorted_actions[j][2] < count
j -= 1
else
break
end
end
@sorted_actions.insert(j + 1, action)
end
end
def debug_sorted_actions
ary = Array.new
@sorted_actions.each do |state_id, froms_and_tos, count, width|
ary[state_id] = [state_id, froms_and_tos, count, width]
end
print sprintf("table_print:\n\n")
print sprintf("order [\n")
vectors_count.times do |i|
print sprintf("%d, ", @sorted_actions[i] ? @sorted_actions[i][0] : 0)
print "\n" if i % 10 == 9
end
print sprintf("]\n\n")
print sprintf("width [\n")
vectors_count.times do |i|
print sprintf("%d, ", ary[i] ? ary[i][3] : 0)
print "\n" if i % 10 == 9
end
print sprintf("]\n\n")
print sprintf("tally [\n")
vectors_count.times do |i|
print sprintf("%d, ", ary[i] ? ary[i][2] : 0)
print "\n" if i % 10 == 9
end
print sprintf("]\n\n")
end
def compute_packed_table
# yypact and yypgoto
@base = Array.new(vectors_count, BaseMin)
# yytable
@table = []
# yycheck
@check = []
# Key is froms_and_tos, value is index position
pushed = {}
userd_res = {}
lowzero = 0
high = 0
@sorted_actions.each do |state_id, froms_and_tos, _, _|
if (res = pushed[froms_and_tos])
@base[state_id] = res
next
end
res = lowzero - froms_and_tos.first[0]
while true do
ok = true
froms_and_tos.each do |from, to|
loc = res + from
if @table[loc]
# If the cell of table is set, can not use the cell.
ok = false
break
end
end
if ok && userd_res[res]
ok = false
end
if ok
break
else
res += 1
end
end
loc = 0
froms_and_tos.each do |from, to|
loc = res + from
@table[loc] = to
@check[loc] = from
end
while (@table[lowzero]) do
lowzero += 1
end
high = loc if high < loc
@base[state_id] = res
pushed[froms_and_tos] = res
userd_res[res] = true
end
@yylast = high
# replace_ninf
@yypact_ninf = (@base.reject {|i| i == BaseMin } + [0]).min - 1
@base.map! do |i|
case i
when BaseMin
@yypact_ninf
else
i
end
end
@yytable_ninf = (@table.compact.reject {|i| i == ErrorActionNumber } + [0]).min - 1
@table.map! do |i|
case i
when nil
0
when ErrorActionNumber
@yytable_ninf
else
i
end
end
@check.map! do |i|
case i
when nil
-1
else
i
end
end
end
end
end
+286
View File
@@ -0,0 +1,286 @@
require "set"
require "lrama/counterexamples/derivation"
require "lrama/counterexamples/example"
require "lrama/counterexamples/path"
require "lrama/counterexamples/production_path"
require "lrama/counterexamples/start_path"
require "lrama/counterexamples/state_item"
require "lrama/counterexamples/transition_path"
require "lrama/counterexamples/triple"
module Lrama
# See: https://www.cs.cornell.edu/andru/papers/cupex/cupex.pdf
# 4. Constructing Nonunifying Counterexamples
class Counterexamples
attr_reader :transitions, :productions
def initialize(states)
@states = states
setup_transitions
setup_productions
end
def to_s
"#<Counterexamples>"
end
alias :inspect :to_s
def compute(conflict_state)
conflict_state.conflicts.flat_map do |conflict|
case conflict.type
when :shift_reduce
shift_reduce_example(conflict_state, conflict)
when :reduce_reduce
reduce_reduce_examples(conflict_state, conflict)
end
end.compact
end
private
def setup_transitions
# Hash [StateItem, Symbol] => StateItem
@transitions = {}
# Hash [StateItem, Symbol] => Set(StateItem)
@reverse_transitions = {}
@states.states.each do |src_state|
trans = {}
src_state.transitions.each do |shift, next_state|
trans[shift.next_sym] = next_state
end
src_state.items.each do |src_item|
next if src_item.end_of_rule?
sym = src_item.next_sym
dest_state = trans[sym]
dest_state.kernels.each do |dest_item|
next unless (src_item.rule == dest_item.rule) && (src_item.position + 1 == dest_item.position)
src_state_item = StateItem.new(src_state, src_item)
dest_state_item = StateItem.new(dest_state, dest_item)
@transitions[[src_state_item, sym]] = dest_state_item
key = [dest_state_item, sym]
@reverse_transitions[key] ||= Set.new
@reverse_transitions[key] << src_state_item
end
end
end
end
def setup_productions
# Hash [StateItem] => Set(Item)
@productions = {}
# Hash [State, Symbol] => Set(Item). Symbol is nterm
@reverse_productions = {}
@states.states.each do |state|
# LHS => Set(Item)
h = {}
state.closure.each do |item|
sym = item.lhs
h[sym] ||= Set.new
h[sym] << item
end
state.items.each do |item|
next if item.end_of_rule?
next if item.next_sym.term?
sym = item.next_sym
state_item = StateItem.new(state, item)
key = [state, sym]
@productions[state_item] = h[sym]
@reverse_productions[key] ||= Set.new
@reverse_productions[key] << item
end
end
end
def shift_reduce_example(conflict_state, conflict)
conflict_symbol = conflict.symbols.first
shift_conflict_item = conflict_state.items.find { |item| item.next_sym == conflict_symbol }
path2 = shortest_path(conflict_state, conflict.reduce.item, conflict_symbol)
path1 = find_shift_conflict_shortest_path(path2, conflict_state, shift_conflict_item)
Example.new(path1, path2, conflict, conflict_symbol, self)
end
def reduce_reduce_examples(conflict_state, conflict)
conflict_symbol = conflict.symbols.first
path1 = shortest_path(conflict_state, conflict.reduce1.item, conflict_symbol)
path2 = shortest_path(conflict_state, conflict.reduce2.item, conflict_symbol)
Example.new(path1, path2, conflict, conflict_symbol, self)
end
def find_shift_conflict_shortest_path(reduce_path, conflict_state, conflict_item)
state_items = find_shift_conflict_shortest_state_items(reduce_path, conflict_state, conflict_item)
build_paths_from_state_items(state_items)
end
def find_shift_conflict_shortest_state_items(reduce_path, conflict_state, conflict_item)
target_state_item = StateItem.new(conflict_state, conflict_item)
result = [target_state_item]
reversed_reduce_path = reduce_path.to_a.reverse
# Index for state_item
i = 0
while (path = reversed_reduce_path[i])
# Index for prev_state_item
j = i + 1
_j = j
while (prev_path = reversed_reduce_path[j])
if prev_path.production?
j += 1
else
break
end
end
state_item = path.to
prev_state_item = prev_path&.to
if target_state_item == state_item || target_state_item.item.start_item?
result.concat(reversed_reduce_path[_j..-1].map(&:to))
break
end
if target_state_item.item.beginning_of_rule?
queue = []
queue << [target_state_item]
# Find reverse production
while (sis = queue.shift)
si = sis.last
# Reach to start state
if si.item.start_item?
sis.shift
result.concat(sis)
target_state_item = si
break
end
if !si.item.beginning_of_rule?
key = [si, si.item.previous_sym]
@reverse_transitions[key].each do |prev_target_state_item|
next if prev_target_state_item.state != prev_state_item.state
sis.shift
result.concat(sis)
result << prev_target_state_item
target_state_item = prev_target_state_item
i = j
queue.clear
break
end
else
key = [si.state, si.item.lhs]
@reverse_productions[key].each do |item|
state_item = StateItem.new(si.state, item)
queue << (sis + [state_item])
end
end
end
else
# Find reverse transition
key = [target_state_item, target_state_item.item.previous_sym]
@reverse_transitions[key].each do |prev_target_state_item|
next if prev_target_state_item.state != prev_state_item.state
result << prev_target_state_item
target_state_item = prev_target_state_item
i = j
break
end
end
end
result.reverse
end
def build_paths_from_state_items(state_items)
state_items.zip([nil] + state_items).map do |si, prev_si|
case
when prev_si.nil?
StartPath.new(si)
when si.item.beginning_of_rule?
ProductionPath.new(prev_si, si)
else
TransitionPath.new(prev_si, si)
end
end
end
def shortest_path(conflict_state, conflict_reduce_item, conflict_term)
# queue: is an array of [Triple, [Path]]
queue = []
visited = {}
start_state = @states.states.first
raise "BUG: Start state should be just one kernel." if start_state.kernels.count != 1
start = Triple.new(start_state, start_state.kernels.first, Set.new([@states.eof_symbol]))
queue << [start, [StartPath.new(start.state_item)]]
while true
triple, paths = queue.shift
next if visited[triple]
visited[triple] = true
# Found
if triple.state == conflict_state && triple.item == conflict_reduce_item && triple.l.include?(conflict_term)
return paths
end
# transition
triple.state.transitions.each do |shift, next_state|
next unless triple.item.next_sym && triple.item.next_sym == shift.next_sym
next_state.kernels.each do |kernel|
next if kernel.rule != triple.item.rule
t = Triple.new(next_state, kernel, triple.l)
queue << [t, paths + [TransitionPath.new(triple.state_item, t.state_item)]]
end
end
# production step
triple.state.closure.each do |item|
next unless triple.item.next_sym && triple.item.next_sym == item.lhs
l = follow_l(triple.item, triple.l)
t = Triple.new(triple.state, item, l)
queue << [t, paths + [ProductionPath.new(triple.state_item, t.state_item)]]
end
break if queue.empty?
end
return nil
end
def follow_l(item, current_l)
# 1. follow_L (A -> X1 ... Xn-1 • Xn) = L
# 2. follow_L (A -> X1 ... Xk • Xk+1 Xk+2 ... Xn) = {Xk+2} if Xk+2 is a terminal
# 3. follow_L (A -> X1 ... Xk • Xk+1 Xk+2 ... Xn) = FIRST(Xk+2) if Xk+2 is a nonnullable nonterminal
# 4. follow_L (A -> X1 ... Xk • Xk+1 Xk+2 ... Xn) = FIRST(Xk+2) + follow_L (A -> X1 ... Xk+1 • Xk+2 ... Xn) if Xk+2 is a nullable nonterminal
case
when item.number_of_rest_symbols == 1
current_l
when item.next_next_sym.term?
Set.new([item.next_next_sym])
when !item.next_next_sym.nullable
item.next_next_sym.first_set
else
item.next_next_sym.first_set + follow_l(item.new_by_next_position, current_l)
end
end
end
end
@@ -0,0 +1,63 @@
module Lrama
class Counterexamples
class Derivation
attr_reader :item, :left, :right
attr_writer :right
def initialize(item, left, right = nil)
@item = item
@left = left
@right = right
end
def to_s
"#<Derivation(#{item.display_name})>"
end
alias :inspect :to_s
def render_strings_for_report
result = []
_render_for_report(self, 0, result, 0)
result.map(&:rstrip)
end
def render_for_report
render_strings_for_report.join("\n")
end
private
def _render_for_report(derivation, offset, strings, index)
item = derivation.item
if strings[index]
strings[index] << " " * (offset - strings[index].length)
else
strings[index] = " " * offset
end
str = strings[index]
str << "#{item.rule_id}: #{item.symbols_before_dot.map(&:display_name).join(" ")} "
if derivation.left
len = str.length
str << "#{item.next_sym.display_name}"
length = _render_for_report(derivation.left, len, strings, index + 1)
# I want String#ljust!
str << " " * (length - str.length)
else
str << "#{item.symbols_after_dot.map(&:display_name).join(" ")} "
return str.length
end
if derivation.right&.left
length = _render_for_report(derivation.right.left, str.length, strings, index + 1)
str << "#{item.symbols_after_dot[1..-1].map(&:display_name).join(" ")} "
str << " " * (length - str.length) if length > str.length
elsif item.next_next_sym
str << "#{item.symbols_after_dot[1..-1].map(&:display_name).join(" ")} "
end
return str.length
end
end
end
end
@@ -0,0 +1,124 @@
module Lrama
class Counterexamples
class Example
attr_reader :path1, :path2, :conflict, :conflict_symbol
# path1 is shift conflict when S/R conflict
# path2 is always reduce conflict
def initialize(path1, path2, conflict, conflict_symbol, counterexamples)
@path1 = path1
@path2 = path2
@conflict = conflict
@conflict_symbol = conflict_symbol
@counterexamples = counterexamples
end
def type
@conflict.type
end
def path1_item
@path1.last.to.item
end
def path2_item
@path2.last.to.item
end
def derivations1
@derivations1 ||= _derivations(path1)
end
def derivations2
@derivations2 ||= _derivations(path2)
end
private
def _derivations(paths)
derivation = nil
current = :production
lookahead_sym = paths.last.to.item.end_of_rule? ? @conflict_symbol : nil
paths.reverse_each do |path|
item = path.to.item
case current
when :production
case path
when StartPath
derivation = Derivation.new(item, derivation)
current = :start
when TransitionPath
derivation = Derivation.new(item, derivation)
current = :transition
when ProductionPath
derivation = Derivation.new(item, derivation)
current = :production
end
if lookahead_sym && item.next_next_sym && item.next_next_sym.first_set.include?(lookahead_sym)
state_item = @counterexamples.transitions[[path.to, item.next_sym]]
derivation2 = find_derivation_for_symbol(state_item, lookahead_sym)
derivation.right = derivation2
lookahead_sym = nil
end
when :transition
case path
when StartPath
derivation = Derivation.new(item, derivation)
current = :start
when TransitionPath
# ignore
current = :transition
when ProductionPath
# ignore
current = :production
end
else
raise "BUG: Unknown #{current}"
end
break if current == :start
end
derivation
end
def find_derivation_for_symbol(state_item, sym)
queue = []
queue << [state_item]
while (sis = queue.shift)
si = sis.last
next_sym = si.item.next_sym
if next_sym == sym
derivation = nil
sis.reverse_each do |si|
derivation = Derivation.new(si.item, derivation)
end
return derivation
end
if next_sym.nterm? && next_sym.first_set.include?(sym)
@counterexamples.productions[si].each do |next_item|
next if next_item.empty_rule?
next_si = StateItem.new(si.state, next_item)
next if sis.include?(next_si)
queue << (sis + [next_si])
end
if next_sym.nullable
next_si = @counterexamples.transitions[[si, next_sym]]
queue << (sis + [next_si])
end
end
end
end
end
end
end
@@ -0,0 +1,23 @@
module Lrama
class Counterexamples
class Path
def initialize(from_state_item, to_state_item)
@from_state_item = from_state_item
@to_state_item = to_state_item
end
def from
@from_state_item
end
def to
@to_state_item
end
def to_s
"#<Path(#{type})>"
end
alias :inspect :to_s
end
end
end
@@ -0,0 +1,17 @@
module Lrama
class Counterexamples
class ProductionPath < Path
def type
:production
end
def transition?
false
end
def production?
true
end
end
end
end
@@ -0,0 +1,21 @@
module Lrama
class Counterexamples
class StartPath < Path
def initialize(to_state_item)
super nil, to_state_item
end
def type
:start
end
def transition?
false
end
def production?
false
end
end
end
end
@@ -0,0 +1,6 @@
module Lrama
class Counterexamples
class StateItem < Struct.new(:state, :item)
end
end
end
@@ -0,0 +1,17 @@
module Lrama
class Counterexamples
class TransitionPath < Path
def type
:transition
end
def transition?
true
end
def production?
false
end
end
end
end
@@ -0,0 +1,21 @@
module Lrama
class Counterexamples
# s: state
# itm: item within s
# l: precise lookahead set
class Triple < Struct.new(:s, :itm, :l)
alias :state :s
alias :item :itm
alias :precise_lookahead_set :l
def state_item
StateItem.new(state, item)
end
def inspect
"#{state.inspect}. #{item.display_name}. #{l.map(&:id).map(&:s_value)}"
end
alias :to_s :inspect
end
end
end
+51
View File
@@ -0,0 +1,51 @@
module Lrama
# Algorithm Digraph of https://dl.acm.org/doi/pdf/10.1145/69622.357187 (P. 625)
class Digraph
def initialize(sets, relation, base_function)
# X in the paper
@sets = sets
# R in the paper
@relation = relation
# F' in the paper
@base_function = base_function
# S in the paper
@stack = []
# N in the paper
@h = Hash.new(0)
# F in the paper
@result = {}
end
def compute
@sets.each do |x|
next if @h[x] != 0
traverse(x)
end
return @result
end
private
def traverse(x)
@stack.push(x)
d = @stack.count
@h[x] = d
@result[x] = @base_function[x] # F x = F' x
@relation[x]&.each do |y|
traverse(y) if @h[y] == 0
@h[x] = [@h[x], @h[y]].min
@result[x] |= @result[y] # F x = F x + F y
end
if @h[x] == d
while (z = @stack.pop) do
@h[z] = Float::INFINITY
break if z == x
@result[z] = @result[x] # F (Top of S) = F x
end
end
end
end
end
+381
View File
@@ -0,0 +1,381 @@
require "forwardable"
require "lrama/grammar/auxiliary"
require "lrama/grammar/binding"
require "lrama/grammar/code"
require "lrama/grammar/counter"
require "lrama/grammar/destructor"
require "lrama/grammar/error_token"
require "lrama/grammar/parameterizing_rule"
require "lrama/grammar/percent_code"
require "lrama/grammar/precedence"
require "lrama/grammar/printer"
require "lrama/grammar/reference"
require "lrama/grammar/rule"
require "lrama/grammar/rule_builder"
require "lrama/grammar/symbol"
require "lrama/grammar/symbols"
require "lrama/grammar/type"
require "lrama/grammar/union"
require "lrama/lexer"
module Lrama
# Grammar is the result of parsing an input grammar file
class Grammar
extend Forwardable
attr_reader :percent_codes, :eof_symbol, :error_symbol, :undef_symbol, :accept_symbol, :aux
attr_accessor :union, :expect,
:printers, :error_tokens,
:lex_param, :parse_param, :initial_action,
:after_shift, :before_reduce, :after_reduce, :after_shift_error_token, :after_pop_stack,
:symbols_resolver, :types,
:rules, :rule_builders,
:sym_to_rules, :no_stdlib
def_delegators "@symbols_resolver", :symbols, :nterms, :terms, :add_nterm, :add_term,
:find_symbol_by_number!, :find_symbol_by_id!, :token_to_symbol,
:find_symbol_by_s_value!, :fill_symbol_number, :fill_nterm_type,
:fill_printer, :fill_destructor, :fill_error_token, :sort_by_number!
def initialize(rule_counter)
@rule_counter = rule_counter
# Code defined by "%code"
@percent_codes = []
@printers = []
@destructors = []
@error_tokens = []
@symbols_resolver = Grammar::Symbols::Resolver.new
@types = []
@rule_builders = []
@rules = []
@sym_to_rules = {}
@parameterizing_rule_resolver = ParameterizingRule::Resolver.new
@empty_symbol = nil
@eof_symbol = nil
@error_symbol = nil
@undef_symbol = nil
@accept_symbol = nil
@aux = Auxiliary.new
@no_stdlib = false
append_special_symbols
end
def add_percent_code(id:, code:)
@percent_codes << PercentCode.new(id.s_value, code.s_value)
end
def add_destructor(ident_or_tags:, token_code:, lineno:)
@destructors << Destructor.new(ident_or_tags: ident_or_tags, token_code: token_code, lineno: lineno)
end
def add_printer(ident_or_tags:, token_code:, lineno:)
@printers << Printer.new(ident_or_tags: ident_or_tags, token_code: token_code, lineno: lineno)
end
def add_error_token(ident_or_tags:, token_code:, lineno:)
@error_tokens << ErrorToken.new(ident_or_tags: ident_or_tags, token_code: token_code, lineno: lineno)
end
def add_type(id:, tag:)
@types << Type.new(id: id, tag: tag)
end
def add_nonassoc(sym, precedence)
set_precedence(sym, Precedence.new(type: :nonassoc, precedence: precedence))
end
def add_left(sym, precedence)
set_precedence(sym, Precedence.new(type: :left, precedence: precedence))
end
def add_right(sym, precedence)
set_precedence(sym, Precedence.new(type: :right, precedence: precedence))
end
def add_precedence(sym, precedence)
set_precedence(sym, Precedence.new(type: :precedence, precedence: precedence))
end
def set_precedence(sym, precedence)
raise "" if sym.nterm?
sym.precedence = precedence
end
def set_union(code, lineno)
@union = Union.new(code: code, lineno: lineno)
end
def add_rule_builder(builder)
@rule_builders << builder
end
def add_parameterizing_rule(rule)
@parameterizing_rule_resolver.add_parameterizing_rule(rule)
end
def parameterizing_rules
@parameterizing_rule_resolver.rules
end
def insert_before_parameterizing_rules(rules)
@parameterizing_rule_resolver.rules = rules + @parameterizing_rule_resolver.rules
end
def prologue_first_lineno=(prologue_first_lineno)
@aux.prologue_first_lineno = prologue_first_lineno
end
def prologue=(prologue)
@aux.prologue = prologue
end
def epilogue_first_lineno=(epilogue_first_lineno)
@aux.epilogue_first_lineno = epilogue_first_lineno
end
def epilogue=(epilogue)
@aux.epilogue = epilogue
end
def prepare
normalize_rules
collect_symbols
set_lhs_and_rhs
fill_default_precedence
fill_symbols
fill_sym_to_rules
compute_nullable
compute_first_set
end
# TODO: More validation methods
#
# * Validation for no_declared_type_reference
def validate!
@symbols_resolver.validate!
validate_rule_lhs_is_nterm!
end
def find_rules_by_symbol!(sym)
find_rules_by_symbol(sym) || (raise "Rules for #{sym} not found")
end
def find_rules_by_symbol(sym)
@sym_to_rules[sym.number]
end
private
def compute_nullable
@rules.each do |rule|
case
when rule.empty_rule?
rule.nullable = true
when rule.rhs.any?(&:term)
rule.nullable = false
else
# noop
end
end
while true do
rs = @rules.select {|e| e.nullable.nil? }
nts = nterms.select {|e| e.nullable.nil? }
rule_count_1 = rs.count
nterm_count_1 = nts.count
rs.each do |rule|
if rule.rhs.all?(&:nullable)
rule.nullable = true
end
end
nts.each do |nterm|
find_rules_by_symbol!(nterm).each do |rule|
if rule.nullable
nterm.nullable = true
end
end
end
rule_count_2 = @rules.count {|e| e.nullable.nil? }
nterm_count_2 = nterms.count {|e| e.nullable.nil? }
if (rule_count_1 == rule_count_2) && (nterm_count_1 == nterm_count_2)
break
end
end
rules.select {|r| r.nullable.nil? }.each do |rule|
rule.nullable = false
end
nterms.select {|e| e.nullable.nil? }.each do |nterm|
nterm.nullable = false
end
end
def compute_first_set
terms.each do |term|
term.first_set = Set.new([term]).freeze
term.first_set_bitmap = Lrama::Bitmap.from_array([term.number])
end
nterms.each do |nterm|
nterm.first_set = Set.new([]).freeze
nterm.first_set_bitmap = Lrama::Bitmap.from_array([])
end
while true do
changed = false
@rules.each do |rule|
rule.rhs.each do |r|
if rule.lhs.first_set_bitmap | r.first_set_bitmap != rule.lhs.first_set_bitmap
changed = true
rule.lhs.first_set_bitmap = rule.lhs.first_set_bitmap | r.first_set_bitmap
end
break unless r.nullable
end
end
break unless changed
end
nterms.each do |nterm|
nterm.first_set = Lrama::Bitmap.to_array(nterm.first_set_bitmap).map do |number|
find_symbol_by_number!(number)
end.to_set
end
end
def setup_rules
@rule_builders.each do |builder|
builder.setup_rules(@parameterizing_rule_resolver)
end
end
def append_special_symbols
# YYEMPTY (token_id: -2, number: -2) is added when a template is evaluated
# term = add_term(id: Token.new(Token::Ident, "YYEMPTY"), token_id: -2)
# term.number = -2
# @empty_symbol = term
# YYEOF
term = add_term(id: Lrama::Lexer::Token::Ident.new(s_value: "YYEOF"), alias_name: "\"end of file\"", token_id: 0)
term.number = 0
term.eof_symbol = true
@eof_symbol = term
# YYerror
term = add_term(id: Lrama::Lexer::Token::Ident.new(s_value: "YYerror"), alias_name: "error")
term.number = 1
term.error_symbol = true
@error_symbol = term
# YYUNDEF
term = add_term(id: Lrama::Lexer::Token::Ident.new(s_value: "YYUNDEF"), alias_name: "\"invalid token\"")
term.number = 2
term.undef_symbol = true
@undef_symbol = term
# $accept
term = add_nterm(id: Lrama::Lexer::Token::Ident.new(s_value: "$accept"))
term.accept_symbol = true
@accept_symbol = term
end
def normalize_rules
# Add $accept rule to the top of rules
lineno = @rule_builders.first ? @rule_builders.first.line : 0
@rules << Rule.new(id: @rule_counter.increment, _lhs: @accept_symbol.id, _rhs: [@rule_builders.first.lhs, @eof_symbol.id], token_code: nil, lineno: lineno)
setup_rules
@rule_builders.each do |builder|
builder.rules.each do |rule|
add_nterm(id: rule._lhs, tag: rule.lhs_tag)
@rules << rule
end
end
@rules.sort_by!(&:id)
end
# Collect symbols from rules
def collect_symbols
@rules.flat_map(&:_rhs).each do |s|
case s
when Lrama::Lexer::Token::Char
add_term(id: s)
when Lrama::Lexer::Token
# skip
else
raise "Unknown class: #{s}"
end
end
end
def set_lhs_and_rhs
@rules.each do |rule|
rule.lhs = token_to_symbol(rule._lhs) if rule._lhs
rule.rhs = rule._rhs.map do |t|
token_to_symbol(t)
end
end
end
# Rule inherits precedence from the last term in RHS.
#
# https://www.gnu.org/software/bison/manual/html_node/How-Precedence.html
def fill_default_precedence
@rules.each do |rule|
# Explicitly specified precedence has the highest priority
next if rule.precedence_sym
precedence_sym = nil
rule.rhs.each do |sym|
precedence_sym = sym if sym.term?
end
rule.precedence_sym = precedence_sym
end
end
def fill_symbols
fill_symbol_number
fill_nterm_type(@types)
fill_printer(@printers)
fill_destructor(@destructors)
fill_error_token(@error_tokens)
sort_by_number!
end
def fill_sym_to_rules
@rules.each do |rule|
key = rule.lhs.number
@sym_to_rules[key] ||= []
@sym_to_rules[key] << rule
end
end
def validate_rule_lhs_is_nterm!
errors = []
rules.each do |rule|
next if rule.lhs.nterm?
errors << "[BUG] LHS of #{rule} (line: #{rule.lineno}) is term. It should be nterm."
end
return if errors.empty?
raise errors.join("\n")
end
end
end
@@ -0,0 +1,7 @@
module Lrama
class Grammar
# Grammar file information not used by States but by Output
class Auxiliary < Struct.new(:prologue_first_lineno, :prologue, :epilogue_first_lineno, :epilogue, keyword_init: true)
end
end
end
+24
View File
@@ -0,0 +1,24 @@
module Lrama
class Grammar
class Binding
attr_reader :actual_args, :count
def initialize(parameterizing_rule, actual_args)
@parameters = parameterizing_rule.parameters
@actual_args = actual_args
@parameter_to_arg = @parameters.zip(actual_args).map do |param, arg|
[param.s_value, arg]
end.to_h
end
def resolve_symbol(symbol)
if symbol.is_a?(Lexer::Token::InstantiateRule)
resolved_args = symbol.args.map { |arg| resolve_symbol(arg) }
Lrama::Lexer::Token::InstantiateRule.new(s_value: symbol.s_value, location: symbol.location, args: resolved_args, lhs_tag: symbol.lhs_tag)
else
@parameter_to_arg[symbol.s_value] || symbol
end
end
end
end
end
+51
View File
@@ -0,0 +1,51 @@
require "forwardable"
require "lrama/grammar/code/destructor_code"
require "lrama/grammar/code/initial_action_code"
require "lrama/grammar/code/no_reference_code"
require "lrama/grammar/code/printer_code"
require "lrama/grammar/code/rule_action"
module Lrama
class Grammar
class Code
extend Forwardable
def_delegators "token_code", :s_value, :line, :column, :references
attr_reader :type, :token_code
def initialize(type:, token_code:)
@type = type
@token_code = token_code
end
def ==(other)
self.class == other.class &&
self.type == other.type &&
self.token_code == other.token_code
end
# $$, $n, @$, @n are translated to C code
def translated_code
t_code = s_value.dup
references.reverse_each do |ref|
first_column = ref.first_column
last_column = ref.last_column
str = reference_to_c(ref)
t_code[first_column...last_column] = str
end
return t_code
end
private
def reference_to_c(ref)
raise NotImplementedError.new("#reference_to_c is not implemented")
end
end
end
end
@@ -0,0 +1,40 @@
module Lrama
class Grammar
class Code
class DestructorCode < Code
def initialize(type:, token_code:, tag:)
super(type: type, token_code: token_code)
@tag = tag
end
private
# * ($$) *yyvaluep
# * (@$) *yylocationp
# * ($:$) error
# * ($1) error
# * (@1) error
# * ($:1) error
def reference_to_c(ref)
case
when ref.type == :dollar && ref.name == "$" # $$
member = @tag.member
"((*yyvaluep).#{member})"
when ref.type == :at && ref.name == "$" # @$
"(*yylocationp)"
when ref.type == :index && ref.name == "$" # $:$
raise "$:#{ref.value} can not be used in #{type}."
when ref.type == :dollar # $n
raise "$#{ref.value} can not be used in #{type}."
when ref.type == :at # @n
raise "@#{ref.value} can not be used in #{type}."
when ref.type == :index # $:n
raise "$:#{ref.value} can not be used in #{type}."
else
raise "Unexpected. #{self}, #{ref}"
end
end
end
end
end
end
@@ -0,0 +1,34 @@
module Lrama
class Grammar
class Code
class InitialActionCode < Code
private
# * ($$) yylval
# * (@$) yylloc
# * ($:$) error
# * ($1) error
# * (@1) error
# * ($:1) error
def reference_to_c(ref)
case
when ref.type == :dollar && ref.name == "$" # $$
"yylval"
when ref.type == :at && ref.name == "$" # @$
"yylloc"
when ref.type == :index && ref.name == "$" # $:$
raise "$:#{ref.value} can not be used in initial_action."
when ref.type == :dollar # $n
raise "$#{ref.value} can not be used in initial_action."
when ref.type == :at # @n
raise "@#{ref.value} can not be used in initial_action."
when ref.type == :index # $:n
raise "$:#{ref.value} can not be used in initial_action."
else
raise "Unexpected. #{self}, #{ref}"
end
end
end
end
end
end
@@ -0,0 +1,28 @@
module Lrama
class Grammar
class Code
class NoReferenceCode < Code
private
# * ($$) error
# * (@$) error
# * ($:$) error
# * ($1) error
# * (@1) error
# * ($:1) error
def reference_to_c(ref)
case
when ref.type == :dollar # $$, $n
raise "$#{ref.value} can not be used in #{type}."
when ref.type == :at # @$, @n
raise "@#{ref.value} can not be used in #{type}."
when ref.type == :index # $:$, $:n
raise "$:#{ref.value} can not be used in #{type}."
else
raise "Unexpected. #{self}, #{ref}"
end
end
end
end
end
end
@@ -0,0 +1,40 @@
module Lrama
class Grammar
class Code
class PrinterCode < Code
def initialize(type:, token_code:, tag:)
super(type: type, token_code: token_code)
@tag = tag
end
private
# * ($$) *yyvaluep
# * (@$) *yylocationp
# * ($:$) error
# * ($1) error
# * (@1) error
# * ($:1) error
def reference_to_c(ref)
case
when ref.type == :dollar && ref.name == "$" # $$
member = @tag.member
"((*yyvaluep).#{member})"
when ref.type == :at && ref.name == "$" # @$
"(*yylocationp)"
when ref.type == :index && ref.name == "$" # $:$
raise "$:#{ref.value} can not be used in #{type}."
when ref.type == :dollar # $n
raise "$#{ref.value} can not be used in #{type}."
when ref.type == :at # @n
raise "@#{ref.value} can not be used in #{type}."
when ref.type == :index # $:n
raise "$:#{ref.value} can not be used in #{type}."
else
raise "Unexpected. #{self}, #{ref}"
end
end
end
end
end
end
@@ -0,0 +1,88 @@
module Lrama
class Grammar
class Code
class RuleAction < Code
def initialize(type:, token_code:, rule:)
super(type: type, token_code: token_code)
@rule = rule
end
private
# * ($$) yyval
# * (@$) yyloc
# * ($:$) error
# * ($1) yyvsp[i]
# * (@1) yylsp[i]
# * ($:1) i - 1
#
#
# Consider a rule like
#
# class: keyword_class { $1 } tSTRING { $2 + $3 } keyword_end { $class = $1 + $keyword_end }
#
# For the semantic action of original rule:
#
# "Rule" class: keyword_class { $1 } tSTRING { $2 + $3 } keyword_end { $class = $1 + $keyword_end }
# "Position in grammar" $1 $2 $3 $4 $5
# "Index for yyvsp" -4 -3 -2 -1 0
# "$:n" $:1 $:2 $:3 $:4 $:5
# "index of $:n" -5 -4 -3 -2 -1
#
#
# For the first midrule action:
#
# "Rule" class: keyword_class { $1 } tSTRING { $2 + $3 } keyword_end { $class = $1 + $keyword_end }
# "Position in grammar" $1
# "Index for yyvsp" 0
# "$:n" $:1
def reference_to_c(ref)
case
when ref.type == :dollar && ref.name == "$" # $$
tag = ref.ex_tag || lhs.tag
raise_tag_not_found_error(ref) unless tag
"(yyval.#{tag.member})"
when ref.type == :at && ref.name == "$" # @$
"(yyloc)"
when ref.type == :index && ref.name == "$" # $:$
raise "$:$ is not supported"
when ref.type == :dollar # $n
i = -position_in_rhs + ref.index
tag = ref.ex_tag || rhs[ref.index - 1].tag
raise_tag_not_found_error(ref) unless tag
"(yyvsp[#{i}].#{tag.member})"
when ref.type == :at # @n
i = -position_in_rhs + ref.index
"(yylsp[#{i}])"
when ref.type == :index # $:n
i = -position_in_rhs + ref.index
"(#{i} - 1)"
else
raise "Unexpected. #{self}, #{ref}"
end
end
def position_in_rhs
# If rule is not derived rule, User Code is only action at
# the end of rule RHS. In such case, the action is located on
# `@rule.rhs.count`.
@rule.position_in_original_rule_rhs || @rule.rhs.count
end
# If this is midrule action, RHS is a RHS of the original rule.
def rhs
(@rule.original_rule || @rule).rhs
end
# Unlike `rhs`, LHS is always a LHS of the rule.
def lhs
@rule.lhs
end
def raise_tag_not_found_error(ref)
raise "Tag is not specified for '$#{ref.value}' in '#{@rule}'"
end
end
end
end
end
+15
View File
@@ -0,0 +1,15 @@
module Lrama
class Grammar
class Counter
def initialize(number)
@number = number
end
def increment
n = @number
@number += 1
n
end
end
end
end
@@ -0,0 +1,9 @@
module Lrama
class Grammar
class Destructor < Struct.new(:ident_or_tags, :token_code, :lineno, keyword_init: true)
def translated_code(tag)
Code::DestructorCode.new(type: :destructor, token_code: token_code, tag: tag).translated_code
end
end
end
end
@@ -0,0 +1,9 @@
module Lrama
class Grammar
class ErrorToken < Struct.new(:ident_or_tags, :token_code, :lineno, keyword_init: true)
def translated_code(tag)
Code::PrinterCode.new(type: :error_token, token_code: token_code, tag: tag).translated_code
end
end
end
end
@@ -0,0 +1,3 @@
require_relative 'parameterizing_rule/resolver'
require_relative 'parameterizing_rule/rhs'
require_relative 'parameterizing_rule/rule'
@@ -0,0 +1,56 @@
module Lrama
class Grammar
class ParameterizingRule
class Resolver
attr_accessor :rules, :created_lhs_list
def initialize
@rules = []
@created_lhs_list = []
end
def add_parameterizing_rule(rule)
@rules << rule
end
def find_rule(token)
select_rules(@rules, token).last
end
def find_inline(token)
@rules.select { |rule| rule.name == token.s_value && rule.is_inline }.last
end
def created_lhs(lhs_s_value)
@created_lhs_list.reverse.find { |created_lhs| created_lhs.s_value == lhs_s_value }
end
private
def select_rules(rules, token)
rules = select_not_inline_rules(rules)
rules = select_rules_by_name(rules, token.rule_name)
rules = rules.select { |rule| rule.required_parameters_count == token.args_count }
if rules.empty?
raise "Invalid number of arguments. `#{token.rule_name}`"
else
rules
end
end
def select_not_inline_rules(rules)
rules.select { |rule| !rule.is_inline }
end
def select_rules_by_name(rules, rule_name)
rules = rules.select { |rule| rule.name == rule_name }
if rules.empty?
raise "Parameterizing rule does not exist. `#{rule_name}`"
else
rules
end
end
end
end
end
end
@@ -0,0 +1,37 @@
module Lrama
class Grammar
class ParameterizingRule
class Rhs
attr_accessor :symbols, :user_code, :precedence_sym
def initialize
@symbols = []
@user_code = nil
@precedence_sym = nil
end
def resolve_user_code(bindings)
return unless user_code
var_to_arg = {}
symbols.each do |sym|
resolved_sym = bindings.resolve_symbol(sym)
if resolved_sym != sym
var_to_arg[sym.s_value] = resolved_sym.s_value
end
end
var_to_arg.each do |var, arg|
user_code.references.each do |ref|
if ref.name == var
ref.name = arg
end
end
end
return user_code
end
end
end
end
end
@@ -0,0 +1,18 @@
module Lrama
class Grammar
class ParameterizingRule
class Rule
attr_reader :name, :parameters, :rhs_list, :required_parameters_count, :tag, :is_inline
def initialize(name, parameters, rhs_list, tag: nil, is_inline: false)
@name = name
@parameters = parameters
@rhs_list = rhs_list
@tag = tag
@is_inline = is_inline
@required_parameters_count = parameters.count
end
end
end
end
end
@@ -0,0 +1,12 @@
module Lrama
class Grammar
class PercentCode
attr_reader :name, :code
def initialize(name, code)
@name = name
@code = code
end
end
end
end
@@ -0,0 +1,11 @@
module Lrama
class Grammar
class Precedence < Struct.new(:type, :precedence, keyword_init: true)
include Comparable
def <=>(other)
self.precedence <=> other.precedence
end
end
end
end
+9
View File
@@ -0,0 +1,9 @@
module Lrama
class Grammar
class Printer < Struct.new(:ident_or_tags, :token_code, :lineno, keyword_init: true)
def translated_code(tag)
Code::PrinterCode.new(type: :printer, token_code: token_code, tag: tag).translated_code
end
end
end
end
@@ -0,0 +1,14 @@
module Lrama
class Grammar
# type: :dollar or :at
# name: String (e.g. $$, $foo, $expr.right)
# number: Integer (e.g. $1)
# index: Integer
# ex_tag: "$<tag>1" (Optional)
class Reference < Struct.new(:type, :name, :number, :index, :ex_tag, :first_column, :last_column, keyword_init: true)
def value
name || number
end
end
end
end
+60
View File
@@ -0,0 +1,60 @@
module Lrama
class Grammar
# _rhs holds original RHS element. Use rhs to refer to Symbol.
class Rule < Struct.new(:id, :_lhs, :lhs, :lhs_tag, :_rhs, :rhs, :token_code, :position_in_original_rule_rhs, :nullable, :precedence_sym, :lineno, keyword_init: true)
attr_accessor :original_rule
def ==(other)
self.class == other.class &&
self.lhs == other.lhs &&
self.lhs_tag == other.lhs_tag &&
self.rhs == other.rhs &&
self.token_code == other.token_code &&
self.position_in_original_rule_rhs == other.position_in_original_rule_rhs &&
self.nullable == other.nullable &&
self.precedence_sym == other.precedence_sym &&
self.lineno == other.lineno
end
# TODO: Change this to display_name
def to_s
l = lhs.id.s_value
r = empty_rule? ? "ε" : rhs.map {|r| r.id.s_value }.join(" ")
"#{l} -> #{r}"
end
# Used by #user_actions
def as_comment
l = lhs.id.s_value
r = empty_rule? ? "%empty" : rhs.map(&:display_name).join(" ")
"#{l}: #{r}"
end
def with_actions
"#{to_s} {#{token_code&.s_value}}"
end
# opt_nl: ε <-- empty_rule
# | '\n' <-- not empty_rule
def empty_rule?
rhs.empty?
end
def precedence
precedence_sym&.precedence
end
def initial_rule?
id == 0
end
def translated_code
return nil unless token_code
Code::RuleAction.new(type: :rule_action, token_code: token_code, rule: self).translated_code
end
end
end
end
@@ -0,0 +1,268 @@
module Lrama
class Grammar
class RuleBuilder
attr_accessor :lhs, :line
attr_reader :lhs_tag, :rhs, :user_code, :precedence_sym
def initialize(rule_counter, midrule_action_counter, position_in_original_rule_rhs = nil, lhs_tag: nil, skip_preprocess_references: false)
@rule_counter = rule_counter
@midrule_action_counter = midrule_action_counter
@position_in_original_rule_rhs = position_in_original_rule_rhs
@skip_preprocess_references = skip_preprocess_references
@lhs = nil
@lhs_tag = lhs_tag
@rhs = []
@user_code = nil
@precedence_sym = nil
@line = nil
@rules = []
@rule_builders_for_parameterizing_rules = []
@rule_builders_for_derived_rules = []
@rule_builders_for_inline_rules = []
@parameterizing_rules = []
@inline_rules = []
@midrule_action_rules = []
end
def add_rhs(rhs)
if !@line
@line = rhs.line
end
flush_user_code
@rhs << rhs
end
def user_code=(user_code)
if !@line
@line = user_code&.line
end
flush_user_code
@user_code = user_code
end
def precedence_sym=(precedence_sym)
flush_user_code
@precedence_sym = precedence_sym
end
def complete_input
freeze_rhs
end
def setup_rules(parameterizing_rule_resolver)
preprocess_references unless @skip_preprocess_references
if rhs.any? { |token| parameterizing_rule_resolver.find_inline(token) }
resolve_inline(parameterizing_rule_resolver)
else
process_rhs(parameterizing_rule_resolver)
end
build_rules
end
def rules
@parameterizing_rules + @inline_rules + @midrule_action_rules + @rules
end
private
def freeze_rhs
@rhs.freeze
end
def preprocess_references
numberize_references
end
def build_rules
tokens = @replaced_rhs
if tokens
rule = Rule.new(
id: @rule_counter.increment, _lhs: lhs, _rhs: tokens, lhs_tag: lhs_tag, token_code: user_code,
position_in_original_rule_rhs: @position_in_original_rule_rhs, precedence_sym: precedence_sym, lineno: line
)
@rules = [rule]
@parameterizing_rules = @rule_builders_for_parameterizing_rules.map do |rule_builder|
rule_builder.rules
end.flatten
@midrule_action_rules = @rule_builders_for_derived_rules.map do |rule_builder|
rule_builder.rules
end.flatten
@midrule_action_rules.each do |r|
r.original_rule = rule
end
else
@inline_rules = @rule_builders_for_inline_rules.map do |rule_builder|
rule_builder.rules
end.flatten
end
end
# rhs is a mixture of variety type of tokens like `Ident`, `InstantiateRule`, `UserCode` and so on.
# `#process_rhs` replaces some kind of tokens to `Ident` so that all `@replaced_rhs` are `Ident` or `Char`.
def process_rhs(parameterizing_rule_resolver)
return if @replaced_rhs
@replaced_rhs = []
rhs.each_with_index do |token, i|
case token
when Lrama::Lexer::Token::Char
@replaced_rhs << token
when Lrama::Lexer::Token::Ident
@replaced_rhs << token
when Lrama::Lexer::Token::InstantiateRule
parameterizing_rule = parameterizing_rule_resolver.find_rule(token)
raise "Unexpected token. #{token}" unless parameterizing_rule
bindings = Binding.new(parameterizing_rule, token.args)
lhs_s_value = lhs_s_value(token, bindings)
if (created_lhs = parameterizing_rule_resolver.created_lhs(lhs_s_value))
@replaced_rhs << created_lhs
else
lhs_token = Lrama::Lexer::Token::Ident.new(s_value: lhs_s_value, location: token.location)
@replaced_rhs << lhs_token
parameterizing_rule_resolver.created_lhs_list << lhs_token
parameterizing_rule.rhs_list.each do |r|
rule_builder = RuleBuilder.new(@rule_counter, @midrule_action_counter, lhs_tag: token.lhs_tag || parameterizing_rule.tag)
rule_builder.lhs = lhs_token
r.symbols.each { |sym| rule_builder.add_rhs(bindings.resolve_symbol(sym)) }
rule_builder.line = line
rule_builder.precedence_sym = r.precedence_sym
rule_builder.user_code = r.resolve_user_code(bindings)
rule_builder.complete_input
rule_builder.setup_rules(parameterizing_rule_resolver)
@rule_builders_for_parameterizing_rules << rule_builder
end
end
when Lrama::Lexer::Token::UserCode
prefix = token.referred ? "@" : "$@"
tag = token.tag || lhs_tag
new_token = Lrama::Lexer::Token::Ident.new(s_value: prefix + @midrule_action_counter.increment.to_s)
@replaced_rhs << new_token
rule_builder = RuleBuilder.new(@rule_counter, @midrule_action_counter, i, lhs_tag: tag, skip_preprocess_references: true)
rule_builder.lhs = new_token
rule_builder.user_code = token
rule_builder.complete_input
rule_builder.setup_rules(parameterizing_rule_resolver)
@rule_builders_for_derived_rules << rule_builder
else
raise "Unexpected token. #{token}"
end
end
end
def lhs_s_value(token, bindings)
s_values = token.args.map do |arg|
resolved = bindings.resolve_symbol(arg)
if resolved.is_a?(Lexer::Token::InstantiateRule)
[resolved.s_value, resolved.args.map(&:s_value)]
else
resolved.s_value
end
end
"#{token.rule_name}_#{s_values.join('_')}"
end
def resolve_inline(parameterizing_rule_resolver)
rhs.each_with_index do |token, i|
if inline_rule = parameterizing_rule_resolver.find_inline(token)
inline_rule.rhs_list.each_with_index do |inline_rhs|
rule_builder = RuleBuilder.new(@rule_counter, @midrule_action_counter, lhs_tag: lhs_tag, skip_preprocess_references: true)
resolve_inline_rhs(rule_builder, inline_rhs, i)
rule_builder.lhs = lhs
rule_builder.line = line
rule_builder.user_code = replace_inline_user_code(inline_rhs, i)
rule_builder.complete_input
rule_builder.setup_rules(parameterizing_rule_resolver)
@rule_builders_for_inline_rules << rule_builder
end
end
end
end
def resolve_inline_rhs(rule_builder, inline_rhs, index)
rhs.each_with_index do |token, i|
if index == i
inline_rhs.symbols.each { |sym| rule_builder.add_rhs(sym) }
else
rule_builder.add_rhs(token)
end
end
end
def replace_inline_user_code(inline_rhs, index)
return user_code if inline_rhs.user_code.nil?
return user_code if user_code.nil?
code = user_code.s_value.gsub(/\$#{index + 1}/, inline_rhs.user_code.s_value)
Lrama::Lexer::Token::UserCode.new(s_value: code, location: user_code.location)
end
def numberize_references
# Bison n'th component is 1-origin
(rhs + [user_code]).compact.each.with_index(1) do |token, i|
next unless token.is_a?(Lrama::Lexer::Token::UserCode)
token.references.each do |ref|
ref_name = ref.name
if ref_name
if ref_name == '$'
ref.name = '$'
else
candidates = ([lhs] + rhs).each_with_index.select {|token, _i| token.referred_by?(ref_name) }
if candidates.size >= 2
token.invalid_ref(ref, "Referring symbol `#{ref_name}` is duplicated.")
end
unless (referring_symbol = candidates.first)
token.invalid_ref(ref, "Referring symbol `#{ref_name}` is not found.")
end
if referring_symbol[1] == 0 # Refers to LHS
ref.name = '$'
else
ref.number = referring_symbol[1]
end
end
end
if ref.number
# TODO: When Inlining is implemented, for example, if `$1` is expanded to multiple RHS tokens,
# `$2` needs to access `$2 + n` to actually access it. So, after the Inlining implementation,
# it needs resolves from number to index.
ref.index = ref.number
end
# TODO: Need to check index of @ too?
next if ref.type == :at
if ref.index
# TODO: Prohibit $0 even so Bison allows it?
# See: https://www.gnu.org/software/bison/manual/html_node/Actions.html
token.invalid_ref(ref, "Can not refer following component. #{ref.index} >= #{i}.") if ref.index >= i
rhs[ref.index - 1].referred = true
end
end
end
end
def flush_user_code
if (c = @user_code)
@rhs << c
@user_code = nil
end
end
end
end
end
+122
View File
@@ -0,0 +1,122 @@
/**********************************************************************
stdlib.y
This is lrama's standard library. It provides a number of
parameterizing rule definitions, such as options and lists,
that should be useful in a number of situations.
**********************************************************************/
// -------------------------------------------------------------------
// Options
/*
* program: option(number)
*
* =>
*
* program: option_number
* option_number: %empty
* option_number: number
*/
%rule option(X): /* empty */
| X
;
// -------------------------------------------------------------------
// Sequences
/*
* program: preceded(opening, X)
*
* =>
*
* program: preceded_opening_X
* preceded_opening_X: opening X
*/
%rule preceded(opening, X): opening X { $$ = $2; }
;
/*
* program: terminated(X, closing)
*
* =>
*
* program: terminated_X_closing
* terminated_X_closing: X closing
*/
%rule terminated(X, closing): X closing { $$ = $1; }
;
/*
* program: delimited(opening, X, closing)
*
* =>
*
* program: delimited_opening_X_closing
* delimited_opening_X_closing: opening X closing
*/
%rule delimited(opening, X, closing): opening X closing { $$ = $2; }
;
// -------------------------------------------------------------------
// Lists
/*
* program: list(number)
*
* =>
*
* program: list_number
* list_number: %empty
* list_number: list_number number
*/
%rule list(X): /* empty */
| list(X) X
;
/*
* program: nonempty_list(number)
*
* =>
*
* program: nonempty_list_number
* nonempty_list_number: number
* nonempty_list_number: nonempty_list_number number
*/
%rule nonempty_list(X): X
| nonempty_list(X) X
;
/*
* program: separated_nonempty_list(comma, number)
*
* =>
*
* program: separated_nonempty_list_comma_number
* separated_nonempty_list_comma_number: number
* separated_nonempty_list_comma_number: separated_nonempty_list_comma_number comma number
*/
%rule separated_nonempty_list(separator, X): X
| separated_nonempty_list(separator, X) separator X
;
/*
* program: separated_list(comma, number)
*
* =>
*
* program: separated_list_comma_number
* separated_list_comma_number: option_separated_nonempty_list_comma_number
* option_separated_nonempty_list_comma_number: %empty
* option_separated_nonempty_list_comma_number: separated_nonempty_list_comma_number
* separated_nonempty_list_comma_number: number
* separated_nonempty_list_comma_number: comma separated_nonempty_list_comma_number number
*/
%rule separated_list(separator, X): option(separated_nonempty_list(separator, X))
;
%%
%union{};
+103
View File
@@ -0,0 +1,103 @@
# Symbol is both of nterm and term
# `number` is both for nterm and term
# `token_id` is tokentype for term, internal sequence number for nterm
#
# TODO: Add validation for ASCII code range for Token::Char
module Lrama
class Grammar
class Symbol
attr_accessor :id, :alias_name, :tag, :number, :token_id, :nullable, :precedence,
:printer, :destructor, :error_token, :first_set, :first_set_bitmap
attr_reader :term
attr_writer :eof_symbol, :error_symbol, :undef_symbol, :accept_symbol
def initialize(id:, term:, alias_name: nil, number: nil, tag: nil, token_id: nil, nullable: nil, precedence: nil, printer: nil, destructor: nil)
@id = id
@alias_name = alias_name
@number = number
@tag = tag
@term = term
@token_id = token_id
@nullable = nullable
@precedence = precedence
@printer = printer
@destructor = destructor
end
def term?
term
end
def nterm?
!term
end
def eof_symbol?
!!@eof_symbol
end
def error_symbol?
!!@error_symbol
end
def undef_symbol?
!!@undef_symbol
end
def accept_symbol?
!!@accept_symbol
end
def display_name
alias_name || id.s_value
end
# name for yysymbol_kind_t
#
# See: b4_symbol_kind_base
# @type var name: String
def enum_name
case
when accept_symbol?
name = "YYACCEPT"
when eof_symbol?
name = "YYEOF"
when term? && id.is_a?(Lrama::Lexer::Token::Char)
name = number.to_s + display_name
when term? && id.is_a?(Lrama::Lexer::Token::Ident)
name = id.s_value
when nterm? && (id.s_value.include?("$") || id.s_value.include?("@"))
name = number.to_s + id.s_value
when nterm?
name = id.s_value
else
raise "Unexpected #{self}"
end
"YYSYMBOL_" + name.gsub(/\W+/, "_")
end
# comment for yysymbol_kind_t
def comment
case
when accept_symbol?
# YYSYMBOL_YYACCEPT
id.s_value
when eof_symbol?
# YYEOF
alias_name
when (term? && 0 < token_id && token_id < 128)
# YYSYMBOL_3_backslash_, YYSYMBOL_14_
alias_name || id.s_value
when id.s_value.include?("$") || id.s_value.include?("@")
# YYSYMBOL_21_1
id.s_value
else
# YYSYMBOL_keyword_class, YYSYMBOL_strings_1
alias_name || id.s_value
end
end
end
end
end
+1
View File
@@ -0,0 +1 @@
require_relative "symbols/resolver"
@@ -0,0 +1,293 @@
module Lrama
class Grammar
class Symbols
class Resolver
attr_reader :terms, :nterms
def initialize
@terms = []
@nterms = []
end
def symbols
@symbols ||= (@terms + @nterms)
end
def sort_by_number!
symbols.sort_by!(&:number)
end
def add_term(id:, alias_name: nil, tag: nil, token_id: nil, replace: false)
if token_id && (sym = find_symbol_by_token_id(token_id))
if replace
sym.id = id
sym.alias_name = alias_name
sym.tag = tag
end
return sym
end
if (sym = find_symbol_by_id(id))
return sym
end
@symbols = nil
term = Symbol.new(
id: id, alias_name: alias_name, number: nil, tag: tag,
term: true, token_id: token_id, nullable: false
)
@terms << term
term
end
def add_nterm(id:, alias_name: nil, tag: nil)
return if find_symbol_by_id(id)
@symbols = nil
nterm = Symbol.new(
id: id, alias_name: alias_name, number: nil, tag: tag,
term: false, token_id: nil, nullable: nil,
)
@nterms << nterm
nterm
end
def find_symbol_by_s_value(s_value)
symbols.find { |s| s.id.s_value == s_value }
end
def find_symbol_by_s_value!(s_value)
find_symbol_by_s_value(s_value) || (raise "Symbol not found. value: `#{s_value}`")
end
def find_symbol_by_id(id)
symbols.find do |s|
s.id == id || s.alias_name == id.s_value
end
end
def find_symbol_by_id!(id)
find_symbol_by_id(id) || (raise "Symbol not found. #{id}")
end
def find_symbol_by_token_id(token_id)
symbols.find {|s| s.token_id == token_id }
end
def find_symbol_by_number!(number)
sym = symbols[number]
raise "Symbol not found. number: `#{number}`" unless sym
raise "[BUG] Symbol number mismatch. #{number}, #{sym}" if sym.number != number
sym
end
def fill_symbol_number
# YYEMPTY = -2
# YYEOF = 0
# YYerror = 1
# YYUNDEF = 2
@number = 3
fill_terms_number
fill_nterms_number
end
def fill_nterm_type(types)
types.each do |type|
nterm = find_nterm_by_id!(type.id)
nterm.tag = type.tag
end
end
def fill_printer(printers)
symbols.each do |sym|
printers.each do |printer|
printer.ident_or_tags.each do |ident_or_tag|
case ident_or_tag
when Lrama::Lexer::Token::Ident
sym.printer = printer if sym.id == ident_or_tag
when Lrama::Lexer::Token::Tag
sym.printer = printer if sym.tag == ident_or_tag
else
raise "Unknown token type. #{printer}"
end
end
end
end
end
def fill_destructor(destructors)
symbols.each do |sym|
destructors.each do |destructor|
destructor.ident_or_tags.each do |ident_or_tag|
case ident_or_tag
when Lrama::Lexer::Token::Ident
sym.destructor = destructor if sym.id == ident_or_tag
when Lrama::Lexer::Token::Tag
sym.destructor = destructor if sym.tag == ident_or_tag
else
raise "Unknown token type. #{destructor}"
end
end
end
end
end
def fill_error_token(error_tokens)
symbols.each do |sym|
error_tokens.each do |token|
token.ident_or_tags.each do |ident_or_tag|
case ident_or_tag
when Lrama::Lexer::Token::Ident
sym.error_token = token if sym.id == ident_or_tag
when Lrama::Lexer::Token::Tag
sym.error_token = token if sym.tag == ident_or_tag
else
raise "Unknown token type. #{token}"
end
end
end
end
end
def token_to_symbol(token)
case token
when Lrama::Lexer::Token
find_symbol_by_id!(token)
else
raise "Unknown class: #{token}"
end
end
def validate!
validate_number_uniqueness!
validate_alias_name_uniqueness!
end
private
def find_nterm_by_id!(id)
@nterms.find do |s|
s.id == id
end || (raise "Symbol not found. #{id}")
end
def fill_terms_number
# Character literal in grammar file has
# token id corresponding to ASCII code by default,
# so start token_id from 256.
token_id = 256
@terms.each do |sym|
while used_numbers[@number] do
@number += 1
end
if sym.number.nil?
sym.number = @number
used_numbers[@number] = true
@number += 1
end
# If id is Token::Char, it uses ASCII code
if sym.token_id.nil?
if sym.id.is_a?(Lrama::Lexer::Token::Char)
# Ignore ' on the both sides
case sym.id.s_value[1..-2]
when "\\b"
sym.token_id = 8
when "\\f"
sym.token_id = 12
when "\\n"
sym.token_id = 10
when "\\r"
sym.token_id = 13
when "\\t"
sym.token_id = 9
when "\\v"
sym.token_id = 11
when "\""
sym.token_id = 34
when "'"
sym.token_id = 39
when "\\\\"
sym.token_id = 92
when /\A\\(\d+)\z/
unless (id = Integer($1, 8)).nil?
sym.token_id = id
else
raise "Unknown Char s_value #{sym}"
end
when /\A(.)\z/
unless (id = $1&.bytes&.first).nil?
sym.token_id = id
else
raise "Unknown Char s_value #{sym}"
end
else
raise "Unknown Char s_value #{sym}"
end
else
sym.token_id = token_id
token_id += 1
end
end
end
end
def fill_nterms_number
token_id = 0
@nterms.each do |sym|
while used_numbers[@number] do
@number += 1
end
if sym.number.nil?
sym.number = @number
used_numbers[@number] = true
@number += 1
end
if sym.token_id.nil?
sym.token_id = token_id
token_id += 1
end
end
end
def used_numbers
return @used_numbers if defined?(@used_numbers)
@used_numbers = {}
symbols.map(&:number).each do |n|
@used_numbers[n] = true
end
@used_numbers
end
def validate_number_uniqueness!
invalid = symbols.group_by(&:number).select do |number, syms|
syms.count > 1
end
return if invalid.empty?
raise "Symbol number is duplicated. #{invalid}"
end
def validate_alias_name_uniqueness!
invalid = symbols.select(&:alias_name).group_by(&:alias_name).select do |alias_name, syms|
syms.count > 1
end
return if invalid.empty?
raise "Symbol alias name is duplicated. #{invalid}"
end
end
end
end
end
+18
View File
@@ -0,0 +1,18 @@
module Lrama
class Grammar
class Type
attr_reader :id, :tag
def initialize(id:, tag:)
@id = id
@tag = tag
end
def ==(other)
self.class == other.class &&
self.id == other.id &&
self.tag == other.tag
end
end
end
end
+10
View File
@@ -0,0 +1,10 @@
module Lrama
class Grammar
class Union < Struct.new(:code, :lineno, keyword_init: true)
def braces_less_code
# Braces is already removed by lexer
code.s_value
end
end
end
end
+188
View File
@@ -0,0 +1,188 @@
require "strscan"
require "lrama/lexer/grammar_file"
require "lrama/lexer/location"
require "lrama/lexer/token"
module Lrama
class Lexer
attr_reader :head_line, :head_column, :line
attr_accessor :status, :end_symbol
SYMBOLS = ['%{', '%}', '%%', '{', '}', '\[', '\]', '\(', '\)', '\,', ':', '\|', ';']
PERCENT_TOKENS = %w(
%union
%token
%type
%left
%right
%nonassoc
%expect
%define
%require
%printer
%destructor
%lex-param
%parse-param
%initial-action
%precedence
%prec
%error-token
%before-reduce
%after-reduce
%after-shift-error-token
%after-shift
%after-pop-stack
%empty
%code
%rule
%no-stdlib
%inline
)
def initialize(grammar_file)
@grammar_file = grammar_file
@scanner = StringScanner.new(grammar_file.text)
@head_column = @head = @scanner.pos
@head_line = @line = 1
@status = :initial
@end_symbol = nil
end
def next_token
case @status
when :initial
lex_token
when :c_declaration
lex_c_code
end
end
def column
@scanner.pos - @head
end
def location
Location.new(
grammar_file: @grammar_file,
first_line: @head_line, first_column: @head_column,
last_line: line, last_column: column
)
end
def lex_token
while !@scanner.eos? do
case
when @scanner.scan(/\n/)
newline
when @scanner.scan(/\s+/)
# noop
when @scanner.scan(/\/\*/)
lex_comment
when @scanner.scan(/\/\/.*(?<newline>\n)?/)
newline if @scanner[:newline]
else
break
end
end
reset_first_position
case
when @scanner.eos?
return
when @scanner.scan(/#{SYMBOLS.join('|')}/)
return [@scanner.matched, @scanner.matched]
when @scanner.scan(/#{PERCENT_TOKENS.join('|')}/)
return [@scanner.matched, @scanner.matched]
when @scanner.scan(/[\?\+\*]/)
return [@scanner.matched, @scanner.matched]
when @scanner.scan(/<\w+>/)
return [:TAG, Lrama::Lexer::Token::Tag.new(s_value: @scanner.matched, location: location)]
when @scanner.scan(/'.'/)
return [:CHARACTER, Lrama::Lexer::Token::Char.new(s_value: @scanner.matched, location: location)]
when @scanner.scan(/'\\\\'|'\\b'|'\\t'|'\\f'|'\\r'|'\\n'|'\\v'|'\\13'/)
return [:CHARACTER, Lrama::Lexer::Token::Char.new(s_value: @scanner.matched, location: location)]
when @scanner.scan(/".*?"/)
return [:STRING, %Q(#{@scanner.matched})]
when @scanner.scan(/\d+/)
return [:INTEGER, Integer(@scanner.matched)]
when @scanner.scan(/([a-zA-Z_.][-a-zA-Z0-9_.]*)/)
token = Lrama::Lexer::Token::Ident.new(s_value: @scanner.matched, location: location)
type =
if @scanner.check(/\s*(\[\s*[a-zA-Z_.][-a-zA-Z0-9_.]*\s*\])?\s*:/)
:IDENT_COLON
else
:IDENTIFIER
end
return [type, token]
else
raise ParseError, "Unexpected token: #{@scanner.peek(10).chomp}."
end
end
def lex_c_code
nested = 0
code = ''
reset_first_position
while !@scanner.eos? do
case
when @scanner.scan(/{/)
code += @scanner.matched
nested += 1
when @scanner.scan(/}/)
if nested == 0 && @end_symbol == '}'
@scanner.unscan
return [:C_DECLARATION, Lrama::Lexer::Token::UserCode.new(s_value: code, location: location)]
else
code += @scanner.matched
nested -= 1
end
when @scanner.check(/#{@end_symbol}/)
return [:C_DECLARATION, Lrama::Lexer::Token::UserCode.new(s_value: code, location: location)]
when @scanner.scan(/\n/)
code += @scanner.matched
newline
when @scanner.scan(/".*?"/)
code += %Q(#{@scanner.matched})
@line += @scanner.matched.count("\n")
when @scanner.scan(/'.*?'/)
code += %Q(#{@scanner.matched})
when @scanner.scan(/[^\"'\{\}\n]+/)
code += @scanner.matched
when @scanner.scan(/#{Regexp.escape(@end_symbol)}/)
code += @scanner.matched
else
code += @scanner.getch
end
end
raise ParseError, "Unexpected code: #{code}."
end
private
def lex_comment
while !@scanner.eos? do
case
when @scanner.scan(/\n/)
newline
when @scanner.scan(/\*\//)
return
else
@scanner.getch
end
end
end
def reset_first_position
@head_line = line
@head_column = column
end
def newline
@line += 1
@head = @scanner.pos
end
end
end
@@ -0,0 +1,31 @@
module Lrama
class Lexer
class GrammarFile
class Text < String
def inspect
length <= 50 ? super : "#{self[0..47]}...".inspect
end
end
attr_reader :path, :text
def initialize(path, text)
@path = path
@text = Text.new(text).freeze
end
def inspect
"<#{self.class}: @path=#{path}, @text=#{text.inspect}>"
end
def ==(other)
self.class == other.class &&
self.path == other.path
end
def lines
@lines ||= text.split("\n")
end
end
end
end
+97
View File
@@ -0,0 +1,97 @@
module Lrama
class Lexer
class Location
attr_reader :grammar_file, :first_line, :first_column, :last_line, :last_column
def initialize(grammar_file:, first_line:, first_column:, last_line:, last_column:)
@grammar_file = grammar_file
@first_line = first_line
@first_column = first_column
@last_line = last_line
@last_column = last_column
end
def ==(other)
self.class == other.class &&
self.grammar_file == other.grammar_file &&
self.first_line == other.first_line &&
self.first_column == other.first_column &&
self.last_line == other.last_line &&
self.last_column == other.last_column
end
def partial_location(left, right)
offset = -first_column
new_first_line = -1
new_first_column = -1
new_last_line = -1
new_last_column = -1
_text.each.with_index do |line, index|
new_offset = offset + line.length + 1
if offset <= left && left <= new_offset
new_first_line = first_line + index
new_first_column = left - offset
end
if offset <= right && right <= new_offset
new_last_line = first_line + index
new_last_column = right - offset
end
offset = new_offset
end
Location.new(
grammar_file: grammar_file,
first_line: new_first_line, first_column: new_first_column,
last_line: new_last_line, last_column: new_last_column
)
end
def to_s
"#{path} (#{first_line},#{first_column})-(#{last_line},#{last_column})"
end
def generate_error_message(error_message)
<<~ERROR.chomp
#{path}:#{first_line}:#{first_column}: #{error_message}
#{line_with_carets}
ERROR
end
def line_with_carets
<<~TEXT
#{text}
#{carets}
TEXT
end
private
def path
grammar_file.path
end
def blanks
(text[0...first_column] or raise "#{first_column} is invalid").gsub(/[^\t]/, ' ')
end
def carets
blanks + '^' * (last_column - first_column)
end
def text
@text ||= _text.join("\n")
end
def _text
@_text ||=begin
range = (first_line - 1)...last_line
grammar_file.lines[range] or raise "#{range} is invalid"
end
end
end
end
end
+56
View File
@@ -0,0 +1,56 @@
require 'lrama/lexer/token/char'
require 'lrama/lexer/token/ident'
require 'lrama/lexer/token/instantiate_rule'
require 'lrama/lexer/token/tag'
require 'lrama/lexer/token/user_code'
module Lrama
class Lexer
class Token
attr_reader :s_value, :location
attr_accessor :alias_name, :referred
def initialize(s_value:, alias_name: nil, location: nil)
s_value.freeze
@s_value = s_value
@alias_name = alias_name
@location = location
end
def to_s
"value: `#{s_value}`, location: #{location}"
end
def referred_by?(string)
[self.s_value, self.alias_name].compact.include?(string)
end
def ==(other)
self.class == other.class && self.s_value == other.s_value
end
def first_line
location.first_line
end
alias :line :first_line
def first_column
location.first_column
end
alias :column :first_column
def last_line
location.last_line
end
def last_column
location.last_column
end
def invalid_ref(ref, message)
location = self.location.partial_location(ref.first_column, ref.last_column)
raise location.generate_error_message(message)
end
end
end
end
@@ -0,0 +1,8 @@
module Lrama
class Lexer
class Token
class Char < Token
end
end
end
end
@@ -0,0 +1,8 @@
module Lrama
class Lexer
class Token
class Ident < Token
end
end
end
end
@@ -0,0 +1,23 @@
module Lrama
class Lexer
class Token
class InstantiateRule < Token
attr_reader :args, :lhs_tag
def initialize(s_value:, alias_name: nil, location: nil, args: [], lhs_tag: nil)
super s_value: s_value, alias_name: alias_name, location: location
@args = args
@lhs_tag = lhs_tag
end
def rule_name
s_value
end
def args_count
args.count
end
end
end
end
end
+12
View File
@@ -0,0 +1,12 @@
module Lrama
class Lexer
class Token
class Tag < Token
# Omit "<>"
def member
s_value[1..-2] or raise "Unexpected Tag format (#{s_value})"
end
end
end
end
end
@@ -0,0 +1,77 @@
require "strscan"
module Lrama
class Lexer
class Token
class UserCode < Token
attr_accessor :tag
def references
@references ||= _references
end
private
def _references
scanner = StringScanner.new(s_value)
references = []
while !scanner.eos? do
case
when reference = scan_reference(scanner)
references << reference
when scanner.scan(/\/\*/)
scanner.scan_until(/\*\//)
else
scanner.getch
end
end
references
end
def scan_reference(scanner)
start = scanner.pos
case
# $ references
# It need to wrap an identifier with brackets to use ".-" for identifiers
when scanner.scan(/\$(<[a-zA-Z0-9_]+>)?\$/) # $$, $<long>$
tag = scanner[1] ? Lrama::Lexer::Token::Tag.new(s_value: scanner[1]) : nil
return Lrama::Grammar::Reference.new(type: :dollar, name: "$", ex_tag: tag, first_column: start, last_column: scanner.pos)
when scanner.scan(/\$(<[a-zA-Z0-9_]+>)?(\d+)/) # $1, $2, $<long>1
tag = scanner[1] ? Lrama::Lexer::Token::Tag.new(s_value: scanner[1]) : nil
return Lrama::Grammar::Reference.new(type: :dollar, number: Integer(scanner[2]), index: Integer(scanner[2]), ex_tag: tag, first_column: start, last_column: scanner.pos)
when scanner.scan(/\$(<[a-zA-Z0-9_]+>)?([a-zA-Z_][a-zA-Z0-9_]*)/) # $foo, $expr, $<long>program (named reference without brackets)
tag = scanner[1] ? Lrama::Lexer::Token::Tag.new(s_value: scanner[1]) : nil
return Lrama::Grammar::Reference.new(type: :dollar, name: scanner[2], ex_tag: tag, first_column: start, last_column: scanner.pos)
when scanner.scan(/\$(<[a-zA-Z0-9_]+>)?\[([a-zA-Z_.][-a-zA-Z0-9_.]*)\]/) # $[expr.right], $[expr-right], $<long>[expr.right] (named reference with brackets)
tag = scanner[1] ? Lrama::Lexer::Token::Tag.new(s_value: scanner[1]) : nil
return Lrama::Grammar::Reference.new(type: :dollar, name: scanner[2], ex_tag: tag, first_column: start, last_column: scanner.pos)
# @ references
# It need to wrap an identifier with brackets to use ".-" for identifiers
when scanner.scan(/@\$/) # @$
return Lrama::Grammar::Reference.new(type: :at, name: "$", first_column: start, last_column: scanner.pos)
when scanner.scan(/@(\d+)/) # @1
return Lrama::Grammar::Reference.new(type: :at, number: Integer(scanner[1]), index: Integer(scanner[1]), first_column: start, last_column: scanner.pos)
when scanner.scan(/@([a-zA-Z][a-zA-Z0-9_]*)/) # @foo, @expr (named reference without brackets)
return Lrama::Grammar::Reference.new(type: :at, name: scanner[1], first_column: start, last_column: scanner.pos)
when scanner.scan(/@\[([a-zA-Z_.][-a-zA-Z0-9_.]*)\]/) # @[expr.right], @[expr-right] (named reference with brackets)
return Lrama::Grammar::Reference.new(type: :at, name: scanner[1], first_column: start, last_column: scanner.pos)
# $: references
when scanner.scan(/\$:\$/) # $:$
return Lrama::Grammar::Reference.new(type: :index, name: "$", first_column: start, last_column: scanner.pos)
when scanner.scan(/\$:(\d+)/) # $:1
return Lrama::Grammar::Reference.new(type: :index, number: Integer(scanner[1]), first_column: start, last_column: scanner.pos)
when scanner.scan(/\$:([a-zA-Z_][a-zA-Z0-9_]*)/) # $:foo, $:expr (named reference without brackets)
return Lrama::Grammar::Reference.new(type: :index, name: scanner[1], first_column: start, last_column: scanner.pos)
when scanner.scan(/\$:\[([a-zA-Z_.][-a-zA-Z0-9_.]*)\]/) # $:[expr.right], $:[expr-right] (named reference with brackets)
return Lrama::Grammar::Reference.new(type: :index, name: scanner[1], first_column: start, last_column: scanner.pos)
end
end
end
end
end
end
+142
View File
@@ -0,0 +1,142 @@
require 'optparse'
module Lrama
# Handle option parsing for the command line interface.
class OptionParser
def initialize
@options = Options.new
@trace = []
@report = []
end
def parse(argv)
parse_by_option_parser(argv)
@options.trace_opts = validate_trace(@trace)
@options.report_opts = validate_report(@report)
@options.grammar_file = argv.shift
if !@options.grammar_file
abort "File should be specified\n"
end
if @options.grammar_file == '-'
@options.grammar_file = argv.shift or abort "File name for STDIN should be specified\n"
else
@options.y = File.open(@options.grammar_file, 'r')
end
if !@report.empty? && @options.report_file.nil? && @options.grammar_file
@options.report_file = File.dirname(@options.grammar_file) + "/" + File.basename(@options.grammar_file, ".*") + ".output"
end
if !@options.header_file && @options.header
case
when @options.outfile
@options.header_file = File.dirname(@options.outfile) + "/" + File.basename(@options.outfile, ".*") + ".h"
when @options.grammar_file
@options.header_file = File.dirname(@options.grammar_file) + "/" + File.basename(@options.grammar_file, ".*") + ".h"
end
end
@options
end
private
def parse_by_option_parser(argv)
::OptionParser.new do |o|
o.banner = <<~BANNER
Lrama is LALR (1) parser generator written by Ruby.
Usage: lrama [options] FILE
BANNER
o.separator ''
o.separator 'STDIN mode:'
o.separator 'lrama [options] - FILE read grammar from STDIN'
o.separator ''
o.separator 'Tuning the Parser:'
o.on('-S', '--skeleton=FILE', 'specify the skeleton to use') {|v| @options.skeleton = v }
o.on('-t', 'reserved, do nothing') { }
o.on('--debug', 'display debugging outputs of internal parser') {|v| @options.debug = true }
o.separator ''
o.separator 'Output:'
o.on('-H', '--header=[FILE]', 'also produce a header file named FILE') {|v| @options.header = true; @options.header_file = v }
o.on('-d', 'also produce a header file') { @options.header = true }
o.on('-r', '--report=THINGS', Array, 'also produce details on the automaton') {|v| @report = v }
o.on_tail ''
o.on_tail 'Valid Reports:'
o.on_tail " #{VALID_REPORTS.join(' ')}"
o.on('--report-file=FILE', 'also produce details on the automaton output to a file named FILE') {|v| @options.report_file = v }
o.on('-o', '--output=FILE', 'leave output to FILE') {|v| @options.outfile = v }
o.on('--trace=THINGS', Array, 'also output trace logs at runtime') {|v| @trace = v }
o.on_tail ''
o.on_tail 'Valid Traces:'
o.on_tail " #{VALID_TRACES.join(' ')}"
o.on('-v', 'reserved, do nothing') { }
o.separator ''
o.separator 'Error Recovery:'
o.on('-e', 'enable error recovery') {|v| @options.error_recovery = true }
o.separator ''
o.separator 'Other options:'
o.on('-V', '--version', "output version information and exit") {|v| puts "lrama #{Lrama::VERSION}"; exit 0 }
o.on('-h', '--help', "display this help and exit") {|v| puts o; exit 0 }
o.on_tail
o.parse!(argv)
end
end
BISON_REPORTS = %w[states itemsets lookaheads solved counterexamples cex all none]
OTHER_REPORTS = %w[verbose]
NOT_SUPPORTED_REPORTS = %w[cex none]
VALID_REPORTS = BISON_REPORTS + OTHER_REPORTS - NOT_SUPPORTED_REPORTS
def validate_report(report)
list = VALID_REPORTS
h = { grammar: true }
report.each do |r|
if list.include?(r)
h[r.to_sym] = true
else
raise "Invalid report option \"#{r}\"."
end
end
if h[:all]
(BISON_REPORTS - NOT_SUPPORTED_REPORTS).each do |r|
h[r.to_sym] = true
end
h.delete(:all)
end
return h
end
VALID_TRACES = %w[
none locations scan parse automaton bitsets
closure grammar rules actions resource
sets muscles tools m4-early m4 skeleton time
ielr cex all
]
def validate_trace(trace)
list = VALID_TRACES
h = {}
trace.each do |t|
if list.include?(t)
h[t.to_sym] = true
else
raise "Invalid trace option \"#{t}\"."
end
end
return h
end
end
end
+24
View File
@@ -0,0 +1,24 @@
module Lrama
# Command line options.
class Options
attr_accessor :skeleton, :header, :header_file,
:report_file, :outfile,
:error_recovery, :grammar_file,
:trace_opts, :report_opts, :y,
:debug
def initialize
@skeleton = "bison/yacc.c"
@header = false
@header_file = nil
@report_file = nil
@outfile = "y.tab.c"
@error_recovery = false
@grammar_file = nil
@trace_opts = nil
@report_opts = nil
@y = STDIN
@debug = false
end
end
end
+490
View File
@@ -0,0 +1,490 @@
require "erb"
require "forwardable"
require "lrama/report/duration"
module Lrama
class Output
extend Forwardable
include Report::Duration
attr_reader :grammar_file_path, :context, :grammar, :error_recovery, :include_header
def_delegators "@context", :yyfinal, :yylast, :yyntokens, :yynnts, :yynrules, :yynstates,
:yymaxutok, :yypact_ninf, :yytable_ninf
def_delegators "@grammar", :eof_symbol, :error_symbol, :undef_symbol, :accept_symbol
def initialize(
out:, output_file_path:, template_name:, grammar_file_path:,
context:, grammar:, header_out: nil, header_file_path: nil, error_recovery: false
)
@out = out
@output_file_path = output_file_path
@template_name = template_name
@grammar_file_path = grammar_file_path
@header_out = header_out
@header_file_path = header_file_path
@context = context
@grammar = grammar
@error_recovery = error_recovery
@include_header = header_file_path ? header_file_path.sub("./", "") : nil
end
if ERB.instance_method(:initialize).parameters.last.first == :key
def self.erb(input)
ERB.new(input, trim_mode: '-')
end
else
def self.erb(input)
ERB.new(input, nil, '-')
end
end
def render_partial(file)
render_template(partial_file(file))
end
def render
report_duration(:render) do
tmp = eval_template(template_file, @output_file_path)
@out << tmp
if @header_file_path
tmp = eval_template(header_template_file, @header_file_path)
if @header_out
@header_out << tmp
else
File.write(@header_file_path, tmp)
end
end
end
end
# A part of b4_token_enums
def token_enums
str = ""
@context.yytokentype.each do |s_value, token_id, display_name|
s = sprintf("%s = %d%s", s_value, token_id, token_id == yymaxutok ? "" : ",")
if display_name
str << sprintf(" %-30s /* %s */\n", s, display_name)
else
str << sprintf(" %s\n", s)
end
end
str
end
# b4_symbol_enum
def symbol_enum
str = ""
last_sym_number = @context.yysymbol_kind_t.last[1]
@context.yysymbol_kind_t.each do |s_value, sym_number, display_name|
s = sprintf("%s = %d%s", s_value, sym_number, (sym_number == last_sym_number) ? "" : ",")
if display_name
str << sprintf(" %-40s /* %s */\n", s, display_name)
else
str << sprintf(" %s\n", s)
end
end
str
end
def yytranslate
int_array_to_string(@context.yytranslate)
end
def yytranslate_inverted
int_array_to_string(@context.yytranslate_inverted)
end
def yyrline
int_array_to_string(@context.yyrline)
end
def yytname
string_array_to_string(@context.yytname) + " YY_NULLPTR"
end
# b4_int_type_for
def int_type_for(ary)
min = ary.min
max = ary.max
case
when (-127 <= min && min <= 127) && (-127 <= max && max <= 127)
"yytype_int8"
when (0 <= min && min <= 255) && (0 <= max && max <= 255)
"yytype_uint8"
when (-32767 <= min && min <= 32767) && (-32767 <= max && max <= 32767)
"yytype_int16"
when (0 <= min && min <= 65535) && (0 <= max && max <= 65535)
"yytype_uint16"
else
"int"
end
end
def symbol_actions_for_printer
str = ""
@grammar.symbols.each do |sym|
next unless sym.printer
str << <<-STR
case #{sym.enum_name}: /* #{sym.comment} */
#line #{sym.printer.lineno} "#{@grammar_file_path}"
{#{sym.printer.translated_code(sym.tag)}}
#line [@oline@] [@ofile@]
break;
STR
end
str
end
def symbol_actions_for_destructor
str = ""
@grammar.symbols.each do |sym|
next unless sym.destructor
str << <<-STR
case #{sym.enum_name}: /* #{sym.comment} */
#line #{sym.destructor.lineno} "#{@grammar_file_path}"
{#{sym.destructor.translated_code(sym.tag)}}
#line [@oline@] [@ofile@]
break;
STR
end
str
end
# b4_user_initial_action
def user_initial_action(comment = "")
return "" unless @grammar.initial_action
<<-STR
#{comment}
#line #{@grammar.initial_action.line} "#{@grammar_file_path}"
{#{@grammar.initial_action.translated_code}}
STR
end
def after_shift_function(comment = "")
return "" unless @grammar.after_shift
<<-STR
#{comment}
#line #{@grammar.after_shift.line} "#{@grammar_file_path}"
{#{@grammar.after_shift.s_value}(#{parse_param_name});}
#line [@oline@] [@ofile@]
STR
end
def before_reduce_function(comment = "")
return "" unless @grammar.before_reduce
<<-STR
#{comment}
#line #{@grammar.before_reduce.line} "#{@grammar_file_path}"
{#{@grammar.before_reduce.s_value}(yylen#{user_args});}
#line [@oline@] [@ofile@]
STR
end
def after_reduce_function(comment = "")
return "" unless @grammar.after_reduce
<<-STR
#{comment}
#line #{@grammar.after_reduce.line} "#{@grammar_file_path}"
{#{@grammar.after_reduce.s_value}(yylen#{user_args});}
#line [@oline@] [@ofile@]
STR
end
def after_shift_error_token_function(comment = "")
return "" unless @grammar.after_shift_error_token
<<-STR
#{comment}
#line #{@grammar.after_shift_error_token.line} "#{@grammar_file_path}"
{#{@grammar.after_shift_error_token.s_value}(#{parse_param_name});}
#line [@oline@] [@ofile@]
STR
end
def after_pop_stack_function(len, comment = "")
return "" unless @grammar.after_pop_stack
<<-STR
#{comment}
#line #{@grammar.after_pop_stack.line} "#{@grammar_file_path}"
{#{@grammar.after_pop_stack.s_value}(#{len}#{user_args});}
#line [@oline@] [@ofile@]
STR
end
def symbol_actions_for_error_token
str = ""
@grammar.symbols.each do |sym|
next unless sym.error_token
str << <<-STR
case #{sym.enum_name}: /* #{sym.comment} */
#line #{sym.error_token.lineno} "#{@grammar_file_path}"
{#{sym.error_token.translated_code(sym.tag)}}
#line [@oline@] [@ofile@]
break;
STR
end
str
end
# b4_user_actions
def user_actions
str = ""
@context.states.rules.each do |rule|
next unless rule.token_code
code = rule.token_code
spaces = " " * (code.column - 1)
str << <<-STR
case #{rule.id + 1}: /* #{rule.as_comment} */
#line #{code.line} "#{@grammar_file_path}"
#{spaces}{#{rule.translated_code}}
#line [@oline@] [@ofile@]
break;
STR
end
str << <<-STR
#line [@oline@] [@ofile@]
STR
str
end
def omit_blanks(param)
param.strip
end
# b4_parse_param
def parse_param
if @grammar.parse_param
omit_blanks(@grammar.parse_param)
else
""
end
end
def lex_param
if @grammar.lex_param
omit_blanks(@grammar.lex_param)
else
""
end
end
# b4_user_formals
def user_formals
if @grammar.parse_param
", #{parse_param}"
else
""
end
end
# b4_user_args
def user_args
if @grammar.parse_param
", #{parse_param_name}"
else
""
end
end
def extract_param_name(param)
param[/\b([a-zA-Z0-9_]+)(?=\s*\z)/]
end
def parse_param_name
if @grammar.parse_param
extract_param_name(parse_param)
else
""
end
end
def lex_param_name
if @grammar.lex_param
extract_param_name(lex_param)
else
""
end
end
# b4_parse_param_use
def parse_param_use(val, loc)
str = <<-STR
YY_USE (#{val});
YY_USE (#{loc});
STR
if @grammar.parse_param
str << " YY_USE (#{parse_param_name});"
end
str
end
# b4_yylex_formals
def yylex_formals
ary = ["&yylval", "&yylloc"]
if @grammar.lex_param
ary << lex_param_name
end
"(#{ary.join(', ')})"
end
# b4_table_value_equals
def table_value_equals(table, value, literal, symbol)
if literal < table.min || table.max < literal
"0"
else
"((#{value}) == #{symbol})"
end
end
# b4_yyerror_args
def yyerror_args
ary = ["&yylloc"]
if @grammar.parse_param
ary << parse_param_name
end
"#{ary.join(', ')}"
end
def template_basename
File.basename(template_file)
end
def aux
@grammar.aux
end
def int_array_to_string(ary)
last = ary.count - 1
s = ary.each_with_index.each_slice(10).map do |slice|
str = " "
slice.each do |e, i|
str << sprintf("%6d%s", e, (i == last) ? "" : ",")
end
str
end
s.join("\n")
end
def spec_mapped_header_file
@header_file_path
end
def b4_cpp_guard__b4_spec_mapped_header_file
if @header_file_path
"YY_YY_" + @header_file_path.gsub(/[^a-zA-Z_0-9]+/, "_").upcase + "_INCLUDED"
else
""
end
end
# b4_percent_code_get
def percent_code(name)
@grammar.percent_codes.select do |percent_code|
percent_code.name == name
end.map do |percent_code|
percent_code.code
end.join
end
private
def eval_template(file, path)
tmp = render_template(file)
replace_special_variables(tmp, path)
end
def render_template(file)
erb = self.class.erb(File.read(file))
erb.filename = file
erb.result_with_hash(context: @context, output: self)
end
def template_file
File.join(template_dir, @template_name)
end
def header_template_file
File.join(template_dir, "bison/yacc.h")
end
def partial_file(file)
File.join(template_dir, file)
end
def template_dir
File.expand_path("../../../template", __FILE__)
end
def string_array_to_string(ary)
str = ""
tmp = " "
ary.each do |s|
s = s.gsub('\\', '\\\\\\\\')
s = s.gsub('"', '\\"')
if (tmp + s + " \"\",").length > 75
str << tmp << "\n"
tmp = " \"#{s}\","
else
tmp << " \"#{s}\","
end
end
str << tmp
end
def replace_special_variables(str, ofile)
str.each_line.with_index(1).map do |line, i|
line.gsub!("[@oline@]", (i + 1).to_s)
line.gsub!("[@ofile@]", "\"#{ofile}\"")
line
end.join
end
end
end
File diff suppressed because it is too large Load Diff
+2
View File
@@ -0,0 +1,2 @@
require 'lrama/report/duration'
require 'lrama/report/profile'
+25
View File
@@ -0,0 +1,25 @@
module Lrama
class Report
module Duration
def self.enable
@_report_duration_enabled = true
end
def self.enabled?
!!@_report_duration_enabled
end
def report_duration(method_name)
time1 = Time.now.to_f
result = yield
time2 = Time.now.to_f
if Duration.enabled?
puts sprintf("%s %10.5f s", method_name, time2 - time1)
end
return result
end
end
end
end
+14
View File
@@ -0,0 +1,14 @@
module Lrama
class Report
module Profile
# See "Profiling Lrama" in README.md for how to use.
def self.report_profile
require "stackprof"
StackProf.run(mode: :cpu, raw: true, out: 'tmp/stackprof-cpu-myapp.dump') do
yield
end
end
end
end
end
+144
View File
@@ -0,0 +1,144 @@
require "lrama/state/reduce"
require "lrama/state/reduce_reduce_conflict"
require "lrama/state/resolved_conflict"
require "lrama/state/shift"
require "lrama/state/shift_reduce_conflict"
module Lrama
class State
attr_reader :id, :accessing_symbol, :kernels, :conflicts, :resolved_conflicts,
:default_reduction_rule, :closure, :items
attr_accessor :shifts, :reduces
def initialize(id, accessing_symbol, kernels)
@id = id
@accessing_symbol = accessing_symbol
@kernels = kernels.freeze
@items = @kernels
# Manage relationships between items to state
# to resolve next state
@items_to_state = {}
@conflicts = []
@resolved_conflicts = []
@default_reduction_rule = nil
end
def closure=(closure)
@closure = closure
@items = @kernels + @closure
end
def non_default_reduces
reduces.reject do |reduce|
reduce.rule == @default_reduction_rule
end
end
def compute_shifts_reduces
_shifts = {}
reduces = []
items.each do |item|
# TODO: Consider what should be pushed
if item.end_of_rule?
reduces << Reduce.new(item)
else
key = item.next_sym
_shifts[key] ||= []
_shifts[key] << item.new_by_next_position
end
end
# It seems Bison 3.8.2 iterates transitions order by symbol number
shifts = _shifts.sort_by do |next_sym, new_items|
next_sym.number
end.map do |next_sym, new_items|
Shift.new(next_sym, new_items.flatten)
end
self.shifts = shifts.freeze
self.reduces = reduces.freeze
end
def set_items_to_state(items, next_state)
@items_to_state[items] = next_state
end
def set_look_ahead(rule, look_ahead)
reduce = reduces.find do |r|
r.rule == rule
end
reduce.look_ahead = look_ahead
end
def nterm_transitions
@nterm_transitions ||= transitions.select {|shift, _| shift.next_sym.nterm? }
end
def term_transitions
@term_transitions ||= transitions.select {|shift, _| shift.next_sym.term? }
end
def transitions
@transitions ||= shifts.map {|shift| [shift, @items_to_state[shift.next_items]] }
end
def selected_term_transitions
term_transitions.reject do |shift, next_state|
shift.not_selected
end
end
# Move to next state by sym
def transition(sym)
result = nil
if sym.term?
term_transitions.each do |shift, next_state|
term = shift.next_sym
result = next_state if term == sym
end
else
nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
result = next_state if nterm == sym
end
end
raise "Can not transit by #{sym} #{self}" if result.nil?
result
end
def find_reduce_by_item!(item)
reduces.find do |r|
r.item == item
end || (raise "reduce is not found. #{item}")
end
def default_reduction_rule=(default_reduction_rule)
@default_reduction_rule = default_reduction_rule
reduces.each do |r|
if r.rule == default_reduction_rule
r.default_reduction = true
end
end
end
def has_conflicts?
!@conflicts.empty?
end
def sr_conflicts
@conflicts.select do |conflict|
conflict.type == :shift_reduce
end
end
def rr_conflicts
@conflicts.select do |conflict|
conflict.type == :reduce_reduce
end
end
end
end
+35
View File
@@ -0,0 +1,35 @@
module Lrama
class State
class Reduce
# https://www.gnu.org/software/bison/manual/html_node/Default-Reductions.html
attr_reader :item, :look_ahead, :not_selected_symbols
attr_accessor :default_reduction
def initialize(item)
@item = item
@look_ahead = nil
@not_selected_symbols = []
end
def rule
@item.rule
end
def look_ahead=(look_ahead)
@look_ahead = look_ahead.freeze
end
def add_not_selected_symbol(sym)
@not_selected_symbols << sym
end
def selected_look_ahead
if @look_ahead
@look_ahead - @not_selected_symbols
else
[]
end
end
end
end
end
@@ -0,0 +1,9 @@
module Lrama
class State
class ReduceReduceConflict < Struct.new(:symbols, :reduce1, :reduce2, keyword_init: true)
def type
:reduce_reduce
end
end
end
end
@@ -0,0 +1,29 @@
module Lrama
class State
# * symbol: A symbol under discussion
# * reduce: A reduce under discussion
# * which: For which a conflict is resolved. :shift, :reduce or :error (for nonassociative)
class ResolvedConflict < Struct.new(:symbol, :reduce, :which, :same_prec, keyword_init: true)
def report_message
s = symbol.display_name
r = reduce.rule.precedence_sym.display_name
case
when which == :shift && same_prec
msg = "resolved as #{which} (%right #{s})"
when which == :shift
msg = "resolved as #{which} (#{r} < #{s})"
when which == :reduce && same_prec
msg = "resolved as #{which} (%left #{s})"
when which == :reduce
msg = "resolved as #{which} (#{s} < #{r})"
when which == :error
msg = "resolved as an #{which} (%nonassoc #{s})"
else
raise "Unknown direction. #{self}"
end
"Conflict between rule #{reduce.rule.id} and token #{s} #{msg}."
end
end
end
end
+13
View File
@@ -0,0 +1,13 @@
module Lrama
class State
class Shift
attr_reader :next_sym, :next_items
attr_accessor :not_selected
def initialize(next_sym, next_items)
@next_sym = next_sym
@next_items = next_items
end
end
end
end
@@ -0,0 +1,9 @@
module Lrama
class State
class ShiftReduceConflict < Struct.new(:symbols, :shift, :reduce, keyword_init: true)
def type
:shift_reduce
end
end
end
end
+556
View File
@@ -0,0 +1,556 @@
require "forwardable"
require "lrama/report/duration"
require "lrama/states/item"
module Lrama
# States is passed to a template file
#
# "Efficient Computation of LALR(1) Look-Ahead Sets"
# https://dl.acm.org/doi/pdf/10.1145/69622.357187
class States
extend Forwardable
include Lrama::Report::Duration
def_delegators "@grammar", :symbols, :terms, :nterms, :rules,
:accept_symbol, :eof_symbol, :undef_symbol, :find_symbol_by_s_value!
attr_reader :states, :reads_relation, :includes_relation, :lookback_relation
def initialize(grammar, warning, trace_state: false)
@grammar = grammar
@warning = warning
@trace_state = trace_state
@states = []
# `DR(p, A) = {t ∈ T | p -(A)-> r -(t)-> }`
# where p is state, A is nterm, t is term.
#
# `@direct_read_sets` is a hash whose
# key is [state.id, nterm.token_id],
# value is bitmap of term.
@direct_read_sets = {}
# Reads relation on nonterminal transitions (pair of state and nterm)
# `(p, A) reads (r, C) iff p -(A)-> r -(C)-> and C =>* ε`
# where p, r are state, A, C are nterm.
#
# `@reads_relation` is a hash whose
# key is [state.id, nterm.token_id],
# value is array of [state.id, nterm.token_id].
@reads_relation = {}
# `@read_sets` is a hash whose
# key is [state.id, nterm.token_id],
# value is bitmap of term.
@read_sets = {}
# `(p, A) includes (p', B) iff B -> βAγ, γ =>* ε, p' -(β)-> p`
# where p, p' are state, A, B are nterm, β, γ is sequence of symbol.
#
# `@includes_relation` is a hash whose
# key is [state.id, nterm.token_id],
# value is array of [state.id, nterm.token_id].
@includes_relation = {}
# `(q, A -> ω) lookback (p, A) iff p -(ω)-> q`
# where p, q are state, A -> ω is rule, A is nterm, ω is sequence of symbol.
#
# `@lookback_relation` is a hash whose
# key is [state.id, rule.id],
# value is array of [state.id, nterm.token_id].
@lookback_relation = {}
# `@follow_sets` is a hash whose
# key is [state.id, rule.id],
# value is bitmap of term.
@follow_sets = {}
# `LA(q, A -> ω) = {Follow(p, A) | (q, A -> ω) lookback (p, A)`
#
# `@la` is a hash whose
# key is [state.id, rule.id],
# value is bitmap of term.
@la = {}
end
def compute
# Look Ahead Sets
report_duration(:compute_lr0_states) { compute_lr0_states }
report_duration(:compute_direct_read_sets) { compute_direct_read_sets }
report_duration(:compute_reads_relation) { compute_reads_relation }
report_duration(:compute_read_sets) { compute_read_sets }
report_duration(:compute_includes_relation) { compute_includes_relation }
report_duration(:compute_lookback_relation) { compute_lookback_relation }
report_duration(:compute_follow_sets) { compute_follow_sets }
report_duration(:compute_look_ahead_sets) { compute_look_ahead_sets }
# Conflicts
report_duration(:compute_conflicts) { compute_conflicts }
report_duration(:compute_default_reduction) { compute_default_reduction }
check_conflicts
end
def reporter
StatesReporter.new(self)
end
def states_count
@states.count
end
def direct_read_sets
@direct_read_sets.transform_values do |v|
bitmap_to_terms(v)
end
end
def read_sets
@read_sets.transform_values do |v|
bitmap_to_terms(v)
end
end
def follow_sets
@follow_sets.transform_values do |v|
bitmap_to_terms(v)
end
end
def la
@la.transform_values do |v|
bitmap_to_terms(v)
end
end
private
def sr_conflicts
@states.flat_map(&:sr_conflicts)
end
def rr_conflicts
@states.flat_map(&:rr_conflicts)
end
def trace_state
if @trace_state
yield STDERR
end
end
def create_state(accessing_symbol, kernels, states_created)
# A item can appear in some states,
# so need to use `kernels` (not `kernels.first`) as a key.
#
# For example...
#
# %%
# program: '+' strings_1
# | '-' strings_2
# ;
#
# strings_1: string_1
# ;
#
# strings_2: string_1
# | string_2
# ;
#
# string_1: string
# ;
#
# string_2: string '+'
# ;
#
# string: tSTRING
# ;
# %%
#
# For these grammar, there are 2 states
#
# State A
# string_1: string •
#
# State B
# string_1: string •
# string_2: string • '+'
#
return [states_created[kernels], false] if states_created[kernels]
state = State.new(@states.count, accessing_symbol, kernels)
@states << state
states_created[kernels] = state
return [state, true]
end
def setup_state(state)
# closure
closure = []
visited = {}
queued = {}
items = state.kernels.dup
items.each do |item|
queued[item] = true
end
while (item = items.shift) do
visited[item] = true
if (sym = item.next_sym) && sym.nterm?
@grammar.find_rules_by_symbol!(sym).each do |rule|
i = Item.new(rule: rule, position: 0)
next if queued[i]
closure << i
items << i
queued[i] = true
end
end
end
state.closure = closure.sort_by {|i| i.rule.id }
# Trace
trace_state do |out|
out << "Closure: input\n"
state.kernels.each do |item|
out << " #{item.display_rest}\n"
end
out << "\n\n"
out << "Closure: output\n"
state.items.each do |item|
out << " #{item.display_rest}\n"
end
out << "\n\n"
end
# shift & reduce
state.compute_shifts_reduces
end
def enqueue_state(states, state)
# Trace
previous = state.kernels.first.previous_sym
trace_state do |out|
out << sprintf("state_list_append (state = %d, symbol = %d (%s))",
@states.count, previous.number, previous.display_name)
end
states << state
end
def compute_lr0_states
# State queue
states = []
states_created = {}
state, _ = create_state(symbols.first, [Item.new(rule: @grammar.rules.first, position: 0)], states_created)
enqueue_state(states, state)
while (state = states.shift) do
# Trace
#
# Bison 3.8.2 renders "(reached by "end-of-input")" for State 0 but
# I think it is not correct...
previous = state.kernels.first.previous_sym
trace_state do |out|
out << "Processing state #{state.id} (reached by #{previous.display_name})\n"
end
setup_state(state)
state.shifts.each do |shift|
new_state, created = create_state(shift.next_sym, shift.next_items, states_created)
state.set_items_to_state(shift.next_items, new_state)
enqueue_state(states, new_state) if created
end
end
end
def nterm_transitions
a = []
@states.each do |state|
state.nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
a << [state, nterm, next_state]
end
end
a
end
def compute_direct_read_sets
@states.each do |state|
state.nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
ary = next_state.term_transitions.map do |shift, _|
shift.next_sym.number
end
key = [state.id, nterm.token_id]
@direct_read_sets[key] = Bitmap.from_array(ary)
end
end
end
def compute_reads_relation
@states.each do |state|
state.nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
next_state.nterm_transitions.each do |shift2, _next_state2|
nterm2 = shift2.next_sym
if nterm2.nullable
key = [state.id, nterm.token_id]
@reads_relation[key] ||= []
@reads_relation[key] << [next_state.id, nterm2.token_id]
end
end
end
end
end
def compute_read_sets
sets = nterm_transitions.map do |state, nterm, next_state|
[state.id, nterm.token_id]
end
@read_sets = Digraph.new(sets, @reads_relation, @direct_read_sets).compute
end
# Execute transition of state by symbols
# then return final state.
def transition(state, symbols)
symbols.each do |sym|
state = state.transition(sym)
end
state
end
def compute_includes_relation
@states.each do |state|
state.nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
@grammar.find_rules_by_symbol!(nterm).each do |rule|
i = rule.rhs.count - 1
while (i > -1) do
sym = rule.rhs[i]
break if sym.term?
state2 = transition(state, rule.rhs[0...i])
# p' = state, B = nterm, p = state2, A = sym
key = [state2.id, sym.token_id]
# TODO: need to omit if state == state2 ?
@includes_relation[key] ||= []
@includes_relation[key] << [state.id, nterm.token_id]
break if !sym.nullable
i -= 1
end
end
end
end
end
def compute_lookback_relation
@states.each do |state|
state.nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
@grammar.find_rules_by_symbol!(nterm).each do |rule|
state2 = transition(state, rule.rhs)
# p = state, A = nterm, q = state2, A -> ω = rule
key = [state2.id, rule.id]
@lookback_relation[key] ||= []
@lookback_relation[key] << [state.id, nterm.token_id]
end
end
end
end
def compute_follow_sets
sets = nterm_transitions.map do |state, nterm, next_state|
[state.id, nterm.token_id]
end
@follow_sets = Digraph.new(sets, @includes_relation, @read_sets).compute
end
def compute_look_ahead_sets
@states.each do |state|
rules.each do |rule|
ary = @lookback_relation[[state.id, rule.id]]
next if !ary
ary.each do |state2_id, nterm_token_id|
# q = state, A -> ω = rule, p = state2, A = nterm
follows = @follow_sets[[state2_id, nterm_token_id]]
next if follows == 0
key = [state.id, rule.id]
@la[key] ||= 0
look_ahead = @la[key] | follows
@la[key] |= look_ahead
# No risk of conflict when
# * the state only has single reduce
# * the state only has nterm_transitions (GOTO)
next if state.reduces.count == 1 && state.term_transitions.count == 0
state.set_look_ahead(rule, bitmap_to_terms(look_ahead))
end
end
end
end
def bitmap_to_terms(bit)
ary = Bitmap.to_array(bit)
ary.map do |i|
@grammar.find_symbol_by_number!(i)
end
end
def compute_conflicts
compute_shift_reduce_conflicts
compute_reduce_reduce_conflicts
end
def compute_shift_reduce_conflicts
states.each do |state|
state.shifts.each do |shift|
state.reduces.each do |reduce|
sym = shift.next_sym
next unless reduce.look_ahead
next if !reduce.look_ahead.include?(sym)
# Shift/Reduce conflict
shift_prec = sym.precedence
reduce_prec = reduce.item.rule.precedence
# Can resolve only when both have prec
unless shift_prec && reduce_prec
state.conflicts << State::ShiftReduceConflict.new(symbols: [sym], shift: shift, reduce: reduce)
next
end
case
when shift_prec < reduce_prec
# Reduce is selected
state.resolved_conflicts << State::ResolvedConflict.new(symbol: sym, reduce: reduce, which: :reduce)
shift.not_selected = true
next
when shift_prec > reduce_prec
# Shift is selected
state.resolved_conflicts << State::ResolvedConflict.new(symbol: sym, reduce: reduce, which: :shift)
reduce.add_not_selected_symbol(sym)
next
end
# shift_prec == reduce_prec, then check associativity
case sym.precedence.type
when :precedence
# %precedence only specifies precedence and not specify associativity
# then a conflict is unresolved if precedence is same.
state.conflicts << State::ShiftReduceConflict.new(symbols: [sym], shift: shift, reduce: reduce)
next
when :right
# Shift is selected
state.resolved_conflicts << State::ResolvedConflict.new(symbol: sym, reduce: reduce, which: :shift, same_prec: true)
reduce.add_not_selected_symbol(sym)
next
when :left
# Reduce is selected
state.resolved_conflicts << State::ResolvedConflict.new(symbol: sym, reduce: reduce, which: :reduce, same_prec: true)
shift.not_selected = true
next
when :nonassoc
# Can not resolve
#
# nonassoc creates "run-time" error, precedence creates "compile-time" error.
# Then omit both the shift and reduce.
#
# https://www.gnu.org/software/bison/manual/html_node/Using-Precedence.html
state.resolved_conflicts << State::ResolvedConflict.new(symbol: sym, reduce: reduce, which: :error)
shift.not_selected = true
reduce.add_not_selected_symbol(sym)
else
raise "Unknown precedence type. #{sym}"
end
end
end
end
end
def compute_reduce_reduce_conflicts
states.each do |state|
count = state.reduces.count
for i in 0...count do
reduce1 = state.reduces[i]
next if reduce1.look_ahead.nil?
for j in (i+1)...count do
reduce2 = state.reduces[j]
next if reduce2.look_ahead.nil?
intersection = reduce1.look_ahead & reduce2.look_ahead
if !intersection.empty?
state.conflicts << State::ReduceReduceConflict.new(symbols: intersection, reduce1: reduce1, reduce2: reduce2)
end
end
end
end
end
def compute_default_reduction
states.each do |state|
next if state.reduces.empty?
# Do not set, if conflict exist
next if !state.conflicts.empty?
# Do not set, if shift with `error` exists.
next if state.shifts.map(&:next_sym).include?(@grammar.error_symbol)
state.default_reduction_rule = state.reduces.map do |r|
[r.rule, r.rule.id, (r.look_ahead || []).count]
end.min_by do |rule, rule_id, count|
[-count, rule_id]
end.first
end
end
def check_conflicts
sr_count = sr_conflicts.count
rr_count = rr_conflicts.count
if @grammar.expect
expected_sr_conflicts = @grammar.expect
expected_rr_conflicts = 0
if expected_sr_conflicts != sr_count
@warning.error("shift/reduce conflicts: #{sr_count} found, #{expected_sr_conflicts} expected")
end
if expected_rr_conflicts != rr_count
@warning.error("reduce/reduce conflicts: #{rr_count} found, #{expected_rr_conflicts} expected")
end
else
if sr_count != 0
@warning.warn("shift/reduce conflicts: #{sr_count} found")
end
if rr_count != 0
@warning.warn("reduce/reduce conflicts: #{rr_count} found")
end
end
end
end
end
+81
View File
@@ -0,0 +1,81 @@
# TODO: Validate position is not over rule rhs
require "forwardable"
module Lrama
class States
class Item < Struct.new(:rule, :position, keyword_init: true)
extend Forwardable
def_delegators "rule", :lhs, :rhs
# Optimization for States#setup_state
def hash
[rule_id, position].hash
end
def rule_id
rule.id
end
def empty_rule?
rule.empty_rule?
end
def number_of_rest_symbols
rhs.count - position
end
def next_sym
rhs[position]
end
def next_next_sym
rhs[position + 1]
end
def previous_sym
rhs[position - 1]
end
def end_of_rule?
rhs.count == position
end
def beginning_of_rule?
position == 0
end
def start_item?
rule.initial_rule? && beginning_of_rule?
end
def new_by_next_position
Item.new(rule: rule, position: position + 1)
end
def symbols_before_dot
rhs[0...position]
end
def symbols_after_dot
rhs[position..-1]
end
def to_s
"#{lhs.id.s_value}: #{display_name}"
end
def display_name
r = rhs.map(&:display_name).insert(position, "").join(" ")
"#{r} (rule #{rule_id})"
end
# Right after position
def display_rest
r = rhs[position..-1].map(&:display_name).join(" ")
". #{r} (rule #{rule_id})"
end
end
end
end
+321
View File
@@ -0,0 +1,321 @@
module Lrama
class StatesReporter
include Lrama::Report::Duration
def initialize(states)
@states = states
end
def report(io, **options)
report_duration(:report) do
_report(io, **options)
end
end
private
def _report(io, grammar: false, states: false, itemsets: false, lookaheads: false, solved: false, counterexamples: false, verbose: false)
# TODO: Unused terms
# TODO: Unused rules
report_conflicts(io)
report_grammar(io) if grammar
report_states(io, itemsets, lookaheads, solved, counterexamples, verbose)
end
def report_conflicts(io)
has_conflict = false
@states.states.each do |state|
messages = []
cs = state.conflicts.group_by(&:type)
if cs[:shift_reduce]
messages << "#{cs[:shift_reduce].count} shift/reduce"
end
if cs[:reduce_reduce]
messages << "#{cs[:reduce_reduce].count} reduce/reduce"
end
if !messages.empty?
has_conflict = true
io << "State #{state.id} conflicts: #{messages.join(', ')}\n"
end
end
if has_conflict
io << "\n\n"
end
end
def report_grammar(io)
io << "Grammar\n"
last_lhs = nil
@states.rules.each do |rule|
if rule.empty_rule?
r = "ε"
else
r = rule.rhs.map(&:display_name).join(" ")
end
if rule.lhs == last_lhs
io << sprintf("%5d %s| %s\n", rule.id, " " * rule.lhs.display_name.length, r)
else
io << "\n"
io << sprintf("%5d %s: %s\n", rule.id, rule.lhs.display_name, r)
end
last_lhs = rule.lhs
end
io << "\n\n"
end
def report_states(io, itemsets, lookaheads, solved, counterexamples, verbose)
if counterexamples
cex = Counterexamples.new(@states)
end
@states.states.each do |state|
# Report State
io << "State #{state.id}\n\n"
# Report item
last_lhs = nil
list = itemsets ? state.items : state.kernels
list.sort_by {|i| [i.rule_id, i.position] }.each do |item|
if item.empty_rule?
r = "ε •"
else
r = item.rhs.map(&:display_name).insert(item.position, "").join(" ")
end
if item.lhs == last_lhs
l = " " * item.lhs.id.s_value.length + "|"
else
l = item.lhs.id.s_value + ":"
end
la = ""
if lookaheads && item.end_of_rule?
reduce = state.find_reduce_by_item!(item)
look_ahead = reduce.selected_look_ahead
if !look_ahead.empty?
la = " [#{look_ahead.map(&:display_name).join(", ")}]"
end
end
last_lhs = item.lhs
io << sprintf("%5i %s %s%s\n", item.rule_id, l, r, la)
end
io << "\n"
# Report shifts
tmp = state.term_transitions.reject do |shift, _|
shift.not_selected
end.map do |shift, next_state|
[shift.next_sym, next_state.id]
end
max_len = tmp.map(&:first).map(&:display_name).map(&:length).max
tmp.each do |term, state_id|
io << " #{term.display_name.ljust(max_len)} shift, and go to state #{state_id}\n"
end
io << "\n" if !tmp.empty?
# Report error caused by %nonassoc
nl = false
tmp = state.resolved_conflicts.select do |resolved|
resolved.which == :error
end.map do |error|
error.symbol.display_name
end
max_len = tmp.map(&:length).max
tmp.each do |name|
nl = true
io << " #{name.ljust(max_len)} error (nonassociative)\n"
end
io << "\n" if !tmp.empty?
# Report reduces
nl = false
max_len = state.non_default_reduces.flat_map(&:look_ahead).compact.map(&:display_name).map(&:length).max || 0
max_len = [max_len, "$default".length].max if state.default_reduction_rule
ary = []
state.non_default_reduces.each do |reduce|
reduce.look_ahead.each do |term|
ary << [term, reduce]
end
end
ary.sort_by do |term, reduce|
term.number
end.each do |term, reduce|
rule = reduce.item.rule
io << " #{term.display_name.ljust(max_len)} reduce using rule #{rule.id} (#{rule.lhs.display_name})\n"
nl = true
end
if (r = state.default_reduction_rule)
nl = true
s = "$default".ljust(max_len)
if r.initial_rule?
io << " #{s} accept\n"
else
io << " #{s} reduce using rule #{r.id} (#{r.lhs.display_name})\n"
end
end
io << "\n" if nl
# Report nonterminal transitions
tmp = []
max_len = 0
state.nterm_transitions.each do |shift, next_state|
nterm = shift.next_sym
tmp << [nterm, next_state.id]
max_len = [max_len, nterm.id.s_value.length].max
end
tmp.uniq!
tmp.sort_by! do |nterm, state_id|
nterm.number
end
tmp.each do |nterm, state_id|
io << " #{nterm.id.s_value.ljust(max_len)} go to state #{state_id}\n"
end
io << "\n" if !tmp.empty?
if solved
# Report conflict resolutions
state.resolved_conflicts.each do |resolved|
io << " #{resolved.report_message}\n"
end
io << "\n" if !state.resolved_conflicts.empty?
end
if counterexamples && state.has_conflicts?
# Report counterexamples
examples = cex.compute(state)
examples.each do |example|
label0 = example.type == :shift_reduce ? "shift/reduce" : "reduce/reduce"
label1 = example.type == :shift_reduce ? "Shift derivation" : "First Reduce derivation"
label2 = example.type == :shift_reduce ? "Reduce derivation" : "Second Reduce derivation"
io << " #{label0} conflict on token #{example.conflict_symbol.id.s_value}:\n"
io << " #{example.path1_item}\n"
io << " #{example.path2_item}\n"
io << " #{label1}\n"
example.derivations1.render_strings_for_report.each do |str|
io << " #{str}\n"
end
io << " #{label2}\n"
example.derivations2.render_strings_for_report.each do |str|
io << " #{str}\n"
end
end
end
if verbose
# Report direct_read_sets
io << " [Direct Read sets]\n"
direct_read_sets = @states.direct_read_sets
@states.nterms.each do |nterm|
terms = direct_read_sets[[state.id, nterm.token_id]]
next if !terms
next if terms.empty?
str = terms.map {|sym| sym.id.s_value }.join(", ")
io << " read #{nterm.id.s_value} shift #{str}\n"
end
io << "\n"
# Report reads_relation
io << " [Reads Relation]\n"
@states.nterms.each do |nterm|
a = @states.reads_relation[[state.id, nterm.token_id]]
next if !a
a.each do |state_id2, nterm_id2|
n = @states.nterms.find {|n| n.token_id == nterm_id2 }
io << " (State #{state_id2}, #{n.id.s_value})\n"
end
end
io << "\n"
# Report read_sets
io << " [Read sets]\n"
read_sets = @states.read_sets
@states.nterms.each do |nterm|
terms = read_sets[[state.id, nterm.token_id]]
next if !terms
next if terms.empty?
terms.each do |sym|
io << " #{sym.id.s_value}\n"
end
end
io << "\n"
# Report includes_relation
io << " [Includes Relation]\n"
@states.nterms.each do |nterm|
a = @states.includes_relation[[state.id, nterm.token_id]]
next if !a
a.each do |state_id2, nterm_id2|
n = @states.nterms.find {|n| n.token_id == nterm_id2 }
io << " (State #{state.id}, #{nterm.id.s_value}) -> (State #{state_id2}, #{n.id.s_value})\n"
end
end
io << "\n"
# Report lookback_relation
io << " [Lookback Relation]\n"
@states.rules.each do |rule|
a = @states.lookback_relation[[state.id, rule.id]]
next if !a
a.each do |state_id2, nterm_id2|
n = @states.nterms.find {|n| n.token_id == nterm_id2 }
io << " (Rule: #{rule}) -> (State #{state_id2}, #{n.id.s_value})\n"
end
end
io << "\n"
# Report follow_sets
io << " [Follow sets]\n"
follow_sets = @states.follow_sets
@states.nterms.each do |nterm|
terms = follow_sets[[state.id, nterm.token_id]]
next if !terms
terms.each do |sym|
io << " #{nterm.id.s_value} -> #{sym.id.s_value}\n"
end
end
io << "\n"
# Report LA
io << " [Look-Ahead Sets]\n"
tmp = []
max_len = 0
@states.rules.each do |rule|
syms = @states.la[[state.id, rule.id]]
next if !syms
tmp << [rule, syms]
max_len = ([max_len] + syms.map {|s| s.id.s_value.length }).max
end
tmp.each do |rule, syms|
syms.each do |sym|
io << " #{sym.id.s_value.ljust(max_len)} reduce using rule #{rule.id} (#{rule.lhs.id.s_value})\n"
end
end
io << "\n" if !tmp.empty?
end
# End of Report State
io << "\n"
end
end
end
end
+3
View File
@@ -0,0 +1,3 @@
module Lrama
VERSION = "0.6.9".freeze
end
+25
View File
@@ -0,0 +1,25 @@
module Lrama
class Warning
attr_reader :errors, :warns
def initialize(out = STDERR)
@out = out
@errors = []
@warns = []
end
def error(message)
@out << message << "\n"
@errors << message
end
def warn(message)
@out << message << "\n"
@warns << message
end
def has_error?
!@errors.empty?
end
end
end
+71
View File
@@ -0,0 +1,71 @@
<%# b4_shared_declarations -%>
<%-# b4_cpp_guard_open([b4_spec_mapped_header_file]) -%>
<%- if output.spec_mapped_header_file -%>
#ifndef <%= output.b4_cpp_guard__b4_spec_mapped_header_file %>
# define <%= output.b4_cpp_guard__b4_spec_mapped_header_file %>
<%- end -%>
<%-# b4_declare_yydebug & b4_YYDEBUG_define -%>
/* Debug traces. */
#ifndef YYDEBUG
# define YYDEBUG 0
#endif
#if YYDEBUG && !defined(yydebug)
extern int yydebug;
#endif
<%= output.percent_code("requires") %>
<%-# b4_token_enums_defines -%>
/* Token kinds. */
#ifndef YYTOKENTYPE
# define YYTOKENTYPE
enum yytokentype
{
<%= output.token_enums -%>
};
typedef enum yytokentype yytoken_kind_t;
#endif
<%-# b4_declare_yylstype -%>
<%-# b4_value_type_define -%>
/* Value type. */
#if ! defined YYSTYPE && ! defined YYSTYPE_IS_DECLARED
union YYSTYPE
{
#line <%= output.grammar.union.lineno %> "<%= output.grammar_file_path %>"
<%= output.grammar.union.braces_less_code %>
#line [@oline@] [@ofile@]
};
typedef union YYSTYPE YYSTYPE;
# define YYSTYPE_IS_TRIVIAL 1
# define YYSTYPE_IS_DECLARED 1
#endif
<%-# b4_location_type_define -%>
/* Location type. */
#if ! defined YYLTYPE && ! defined YYLTYPE_IS_DECLARED
typedef struct YYLTYPE YYLTYPE;
struct YYLTYPE
{
int first_line;
int first_column;
int last_line;
int last_column;
};
# define YYLTYPE_IS_DECLARED 1
# define YYLTYPE_IS_TRIVIAL 1
#endif
<%-# b4_declare_yyerror_and_yylex. Not supported -%>
<%-# b4_declare_yyparse -%>
int yyparse (<%= output.parse_param %>);
<%= output.percent_code("provides") %>
<%-# b4_cpp_guard_close([b4_spec_mapped_header_file]) -%>
<%- if output.spec_mapped_header_file -%>
#endif /* !<%= output.b4_cpp_guard__b4_spec_mapped_header_file %> */
<%- end -%>
File diff suppressed because it is too large Load Diff
+40
View File
@@ -0,0 +1,40 @@
<%# b4_generated_by -%>
/* A Bison parser, made by Lrama <%= Lrama::VERSION %>. */
<%# b4_copyright -%>
/* Bison interface for Yacc-like parsers in C
Copyright (C) 1984, 1989-1990, 2000-2015, 2018-2021 Free Software Foundation,
Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>. */
/* As a special exception, you may create a larger work that contains
part or all of the Bison parser skeleton and distribute that work
under terms of your choice, so long as that work isn't itself a
parser generator using the skeleton or a modified version thereof
as a parser skeleton. Alternatively, if you modify or redistribute
the parser skeleton itself, you may (at your option) remove this
special exception, which will cause the skeleton and the resulting
Bison output files to be licensed under the GNU General Public
License without this special exception.
This special exception was added by the Free Software Foundation in
version 2.2 of Bison. */
<%# b4_disclaimer -%>
/* DO NOT RELY ON FEATURES THAT ARE NOT DOCUMENTED in the manual,
especially those whose name start with YY_ or yy_. They are
private implementation details that can be changed or removed. */
<%= output.render_partial("bison/_yacc.h") %>