- Reusing the same 2-dimensional integer array for processing line comments in the same Tokenizer.
- Use ArrayPool<int> in PatternPositionVisitor to avoid creating transient int[].
Expands line continuance for pipelines to allow lines to continue automatically without backticks with the pipe symbol at the start of a line.
This adds to the existing functionality where pipes can be used to continue lines by placing them at the end of a line.
Fixes#7557
* Adds support for binary parsing in format echoing hex: `0b11010110`
* Works with all existing type suffixes and multipliers.
* Supports arbitrary length parsing with `n` suffix using BigInteger; details below.
* Adds `NumberFormat` enum to specify hex/binary/base 10 for the tokenizer, replacing old `bool hex`.
* Adds `n` suffix for all numeric literals to support returning value as a `BigInteger` if requested. This bypasses the issue of large literals losing accuracy when they cast through `double`.
* Adds tests for all new behaviours.
---
### Binary / Hex Parsing Implementation
* Mimics old sign bit behaviour for int and long types. Sign bits accepted for 8 or 16-bit Hex parsing, and 8, 16, 32, 64 for binary.
* i.e., `0xFFFFFFFF -eq ([int]-1)` and `0xFFFFFFFFFFFFFFFF -eq ([long]-1)`, but suffixing `u` creates `int.MaxValue` and `long.MaxValue`, respectively, instead.
* Sign bits higher than this are accepted for bigint-suffixed numerals:
* Hex: Bigint-suffixed hex treats the high bit of any literal with a length multiple of 8 as the sign bit
* Binary: Bigint-suffixed binary accepts sign bits at 96 and 128 chars, and from there on every 8 characters.
* Prefixing the literal with a 0 will bypass this and be treated as unsigned, e.g. `0b011111111`
* Specifying an `u`nsigned suffix (or combination suffix that includes `u`) ignores sign bits, similar to how parsing a hex string using `[Convert]::ToUint32()` would do so.
* Supports negating literals using `-` prefix. This can result in positive numbers due to sign bits being permitted, just like hex literals.
---
### Refactored numeric tokenizer parsing
**New flow:**
1. Check for `real` (`.01`, `0.0`, or `0e0` syntaxes)
1. If the decimal suffix is present, TryParse directly into decimal. If the TryParse fails, TryGetNumberValue returns `false`.
2. TryParse as `Double`, and apply multiplier to value. If the TryParse fails, TryGetNumberValue returns `false`.
1. Check type suffixes and attempt to cast into appropriate type. This will return `false` if the value exceeds the specified type's bounds.
2. Default to parsing as `double` where no suffix has been applied.
2. Check number format.
* If binary, manually parse into BigInteger using optimized helper function to directly construct the BigInteger bytes from the string.
* If hex, TryParse into `BigInteger` using some special casing to retain original behaviours in int/long ranges.
* If neither binary nor hex, TryParse normally as a `BigInteger`.
3. Apply multiplier value before attempting any casts to ensure type bounds can be appropriately checked without overflows.
4. Check type suffixes.
* If a specific type suffix is used, check type bounds and attempt to parse into that type.
* If the value exceeds the type's available values, the parse fails. Otherwise, a straight cast is performed.
5. If no suffix is used, the following types are bounds-checked, in order, resulting in the first successful test determining the type of the number.
* `int`
* `long`
* `decimal` (base-10 literals only)
* `double` (base-10 literals only)
* ~~`BigInteger` for binary or hex literals.~~ If the value is outside `long` range (for hex and binary) or `double` range (for base 10), the parse will fail; higher values must be explicitly requested using the `n`/`N` BigInteger suffix.
---
*This is a breaking change* as binary literals are now read as numbers instead of generic tokens which could potentially have been used as function / cmdlet names or file names.
Notes:
* Binary literal support was approved by the committee in #7557
* ~~The same issue is still under further discussion for underscore support in numeric literals and whether BigInteger parsing ought to be exposed to the user at all.~~
* ~~Supporting underscore literals is a further breaking change causing some generic tokens like `1_000_000` to be read as numerals instead.~~ Per @SteveL-MSFT's [comment](https://github.com/PowerShell/PowerShell/pull/7993#issuecomment-442651543) this proposal was rejected.
* ~~Removing underscore support or preventing standard parsing from accepting BigInteger ranges is a relatively trivial matter. It is my personal opinion that there is no particular reason *not* to hand the user a BigInteger when they enter a sufficiently large literal, but I will defer to the PowerShell Committee's judgement on this.~~
- Adds y suffix that is used to specify a numeric literal as the sbyte data type.
- Can be combined with the existing u suffix as uy to specify the byte data type.
Many error messages in PowerShell currently use a LINQ expression to pass both the name of the error and the message through in the error processing. This PR uses the `nameof` feature to carry the error name/ID and gets rid of LINQ expression reflection, hopefully improving performance, especially in editor scenarios.
Based on standard practices, we need to have a copyright and license notice at the top of each source file. Removed existing copyrights and updated/added copyright notices for .h, .cpp, .cs, .ps1, and .psm1 files.
Updated module manifests for consistency to have Author = "PowerShell" and Company = "Microsoft Corporation". Removed multiple line breaks.
Separate PR coming to update contribution document for new source files: #6140
Manually reviewed each change.
Fix#6073
The tokenizer did multiple scans the script line to get tokens. Before
the fix the tokenizer on the first pass examined that string as
double-quoted (Expandable) string not as here string, figured the
average double quotation mark as a closing and then starting with the
single quotation mark continued processing the line as single-quoted
string which had no closing single quotation mark.
The fix is to stop the first scan pass after getting '=' (assume
assignment-expression '$a=' for next pass).
Our private implementation of FailFast was likely introduced
when each individual application needed it's own code to
get Watson reports. The CLR takes care of this for us now,
so we don't need our implementation.
Our method CheckForSevereException was also introduced in
the early days of the CLR - the exceptions it was checking
for aren't actually raised the CLR anymore, they just FailFast.
I removed them as there is a tiny bit of overhead (in code size)
and also in the generated code, e.g. dynamic sites called the method.
I (Jason Shirk) ran https://github.com/dotnet/codeformatter with the default rules, basically:
codeformatter /nocopyright "/c:DEBUG,UNIX,CORECLR" @files.rsp
This caused a few problems building, which were fixed up manually.
Notable changes:
`this.` is removed unless needed to disambiguate.
private instance fields are renamed to have a `_` prefix.
private static fields are renamed to have a `s_` prefix.
I left some projects alone (like PackageManagement) and also left some generated code alone.
Shayde Nofziger