From 6b9283f3bf6b6e370b06642532dc7fe07fd7ba45 Mon Sep 17 00:00:00 2001 From: "Jim Truher (MSFT)" Date: Mon, 25 Apr 2016 12:27:35 -0700 Subject: [PATCH 1/2] add testing files to repo --- docs/Testing/PesterDoAndDont.md | 23 ++++ docs/Testing/Testing.md | 107 ++++++++++++++++ docs/Testing/WritingPesterTests.md | 191 +++++++++++++++++++++++++++++ 3 files changed, 321 insertions(+) create mode 100644 docs/Testing/PesterDoAndDont.md create mode 100644 docs/Testing/Testing.md create mode 100644 docs/Testing/WritingPesterTests.md diff --git a/docs/Testing/PesterDoAndDont.md b/docs/Testing/PesterDoAndDont.md new file mode 100644 index 0000000000..56bcfe8571 --- /dev/null +++ b/docs/Testing/PesterDoAndDont.md @@ -0,0 +1,23 @@ +## Do +1. Name your files .tests.ps1 +2. Keep tests simple + 1. Test only what you need + 2. Reduce dependencies +3. Be sure to tag your Describe blocks with "inner" and "outer" +4. Make sure that Describe/Context/It descriptions are useful + 1. The error message should not be the place where you describe the test +5. Use "Context" to group tests + 1. Multiple Contexts can help you group your test suite into logical sections +6. Use BeforeAll/AfterAll/BeforeEach/AfterEach instead of custom initiators +7. Use Try-Catch for expected errors and check $_.fullyQualifiedErrorId +8. Loop It blocks for checking multiple properties +9. Use code coverage functionality where appropriate +10. Use Mock functionality when you don't have your entire environment +11. Avoid free code in a Describe block + 1. Use `[Before|After][Each|All]` _see Free Code in a Describe block_ + +## Don't +1. Have too many evaluations in a single It block + 1. The first "Should" failure will stop that block +2. Don't use "Should" anywhere but within an "It" Block + diff --git a/docs/Testing/Testing.md b/docs/Testing/Testing.md new file mode 100644 index 0000000000..3edc4d7780 --- /dev/null +++ b/docs/Testing/Testing.md @@ -0,0 +1,107 @@ +<<<<<<< d74cd7b90a57dd043736c99c54f37e417ed3442d +# DRAFT + +_I have more questions than answers_ + + +#### Current Test Infrastructure +We currently rely heavily on STEX environment for our testing, and we will continue to do so through the Server2016 release. We +need to use that current infrastructure to continue to test full PowerShell builds; it should be possible to build automation +which takes a full PowerShell build and lay it on an existing lab system, update the appropriate test files and execute +a test pass in the same way that we do with official builds. + +The test artifacts which are applicable to full PowerShell are not universally applicable to Core/Nano/OtherPlatform, we will need +to create tooling which allows us to apply a set of test artifacts to a configuration, and then execute tests. Eventually, we need +to have our CI environment test all the flavors of PowerShell we create. +**Question**: Can AppVeyor/Travis service that need? + + +#### Organization +**Proposal**: Create 3 tiers of testing: + +* Checkin + * These are run as part of the CI process, and should run quickly. How quickly is an open question. We need to determine + the right amount of coverage without spending too much time. It may be that we can improve our coverage here through parallelization + but we have not investigated enough to determine whether it's possible. +* Feature + * the tests which look at corner cases, and stand-alone modules (for example, the archive module tests could fall into this + category) +* Scenario + * these are tests which span features, and determine whether the whole product is working correctly. The current P3 tests fall + largely here + +**Actions**: Decide what goes where. My initial thoughts are to migrate our current TTEST unittests into tier 1 (Checkin) + +**Current Migration Activity** + +We have teams working on migrating tests which are in non-portable frameworks (TTest, Lite1, Lite3, etc) to portable frameworks. +The first effort is to migrate our TTEST cmdlet unit tests to Pester, we should be taking those migrated tests and get them into +SD + +##### Layout +We need to have a reasonable layout of our tests, not sure what that looks like yet. We need to make it +easy to find both feature and test code to reduce our maintainance burden. + +##### Self Hosting +Self-Hosting remains problematic while are still so early in the development phase, but it is _imperative_ +that we dog food as early as possible. This is especially true on the non-Windows platforms where we have made +assumptions about the working environment with regard to a number of issues: +* removal of well known aliases +* case sensitivity of some operations +* coverage +We should be using these non-windows platforms as much as possible to + +======= +# DRAFT + +_I have more questions than answers_ + + +#### Current Test Infrastructure +We currently rely heavily on STEX environment for our testing, and we will continue to do so through the Server2016 release. We +need to use that current infrastructure to continue to test full PowerShell builds; it should be possible to build automation +which takes a full PowerShell build and lay it on an existing lab system, update the appropriate test files and execute +a test pass in the same way that we do with official builds. + +The test artifacts which are applicable to full PowerShell are not universally applicable to Core/Nano/OtherPlatform, we will need +to create tooling which allows us to apply a set of test artifacts to a configuration, and then execute tests. Eventually, we need +to have our CI environment test all the flavors of PowerShell we create. +**Question**: Can AppVeyor/Travis service that need? + + +#### Organization +**Proposal**: Create 3 tiers of testing: + +* Checkin + * These are run as part of the CI process, and should run quickly. How quickly is an open question. We need to determine + the right amount of coverage without spending too much time. It may be that we can improve our coverage here through parallelization + but we have not investigated enough to determine whether it's possible. +* Feature + * the tests which look at corner cases, and stand-alone modules (for example, the archive module tests could fall into this + category) +* Scenario + * these are tests which span features, and determine whether the whole product is working correctly. The current P3 tests fall + largely here + +**Actions**: Decide what goes where. My initial thoughts are to migrate our current TTEST unittests into tier 1 (Checkin) + +**Current Migration Activity** + +We have teams working on migrating tests which are in non-portable frameworks (TTest, Lite1, Lite3, etc) to portable frameworks. +The first effort is to migrate our TTEST cmdlet unit tests to Pester, we should be taking those migrated tests and get them into +SD + +##### Layout +We need to have a reasonable layout of our tests, not sure what that looks like yet. We need to make it +easy to find both feature and test code to reduce our maintainance burden. + +##### Self Hosting +Self-Hosting remains problematic while are still so early in the development phase, but it is _imperative_ +that we dog food as early as possible. This is especially true on the non-Windows platforms where we have made +assumptions about the working environment with regard to a number of issues: +* removal of well known aliases +* case sensitivity of some operations +* coverage +We should be using these non-windows platforms as much as possible to + +>>>>>>> Create Testing.md diff --git a/docs/Testing/WritingPesterTests.md b/docs/Testing/WritingPesterTests.md new file mode 100644 index 0000000000..88c532e78c --- /dev/null +++ b/docs/Testing/WritingPesterTests.md @@ -0,0 +1,191 @@ +Because we are planning to continue to extend Pester to support remote/parallel execution, it's important to keep in mind the following when create tests: +* Tests should not be overly complicated and test too many things + * boil down your tests to their essence, test only what you need +* Tests should be as simple as they can +* Tests should generally not rely on any other test + + +Examples: +Here's the simplest of tests + +``` +Describe "A variable can be assigned and retrieved" { + It "Create a variable and make sure it's value is correct" { + $a = 1 + $a | Should be 1 + } +} +``` + +If you need to do type checking, that can be done as well + +``` +Describe "One is really one" { + It "Compare 1 to 1" { + $a = 1 + $a | Should be 1 + } + It "1 is really an int" { + $i = 1 + $i.GetType().FullName | Should Be System.Int32 + } +} +``` +If you are checking for proper errors, do that in a try catch, and then check fully qualified error id +``` +... +it "Error should be PathNotFound" { + try + { + get-item "ThisFileCannotPossiblyExist" -ErrorAction Stop + throw "No Exception!" + } + catch + { + $_.FullyQualifiedErrorId | + should be "PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand" + } +} +``` + +Note that if get-item were to succeed, a different FullyQualifiedErrorId would be thrown and the test will fail because the FQErrorId is wrong. This is the suggested path because Pester wants to check the error message, which will likely not work here because of localized builds, but the FullyQualifiedErrorId is constant regardless of the locale. + +### Describe/Context/It + +From an organizational standpoint, a Describe block roughly compares to a LITE3 Test Suite and an It block roughly compares to a LITE3 testcase. Unlike LITE3, individual tests are not prioritized (remember that a test name in Pester is really a descriptive sentence, rather than a name), but a Describe block (TestSuite) may be tagged with a string. +For creation of PowerShell tests, the Describe block is the level of granularity suggested and one of two tags should be used: "Inner" or "Outer". If the tag is not provided, tests in that describe block will be run any time tests are executed. + +#### Describe +Creates a logical group of tests. All Mocks and TestDrive contents defined within a Describe block are scoped to that Describe ; they will no longer be present when the Describe block exits. A Describe block may contain any number of Context and It blocks. + +#### Context +Provides logical grouping of It blocks within a single Describe block. Any Mocks defined inside a Context are removed at the end of the Context scope, as are any files or folders added to the TestDrive during the Context block's execution. Any BeforeEach or AfterEach blocks defined inside a Context also only apply to tests within that Context . + +#### It +The It block is intended to be used inside of a Describe or Context Block. If you are familiar with the AAA pattern (Arrange-Act-Assert), the body of the It block is the appropriate location for an assert. The convention is to assert a single expectation for each It block. The code inside of the It block should throw a terminating error if the expectation of the test is not met and thus cause the test to fail. The name of the It block should expressively state the expectation of the test. + +### Selected Features + +#### Test Drive +A PSDrive is available for file activity during a tests and this drive is limited to the scope of a single Describe block. The contents of the drive are cleared when a context block is exited. +A test may need to work with file operations and validate certain types of file activities. It is usually desirable not to perform file activity tests that will produce side effects outside of an individual test. Pester creates a PSDrive inside the user's temporary drive that is accessible via a names PSDrive TestDrive:. Pester will remove this drive after the test completes. You may use this drive to isolate the file operations of your test to a temporary store. + +The following example illustrates the feature: + +``` +function Add-Footer($path, $footer) { + Add-Content $path -Value $footer +} + +Describe "Add-Footer" { + $testPath="TestDrive:\test.txt" + Set-Content $testPath -value "my test text." + Add-Footer $testPath "-Footer" + $result = Get-Content $testPath + + It "adds a footer" { + (-join $result) | Should Be("my test text.-Footer") + } +} +``` + +When this test completes, the contents of the TestDrive PSDrive will be removed. + +#### Parameter Generation +This is quite a bit different from LITE3, but still possible, see the following: + +``` +Describe "A test" { +function Test-Xor { + param ($a, $b, $ExpectedResult) + It ("Applies XOR to inputs {0} and {1}" -f $a, $b) { + $a -xor $b | Should be $ExpectedResult + } +} + +$testCases = @( + @{ a = 0; b = 1; ExpectedResult = 1 } + @{ a = 1; b = 0; ExpectedResult = 1 } + @{ a = 1; b = 1; ExpectedResult = 0 } + @{ a = 0; b = 0; ExpectedResult = 0 } + ) + foreach ($testCase in $testCases) { + Test-Xor @testCase + } +} +``` + +You can construct the values to pass as parameters, including the expected value, and use that iteratively in a loop. Note the location of the `It` block + +#### Mocking +Mocks the behavior of an existing command with an alternate implementation. This creates new behavior for any existing command within the scope of a Describe or Context block. The function allows you to specify a script block that will become the command's new behavior. +The following example illustrates simple use: +``` +Context "Get-Random is not random" { + Mock Get-Random { return 3 } + It "Get-Random returns 3" { + Get-Random | Should be 3 + } + } +``` +More information may be found here: https://github.com/pester/Pester/wiki/Mock +### Free Code in a Describe block +Code execution in Pester can be very subtle and can cause issues when executing test code. The execution of code which lays outside of the usual code blocks may not happen as you expect. Consider the following: +``` +Describe it { + Write-Host -For DarkRed "Before Context" + Context "subsection" { + Write-Host -for DarkRed "Before BeforeAll" + BeforeAll { write-host -for Blue "In Context BeforeAll" } + Write-Host -for DarkRed "After BeforeAll" +  + Write-Host -for DarkRed "Before AfterAll" + AfterAll { Write-Host -for Blue "In Context AfterAll" } + Write-Host -for DarkRed "After AfterAll" +  + BeforeEach { Write-Host -for Blue "In BeforeEach" } + AfterEach { Write-Host -for Blue "In AfterEach" } +  + Write-Host -for DarkRed "Before It" + It "should not be a surprise" { + 1 | should be 1 + } + Write-Host -for DarkRed "After It" + } + Write-Host -for DarkRed "After Context" + Write-Host -for DarkGreen "Before Describe BeforeAll" + BeforeAll { Write-Host -for DarkGreen "In Describe BeforeAll" } + AfterAll { Write-Host -for DarkGreen "In Describe AfterAll" } +} +``` + +Now, when run, you can see the execution schedule + +``` +PS# invoke-pester c:\temp\pester.demo.tests.ps1 +Describing it +In Describe BeforeAll +Before Context + Context subsection +In Context BeforeAll +Before BeforeAll +After BeforeAll +Before AfterAll +After AfterAll +Before It +In BeforeEach + [+] should not be a surprise 79ms +In AfterEach +After It +In Context AfterAll +After Context +Before Describe BeforeAll +In Describe AfterAll +Tests completed in 79ms +Passed: 1 Failed: 0 Skipped: 0 Pending: 0 +``` + +The DESCRIBE BeforeAll block is executed before any other code even though it was at the bottom of the Describe block, so if state is set elsewhere in the describe BLOCK, that state will not be visible (as the code will not yet been run). Notice, too, that the BEFOREALL block in Context is executed before any other code in that block. +Generally, you should have code reside in one of the code block elements of `[Before|After][All|Each]`, especially if those block rely on state set by free code elsewhere in the block. + + From 7e156b4abd2b2cd3f32819df457070cedac7ab69 Mon Sep 17 00:00:00 2001 From: "James Truher [MSFT]" Date: Mon, 25 Apr 2016 12:41:54 -0700 Subject: [PATCH 2/2] remove extraneous line --- docs/Testing/Testing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/Testing/Testing.md b/docs/Testing/Testing.md index 3edc4d7780..d322cee698 100644 --- a/docs/Testing/Testing.md +++ b/docs/Testing/Testing.md @@ -1,4 +1,3 @@ -<<<<<<< d74cd7b90a57dd043736c99c54f37e417ed3442d # DRAFT _I have more questions than answers_