update testing guidelines

* correct links in test-guidelines.md (issue #1665) added specific links for PesterDoAndDont and WritingPesterTests added new section in WritingPesterTests.md on bulk skipping tests * update test directories to include a directory for remoting tests * fix lines in document to use semantic line breaks also remove trailing spaces after "."
2026-06-08 12:12:50 +00:00 · 2016-09-27 12:55:47 -07:00
parent d04460657d
commit 379cc52b95
2 changed files with 91 additions and 20 deletions
@@ -203,6 +203,69 @@ Passed: 1 Failed: 0 Skipped: 0 Pending: 0
 The DESCRIBE BeforeAll block is executed before any other code even though it was at the bottom of the Describe block, so if state is set elsewhere in the describe BLOCK, that state will not be visible (as the code will not yet been run). Notice, too, that the BEFOREALL block in Context is executed before any other code in that block.
 Generally, you should have code reside in one of the code block elements of `[Before|After][All|Each]`, especially if those block rely on state set by free code elsewhere in the block.

+#### Skipping tests in bulk
+Sometimes it is beneficial to skip all the tests in a particular `Describe` block. For example, tests which are not applicable to a platform could be skipped, and they would be reported as skipped. The following is an example of how this may be done:
+```powershell
+Describe "Should not run these tests on non-Windows platforms" {
+    BeforeAll {
+        $originalDefaultParameterValues = $PSDefaultParameterValues.Clone()
+        if ( ! $IsWindows ) {
+            $PSDefaultParameterValues["it:skip"] = $true
+        }
+    }
+    AfterAll {
+        $global:PSDefaultParameterValues = $originalDefaultParameterValues
+    }
+    Context "Block 1" {
+        It "This block 1 test 1" {
+            1 | should be 1
+        }
+        It "This is block 1 test 2" {
+            1 | should be 1
+        }
+    }
+    Context "Block 2" {
+        It "This block 2 test 1" {
+            2 | should be 1
+        }
+        It "This is block 2 test 2" {
+            2 | should be 1
+        }
+    }
+}  
+```
+Here is the output when run on a Linux distribution:
+```
+Describing Should not run these tests on non-Windows platforms
+   Context Block 1
+    [!] This block 1 test 1 691ms
+    [!] This is block 1 test 2 114ms
+   Context Block 2
+    [!] This block 2 test 1 73ms
+    [!] This is block 2 test 2 6ms
+```
+and here is the output when run on a Windows distribution:
+```
+Describing Should not run these tests on non-Windows platforms
+   Context Block 1
+    [+] This block 1 test 1 86ms
+    [+] This is block 1 test 2 33ms
+   Context Block 2
+    [-] This block 2 test 1 52ms
+      Expected: {1}
+      But was:  {2}
+      22:             2 | should be 1
+      at <ScriptBlock>, <No file>: line 22
+    [-] This is block 2 test 2 77ms
+      Expected: {1}
+      But was:  {2}
+      25:             2 | should be 1
+      at <ScriptBlock>, <No file>: line 25
+```
+
+this technique uses the `$PSDefaultParameterValues` feature of PowerShell to temporarily set the It block parameter `-skip` to true (or in the case of Windows, it is not set at all)
+
+
 #### Multi-line strings

 You may want to have a test like
@@ -4,13 +4,12 @@
 Testing is a critical and required part of the PowerShell project.

 The Microsoft PowerShell team created nearly 100,000 tests over the last 12 years which we run as part of the release process for Windows PowerShell.
-Having all of those tests available for the initial release of PowerShell was not feasible, and we have targeted those tests which
-we believe will provide us the ability to catch regressions in the areas which have had the largest changes for PowerShell. 
+Having all of those tests available for the initial release of PowerShell was not feasible, and we have targeted those tests which we believe will provide us the ability to catch regressions in the areas which have had the largest changes for PowerShell.
 It is our intent to continue to release more and more of our tests until we have the coverage we need.

-For creating new tests, please review the 
-[documents](https://github.com/PowerShell/PowerShell/tree/master/docs/testing-guidelines) on how to
-create tests for PowerShell.
+For creating new tests, please review the [documents](https://github.com/PowerShell/PowerShell/tree/master/docs/testing-guidelines) on how to create tests for PowerShell.
+There is a best practices document for [writing Pester tests](https://github.com/PowerShell/PowerShell/tree/master/docs/testing-guidelines/WritingPesterTests.md).
+When adding new tests, place them in the directories as [outlined below](#test-layout).

 ## CI System

@@ -30,7 +29,7 @@ From there you can easily navigate to the build history.

 ### Travis CI

-Travis CI works similarly to AppVeyor. 
+Travis CI works similarly to AppVeyor.
 For Travis CI there will be multiple badges.
 The badges indicate the last build status of `master` branch for different platforms.
 Hopefully, it's green:
@@ -46,27 +45,36 @@ CI System builds (AppVeyor and Travis CI) and runs tests on every pull request a

 ![AppVeyor-Github](Images/AppVeyor-Github.png)

-These green check boxes and red crosses are **clickable** as well. 
-They will bring you to the corresponding page with details. 
+These green check boxes and red crosses are **clickable** as well.
+They will bring you to the corresponding page with details.

 ## Test Frameworks
 ### Pester
-Our script-based test framework is [Pester](https://github.com/Pester/Pester). 
+Our script-based test framework is [Pester](https://github.com/Pester/Pester).
 This is the framework which we are using internally at Microsoft for new script-based tests, 
-and a large number of the tests which are part of the PowerShell project have been migrated from that test base. 
+and a large number of the tests which are part of the PowerShell project have been migrated from that test base.
 Pester tests can be used to test most of PowerShell behavior (even some API operations can easily be tested in Pester).

-Substantial changes were required, to get Pester executing on non-Windows systems. 
-These changes are not yet in the official Pester code base. 
-Some features of Pester may not be available or may have incorrect behavior. 
+Substantial changes were required, to get Pester executing on non-Windows systems.
+These changes are not yet in the official Pester code base.
+Some features of Pester may not be available or may have incorrect behavior.
 Please make sure to create issues in [PowerShell/PowerShell](https://github.com/PowerShell/PowerShell/issues) (not Pester) for anything that you find.
+#### Test Tags
+The Pester framework allows `Describe` blocks to be tagged, and our CI system relies on those tags to invoke our tests.
+One of the following tags must be used:
+* `CI` - this tag indicates that the tests in the `Describe` block will be executed as part of the CI/PR process
+* `Feature` - tests with this tag will not be executed as part of the CI/PR process, but they will be executed on a daily basis as part of a `cron` driven build
+They indicate that the test will be validating more behavior, or will be using remote network resources (ex: package management tests)
+* `Scenario` - this tag indicates a larger scale test interacting with multiple areas of functionality and/or remote resources, these tests are also run daily.

+Additionally, the tag:
+* `SLOW` indicates that the test takes a somewhat longer time to execute (97% of our `CI` tests take 100ms or less), a test which takes longer than 1 second should be considered as a candidate for being tagged `Slow`
 ### xUnit
-For those tests which are not easily run via Pester, we have decided to use [xUnit](https://xunit.github.io/) as the test framework. 
+For those tests which are not easily run via Pester, we have decided to use [xUnit](https://xunit.github.io/) as the test framework.
 Currently, we have a minuscule number of tests which are run by using xUnit.

 ## Running tests outside of CI
-When working on new features or fixes, it is natural to want to run those tests locally before making a PR. 
+When working on new features or fixes, it is natural to want to run those tests locally before making a PR.
 Two helper functions are part of the build.psm1 module to help with that:
 * `Start-PSPester` will execute all Pester tests which are run by the CI system
 * `Start-PSxUnit` will execute the available xUnit tests run by the CI system
@@ -91,17 +99,16 @@ Start-PSPester -Directory test/powershell/engine/Api -Test XmlAdapter.Tests.Api

 ### What happens after your PR?
 When your PR has successfully passed the CI test gates, your changes will be used to create PowerShell binaries which can be run
-in Microsoft's internal test frameworks. 
-The tests that you created for your change and the library of historical tests will be run to determine if any regressions are present. 
+in Microsoft's internal test frameworks.
+The tests that you created for your change and the library of historical tests will be run to determine if any regressions are present.
 If these tests find regressions, you'll be notified that your PR is not ready, and provided with enough information to investigate why the failure happened.



 ## Test Layout
-We have taken a functional approach to the layout of our Pester tests. 
-You should place new tests in their appropriate location. 
+We have taken a functional approach to the layout of our Pester tests and you should place new tests in their appropriate location.
 If you are making a fix to a cmdlet in a module, the test belongs in the module directory.
-If you are unsure, you can make it part of your PR, or create an issue. 
+If you are unsure, you can make it part of your PR, or create an issue.
 The current layout of tests is:
 * test/powershell/engine
 * test/powershell/engine/Api
@@ -111,6 +118,7 @@ The current layout of tests is:
 * test/powershell/engine/Logging
 * test/powershell/engine/Module
 * test/powershell/engine/ParameterBinding
+* test/powershell/engine/Remoting
 * test/powershell/engine/Runspace
 * test/powershell/engine/Logging/MessageAnalyzer
 * test/powershell/Host