Update

README.md

@@ -124,7 +124,7 @@ ntdll!LdrpLoadContextReplaceModule+0x126:
 
 Everything else is infrastructure to support this work offloading mechanism.
 
-The `ntdll!LdrpQueueWork` function is how modules are added to the `ntdll!LdrpWorkQueue` linked list data structure.  The start of a loader worker thread (in the `ntdll!LdrpWorkCallback` function) accesses the `ntdll!LdrpWorkQueue` list to pick up a work item. Access to the shared `ntdll!LdrpWorkQueue` data structure is protected by the `ntdll!LdrpWorkQueueLock` critical section lock.
+The `ntdll!LdrpQueueWork` function is how modules are added to the `ntdll!LdrpWorkQueue` linked list data structure.  Work processors (i.e. callers of `ntdll!LdrpProcessWork`) such as loader worker threads or a [concurrent `LoadLibrary`](windows/code/loadlibrary-concurrent-work) access the `ntdll!LdrpWorkQueue` list to pick up a work item. Access to the `ntdll!LdrpWorkQueue` shared data structure is protected by the `ntdll!LdrpWorkQueueLock` critical section lock.
 
 Each list entry in the `ntdll!LdrpWorkQueue` data structure is a `LDRP_LOAD_CONTEXT` structure. This structure is undocumented by Microsoft because its contents are not in the public debug symbols. Each `LDRP_LOAD_CONTEXT` structure relates directly to one module because a module's `LDR_DATA_TABLE_ENTRY` structure is allocated at the same time as its `LDRP_LOAD_CONTEXT` structure in the `LdrpAllocatePlaceHolder` function. In addition, the first member of each `LDRP_LOAD_CONTEXT` structure is a `UNICODE_STRING` of the `BaseDllName` according to the module that it relates to.
 
@@ -349,7 +349,7 @@ A typical library load ranges from <code>LdrModulesPlaceHolder</code> to <code>L
   <tr>
     <th>LdrModulesWaitingForDependencies (3)</th>
     <td><code>LdrpLoadDependentModule</code></td>
-    <td>This state isn't typically set, but during a trace, I was able to observe the loader set it by launching a web browser (Google Chrome) under WinDbg, which triggered the watchpoint in this function when loading app compatibility DLL <code>C:\Windows\System32\ACLayers.dll</code>. Interstingly, the <code>LDR_DDAG_STATE</code> decreases by one here from <code>LdrModulesSnapping</code> to <code>LdrModulesWaitingForDependencies</code>; the only time I've observed this. <code>LdrpModuleDatatableLock</code> is held during this state change.</td>
+    <td>This state isn't typically set, but during a trace, I was able to observe the loader set it by launching a web browser (Google Chrome) under WinDbg, which triggered the watchpoint in this function when loading app compatibility DLL <code>C:\Windows\System32\ACLayers.dll</code>. Interestingly, the <code>LDR_DDAG_STATE</code> decreases by one here from <code>LdrModulesSnapping</code> to <code>LdrModulesWaitingForDependencies</code>; the only time I've observed this. <code>LdrpModuleDatatableLock</code> is held during this state change.</td>
   </tr>
   <tr>
     <th>LdrModulesSnapping (4)</th>
@@ -364,7 +364,7 @@ A typical library load ranges from <code>LdrModulesPlaceHolder</code> to <code>L
   <tr>
     <th>LdrModulesCondensed (6)</th>
     <td><code>LdrpCondenseGraphRecurse</code</td>
-    <td>This function recevies a <code>LDR_DDAG_NODE</code> as its first argument and recursively calls itself to walk <code>LDR_DDAG_NODE.Dependencies</code>. On every recursion, this function checks whether it can remove the passed <code>LDR_DDAG_NODE</code> from the graph. If so, this function acquires <code>LdrpModuleDataTableLock</code> to call the <code>LdrpMergeNodes</code> function, which receives the same first argument, then releasing <code>LdrpModuleDataTableLock</code> after it returns. <code>LdrpMergeNodes</code> discards the uneeded node from the <code>LDR_DDAG_NODE.Dependencies</code> and <code>LDR_DDAG_NODE.IncomingDependencies</code> DAG adjacency lists of any modules starting from the given parent node (first function argument), decrements <code>LDR_DDAG_NODE.LoadCount</code> to zero, and calls <code>RtlFreeHeap</code> to deallocate <code>LDR_DDAG_NODE</code> DAG nodes. After <code>LdrpMergeNodes</code> returns, <code>LdrpCondenseGraphRecurse</code> calls <code>LdrpDestroyNode</code> to deallocate any DAG nodes in the <code>LDR_DDAG_NODE.ServiceTagList</code> list of the parent <code>LDR_DDAG_NODE</code> then deallocate the parent <code>LDR_DDAG_NODE</code> itself. <code>LdrpCondenseGraphRecurse</code> sets the state to <code>LdrModulesCondensed</code> before returning. <strong>Note:</strong> The <code>LdrpCondenseGraphRecurse</code> function and its callees rely heavily on all members of the <code>LDR_DDAG_NODE</code> structure, which needs further reverse engineering to fully understand the inner workings and "whys" of what's occurring here. <strong>Condensing is the process of discarding unnecessary nodes from the dependency graph.</strong></td>
+    <td>This function receives a <code>LDR_DDAG_NODE</code> as its first argument and recursively calls itself to walk <code>LDR_DDAG_NODE.Dependencies</code>. On every recursion, this function checks whether it can remove the passed <code>LDR_DDAG_NODE</code> from the graph. If so, this function acquires <code>LdrpModuleDataTableLock</code> to call the <code>LdrpMergeNodes</code> function, which receives the same first argument, then releasing <code>LdrpModuleDataTableLock</code> after it returns. <code>LdrpMergeNodes</code> discards the uneeded node from the <code>LDR_DDAG_NODE.Dependencies</code> and <code>LDR_DDAG_NODE.IncomingDependencies</code> DAG adjacency lists of any modules starting from the given parent node (first function argument), decrements <code>LDR_DDAG_NODE.LoadCount</code> to zero, and calls <code>RtlFreeHeap</code> to deallocate <code>LDR_DDAG_NODE</code> DAG nodes. After <code>LdrpMergeNodes</code> returns, <code>LdrpCondenseGraphRecurse</code> calls <code>LdrpDestroyNode</code> to deallocate any DAG nodes in the <code>LDR_DDAG_NODE.ServiceTagList</code> list of the parent <code>LDR_DDAG_NODE</code> then deallocate the parent <code>LDR_DDAG_NODE</code> itself. <code>LdrpCondenseGraphRecurse</code> sets the state to <code>LdrModulesCondensed</code> before returning. <strong>Note:</strong> The <code>LdrpCondenseGraphRecurse</code> function and its callees rely heavily on all members of the <code>LDR_DDAG_NODE</code> structure, which needs further reverse engineering to fully understand the inner workings and "whys" of what's occurring here. <strong>Condensing is the process of discarding unnecessary nodes from the dependency graph.</strong></td>
   </tr>
   <tr>
     <th>LdrModulesReadyToInit (7)</th>
@@ -1426,7 +1426,7 @@ How cool is that? That's like if Windows serviced your `LoadLibrary` request by
 
 Microsoft's reasoning behind the return type of `LoadLibrary` stems from the 16-bit Windows API (going back to Windows 1.0 when the function was introduced), where [libraries became conflated with data files or memory-mapped files](https://learn.microsoft.com/en-us/previous-versions/ms810501(v=msdn.10)#:~:text=Whereas%20Windows%203.1%20needs%20to%20copy%20data%20from%20an%20executable%20file%20into%20main%20memory%20to%20build%20code%20segments%20or%20load%20resources%2C%20Windows%20NT%20only%20needs%20to%20create%20a%20memory%2Dmapped%20file%20object%20that%20is%20backed%20by%20the%20executable%20file%20instead%20of%20the%20system%20pagefile.) meaning "libraries" could exist for the sole purpose of containing data with no code. In the first release of Windows NT (i.e. Windows NT 3.1), Microsoft carried forward this unique mistake while also [adding an extension for specifying that a "library" must load only as a memory-mapped file](https://devblogs.microsoft.com/oldnewthing/20141120-00/?p=43573) and how to open this file. Consequently, the modern Windows loader is stuck with maintaining a red-black tree at `ntdll!LdrpModuleBaseAddressIndex` for speeding up base address ➜ `LDR_DATA_TABLE_ENTRY` lookups (the legacy Windows loader slowly [iterated the `InLoadOrderModuleList` linked list](https://github.com/reactos/reactos/blob/053939e27cbf4d6475fb33b6fc16199bd944880d/dll/ntdll/ldr/ldrutils.c#L1610-L1611) in the PEB to do these lookups). This indirection is a contibuting factor to slow process creation on Windows (simply set a read watchpoint on `ntdll!LdrpModuleBaseAddressIndex` during process startup to see what a hot data structure this is). The performance of Windows delay loading is also negatively affected by this indirection, and the synchronization it required to access the shared data structure, because `ntdll!LdrResolveDelayLoadedAPI` calls `ntdll!LdrpFindLoadedDllByHandle` every time it runs.
 
-A reasonable person could expect that given Windows' focus on dynamic loading of [executable modules](data/windows/timeline-verification)—unlike the prevalent static linking of the time—it would have ensured a solid design for its core dynamic loading API. Especially since [Multics](https://en.wikipedia.org/wiki/Multics) had already pioneered the idea of dynamically linking libraries and [used memory-mapped files to share data](https://www.multicians.org/myths.html#nofile) years before Microsoft was even founded. Indeed, this quirk in the library loader exemplifies an avoidable misstep—one that, like virtually all Windows API oversights, exists as an outcome of Microsoft's expedient development style when creating their core technologies.
+A reasonable person could expect that given Windows' focus on dynamic loading of [executable modules](data/windows/timeline-verification)—unlike the prevalent static linking of the time—it would have ensured a solid design for its core dynamic loading API. Especially since [Multics](https://en.wikipedia.org/wiki/Multics) had already pioneered the idea of dynamically linking libraries and [used memory-mapped files to share data](https://www.multicians.org/myths.html#nofile) years before Microsoft was even founded. Indeed, this quirk in the library loader exemplifies an avoidable misstep—one that, like most Windows API oversights, exists as an outcome of Microsoft's expedient development style when creating their core technologies.
 
 An excerpt from *Windows Internals: System architecture, processes, threads, memory management, and more, Part 1 (7th edition)* states this regarding the `ntdll!LdrpModuleBaseAddressIndex` data structure (and `ntdll!LdrpMappingInfoIndex`):
 

analysis-commands.md

@@ -49,7 +49,7 @@ List all module `DdagNode` structures:
 
 Note that `$extret` is a pseudo-register that the WinDbg `!list` command specially uses to store each list entry's address as it iterates through the list.
 
-List all module `DdagNode.State` values:
+List all modules with their `DdagNode.State` values:
 
 ```
 !list -x "dt ntdll!_LDR_DATA_TABLE_ENTRY @$extret -cio -t BaseDllName; dt ntdll!_LDR_DDAG_NODE @@C++(((ntdll!_LDR_DATA_TABLE_ENTRY *)@$extret)->DdagNode) -cio -t State" @@C++(&@$peb->Ldr->InLoadOrderModuleList)

code/windows/loadlibrary-concurrent-work/README.md

@@ -0,0 +1,43 @@
+# `LoadLibrary` Concurrent Work Experiment
+
+Running two library loads concurrently to study the loader's ability for work parallelization. In particular, this experiment verifies whether or not a concurrent `LoadLibrary` will help another `LoadLibrary` in its mapping and snapping work.
+
+1. Disable loader worker threads: `reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\exe-test.exe" /v MaxLoaderThreads /t REG_DWORD /d 1 /f`
+  - This is to keep them from picking up work and messing up our experiment
+2. Set a breakpoint on the `ntdll!LdrpQueueWork` function: `bp ntdll!LdrpQueueWork`
+3. When the first `LoadLibrary` thread hits this breakpoint, continue execution until that function returns: `gu`
+4. Verify `ntdll!LdrpWorkQueue` is not empty: `!list ntdll!LdrpWorkQueue`
+5. Suspend that thread: `~n`
+6. Set a breakpoint on `ntdll!LdrpProcessWork`
+7. Continue execution: `g`
+
+Thread 2 `LoadLibrary` picking up work from from thread 1 `LoadLibrary`:
+
+```
+0:001> k
+ # Child-SP          RetAddr               Call Site
+00 000000aa`a0eff8a8 00007ffa`56b70048     ntdll!LdrpProcessWork
+01 000000aa`a0eff8b0 00007ffa`56b2fad7     ntdll!LdrpDrainWorkQueue+0x184
+02 000000aa`a0eff8f0 00007ffa`56b273e4     ntdll!LdrpLoadDllInternal+0xc3
+03 000000aa`a0eff970 00007ffa`56b26af4     ntdll!LdrpLoadDll+0xa8
+04 000000aa`a0effb20 00007ffa`54522612     ntdll!LdrLoadDll+0xe4
+05 000000aa`a0effc10 00007ff7`4f2b103e     KERNELBASE!LoadLibraryExW+0x162
+06 000000aa`a0effc80 00007ffa`56337374     exe_test!loadlibrary_thread_2+0x3e [C:\Users\user\Documents\loadlibrary-concurrent-work\exe-test.c @ 21]
+07 000000aa`a0effcb0 00007ffa`56b5cc91     KERNEL32!BaseThreadInitThunk+0x14
+08 000000aa`a0effce0 00000000`00000000     ntdll!RtlUserThreadStart+0x21
+```
+
+9. Continue execution until the return of `ntdll!LdrpProcessWork`: `gu`
+10. [List all modules with their `DdagNode.State` values](/analysis-commands.mld##ldr_ddag_node-analysis) to verify work has been done
+
+A concurrent `LoadLibrary` has sucessfully helped out another `LoadLibrary`.
+
+While in the `ntdll!LdrpDrainWorkQueue` function, the `LoadLibrary` on thread 2 thread will keep picking up work until there is none left in the `ntdll!LdrpWorkQueue`. Note that because thread 1 `LoadLibrary` never offloads (with the `ntdll!LdrpQueueWork` function) the work item for the top-level loading DLL (`shell32.dll` in this case) itelf, the program will freeze until thread 1 is resumed (`~m` command).
+
+When thread 2 `ntdll!LdrpDrainWorkQueue` sees that `ntdll!LdrpWorkInProgress` is already `1` and that there is no work left, it will try waiting on `ntdll!LdrpLoadCompleteEvent`. This will transition `ntdll!LdrpLoadCompleteEvent` to a waiting state and `ntdll!LdrpDrainQueue` will loop around in its inner while loop, if there is still no more work left then `ntdll!LdrpDrainWorkQueue` will end up waiting on `ntdll!LdrpLoadCompleteEvent`.
+
+The behavior of `ntdll!LdrpDrainWorkQueue` in regard to its waiting on `ntdll!LdrpLoadCompleteEvent` and when the loader sets `ntdll!LdrpLoadCompleteEvent` in the `ntdll!LdrpDropLastInProgressCount` function means module initialization cannot happen concurrently with module mapping and snapping. Although it would be desirable for performance and doable for these operations to be entirely decoupled, the current design of the loader does not allow for it.
+
+The `ntdll!LdrpQueueWork` function calls `ntdll!TpPostWork` (thread pool internals) to notify loader worker threads that there is work to pick up. A concurrent `LoadLibrary` never signs up for any such notifications, instead only helping out if work is immediately available for it to do, so on an ad-hoc basis. Loader worker threads access the `ntdll!LdrpWorkQueue` in the `ntdll!LdrpWorkCallback` function to get a work item (under the protection of `ntdll!LdrpWorkQueueLock`), then call `ntdll!LdrpProcessWork` on that work item to start processing it.
+
+**Result:** Yes, a concurrent `LoadLibrary` will pitch in to help with library mapping and snapping work, just as a loader worker thread would, if work is immediately available for it to do.

code/windows/loadlibrary-concurrent-work/build.bat

@@ -3,7 +3,6 @@
 rem These options replicate Visual Studio
 rem /MD: Use UCRT instead of statically linking CRT into modules
 rem /INCREMENTAL:NO: Remove "ILT" from symbol names
-cl dll-test-2.c /DUNICODE /D_UNICODE /MD /LD /Zi /DEBUG /link /INCREMENTAL:NO
 cl dll-test.c /DUNICODE /D_UNICODE /MD /LD /Zi /DEBUG /link /INCREMENTAL:NO
 cl exe-test.c /DUNICODE /D_UNICODE /MD /Zi /DEBUG /link /INCREMENTAL:NO
 

code/windows/loadlibrary-concurrent-work/exe-test.c

@@ -7,7 +7,8 @@ DWORD WINAPI loadlibrary_thread_1(LPVOID thread_started) {
     SetEvent(thread_started);
     WaitForSingleObject(start_library_loads, INFINITE);
 
-    LoadLibrary(L"dll-test.dll");
+    // Some library with lots of dependencies
+    LoadLibrary(L"shell32.dll");
 
     return 0;
 }
@@ -16,7 +17,8 @@ DWORD WINAPI loadlibrary_thread_2(LPVOID thread_started) {
     SetEvent(thread_started);
     WaitForSingleObject(start_library_loads, INFINITE);
 
-    LoadLibrary(L"dll-test-2.dll");
+    Sleep(3000);
+    LoadLibrary(L"dll-test.dll");
 
     return 0;
 }
@@ -25,33 +27,44 @@ DWORD WINAPI loadlibrary_thread_2(LPVOID thread_started) {
 
 int main() {
     // Create event for starting library loads
-    HANDLE start_library_loads = CreateEvent(NULL, FALSE, FALSE, NULL);
+    start_library_loads = CreateEvent(NULL, TRUE, FALSE, NULL);
     if (!start_library_loads) {
         printf("Failed to create event: %lu\n", GetLastError());
-        return 1;
+        return EXIT_FAILURE;
     }
 
-    // Create event for signalling when the threads have started
+    // Create an event for signalling when each thread has started
     HANDLE thread_started_events[NUM_THREADS];
     for (int i = 0; i < NUM_THREADS; ++i) {
-        thread_started_events[i] = CreateEvent(NULL, FALSE, FALSE, NULL);
+        thread_started_events[i] = CreateEvent(NULL, TRUE, FALSE, NULL);
         if (!thread_started_events[i]) {
             printf("Failed to create event: %lu\n", GetLastError());
-            return 1;
+            return EXIT_FAILURE;
         }
     }
 
     // Create threads
-    HANDLE loadlibrary_thread_1_handle = CreateThread(NULL, 0, loadlibrary_thread_1, thread_started_events[0], 0, NULL);
-    HANDLE loadlibrary_thread_2_handle = CreateThread(NULL, 0, loadlibrary_thread_2, thread_started_events[1], 0, NULL);
+    HANDLE threads[NUM_THREADS];
+    PVOID routines[NUM_THREADS] = { loadlibrary_thread_1, loadlibrary_thread_2 };
+    for (int i = 0; i < NUM_THREADS; ++i) {
+        threads[i] = CreateThread(NULL, 0, routines[i], thread_started_events[i], 0, NULL);
+        if (!threads[i]) {
+            printf("Failed to create thread: %lu\n", GetLastError());
+            return EXIT_FAILURE;
+        }
+    }
 
     // Wait for all threads to start
     DWORD result = WaitForMultipleObjects(NUM_THREADS, thread_started_events, TRUE, INFINITE);
 
-    // Set any breakpoints to experiment here
+    // Run any debugger commands for experimenting here
     __debugbreak();
 
     // Let the LoadLibrary threads run loose!
     if (result == WAIT_OBJECT_0)
         SetEvent(start_library_loads);
+
+    // Join threads before application exits
+    result = WaitForMultipleObjects(NUM_THREADS, threads, TRUE, INFINITE);
 }
+

code/windows/loadlibrary-parallel-loader/README.md

@@ -1,5 +0,0 @@
-# `LoadLibrary` Parallel Loader Experiment
-
-Running two library loads concurrently to study the loader's ability for work parallelization. In particular, this experiment seeks to find out whether one `LoadLibrary` can perform module initialization at the same time as a concurrent `LoadLibrary` performs module mapping and snapping.
-
-!!! WORK IN PROGRESS !!!

code/windows/loadlibrary-parallel-loader/dll-test-2.c

@@ -1 +0,0 @@
-// Blank