Skip to content

Commit 989ef58

Browse files
lambdageekradicallewing
authored and
mdh1418
committed
[mono][wasm] Bundle assemblies as WebCIL (dotnet#79416)
Define a new container format for .NET assemblies that looks less like a Windows PE file. Use it for bundling assemblies in wasm projects. * Implement WebCIL loader It will try to look for WebCIL formatted images instread of normal .dll files * Checkpoint works on wasm sample; add design doc * Push .dll->.webcil probing lower in the bundle logic * Also convert satellite assemblies and implement satellite matching * [wasm] don't leak .webcil image names to the debugger In particular this will make source and breakpoint URLs look like `dotnet://foo.dll/Foo.cs` which means that grabbing PDBs via source link will work, etc. * Add PE DebugTableDirectory to webcil This is used to retrieve the PPDB data and/or the PDB checksum from an image. Refactor mono_has_pdb_checksum to support webcil in addition to PE images * Implement a WebcilReader for BorwserDebugProxy like PEReader This needs some improvements: - add support for reading CodeView and EmbeddedPDB data - copy/paste less from the WebcilWriter task - copy/paste less from PEReader (will require moving WebcilReader to SRM) * [debug] Match bundled pdbs if we're looking up .webcil files The pdbs are registered by wasm with a notional .dll filename. if the debugger does a lookup using a .webcil name instead, allow the match * Adjust debug directory entries when writing webcil files the PE COFF debug directory entries contain a 'pointer' field which is an offset from the start of the file. When writing the webcil file, the header is typically smaller than a PE file, so the offsets are wrong. Adjust the offsets by the size of the file. We assume (and assert) the debug directory entries actually point at some PE COFF sections in the PE file (as opposed to somewhere past the end of the known PE data). When writing, we initially just copy all the sections directly, then seek to where the debug directory entries are, and overwrite them with updated entries that have the correct 'pointer' * Fix bug in WebcilWriter Stream.CopyTo takes a buffer size, not the number of bytes to copy. * bugfix: the debug directory is at pe_debug_rva not at the CLI header * skip debug fixups if there's no debug directory * WebcilReader: implement CodeView and Emebedded PPDB support * [WBT] Add UseWebcil option (default to true) * rename WebcilWriter -> WebcilConverter [NFC] * fixup AssemblyLoadedEventTest * hack: no extension on assembly for breakpoint * pass normal .dll name for MainAssemblyName in config let the runtime deal with it - bundle matching will resolve it to the .webcil file * Wasm.Debugger.Tests: give CI 10 more minutes * Add Microsoft.NET.WebAssembly.Webcil assembly project Mark it as shipping, but not shipping a nuget package. The idea is that it will be shipped along with the WasmAppBuilder msbuild task, and with the BrowserDebugProxy tool. * Move WebcilConverter to Microsoft.NET.WebAssembly.Webcil * Move WebcilReader to Microsoft.NET.WebAssembly.Webcil delete the duplicated utility classes * make the webcil magic and version longer * Code style improvements from review * Improve some exception messages, when possible * Suggestings from code review * Add WasmEnableWebcil msbuild property. Off by default * Build non-wasm runtimes without .webcil support * Run WBT twice: with and without webcil This is a total of 4 runs: with and without workloads x with and without webcil * do the cartesian product correctly in msbuild * also add webcil to template projects * environment variable has to be non-null and "true" We set it to "false" sometimes * Fix wasm work items They should be the same whether or not webcil is used. Just the WorkloadItemPrefix should be used to change the name. * Update src/libraries/sendtohelix-wasm.targets * PInvokeTableGeneratorTests: don't try to use the net472 WasmAppBuilder Look for the default target framework subdirectory under the tasks directory in the runtime pack when trying to find the tasks dll. In particular don't try to load the net472 version on modern .NET * PInvokeTableGeneratorTests: Add more diagnostic output if tasksDir is not found * simplify prefix comparison in bundled_assembly_match * WasmAppBuilder improve logging Just emit a single Normal importance message about webcil; details as Low importance. * Add missing using Co-authored-by: Ankit Jain <radical@gmail.com> Co-authored-by: Larry Ewing <lewing@microsoft.com>
1 parent 6723350 commit 989ef58

41 files changed

Lines changed: 1508 additions & 62 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/design/mono/webcil.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# WebCIL assembly format
2+
3+
## Version
4+
5+
This is version 0.0 of the Webcil format.
6+
7+
## Motivation
8+
9+
When deploying the .NET runtime to the browser using WebAssembly, we have received some reports from
10+
customers that certain users are unable to use their apps because firewalls and anti-virus software
11+
may prevent browsers from downloading or caching assemblies with a .DLL extension and PE contents.
12+
13+
This document defines a new container format for ECMA-335 assemblies
14+
that uses the `.webcil` extension and uses a new WebCIL container
15+
format.
16+
17+
18+
## Specification
19+
20+
As our starting point we take section II.25.1 "Structure of the
21+
runtime file format" from ECMA-335 6th Edition.
22+
23+
| |
24+
|--------|
25+
| PE Headers |
26+
| CLI Header |
27+
| CLI Data |
28+
| Native Image Sections |
29+
| |
30+
31+
32+
33+
A Webcil file follows a similar structure
34+
35+
36+
| |
37+
|--------|
38+
| Webcil Headers |
39+
| CLI Header |
40+
| CLI Data |
41+
| |
42+
43+
## Webcil Headers
44+
45+
The Webcil headers consist of a Webcil header followed by a sequence of section headers.
46+
(All multi-byte integers are in little endian format).
47+
48+
### Webcil Header
49+
50+
``` c
51+
struct WebcilHeader {
52+
uint8_t id[4]; // 'W' 'b' 'I' 'L'
53+
// 4 bytes
54+
uint16_t version_major; // 0
55+
uint16_t version_minor; // 0
56+
// 8 bytes
57+
uint16_t coff_sections;
58+
uint16_t reserved0; // 0
59+
// 12 bytes
60+
61+
uint32_t pe_cli_header_rva;
62+
uint32_t pe_cli_header_size;
63+
// 20 bytes
64+
65+
uint32_t pe_debug_rva;
66+
uint32_t pe_debug_size;
67+
// 28 bytes
68+
};
69+
```
70+
71+
The Webcil header starts with the magic characters 'W' 'b' 'I' 'L' followed by the version in major
72+
minor format (must be 0 and 0). Then a count of the section headers and two reserved bytes.
73+
74+
The next pairs of integers are a subset of the PE Header data directory specifying the RVA and size
75+
of the CLI header, as well as the directory entry for the PE debug directory.
76+
77+
78+
### Section header table
79+
80+
Immediately following the Webcil header is a sequence (whose length is given by `coff_sections`
81+
above) of section headers giving their virtual address and virtual size, as well as the offset in
82+
the Webcil file and the size in the file. This is a subset of the PE section header that includes
83+
enough information to correctly interpret the RVAs from the webcil header and from the .NET
84+
metadata. Other information (such as the section names) are not included.
85+
86+
``` c
87+
struct SectionHeader {
88+
uint32_t st_virtual_size;
89+
uint32_t st_virtual_address;
90+
uint32_t st_raw_data_size;
91+
uint32_t st_raw_data_ptr;
92+
};
93+
```
94+
95+
### Sections
96+
97+
Immediately following the section table are the sections. These are copied verbatim from the PE file.
98+
99+
## Rationale
100+
101+
The intention is to include only the information necessary for the runtime to locate the metadata
102+
root, and to resolve the RVA references in the metadata (for locating data declarations and method IL).
103+
104+
A goal is for the files not to be executable by .NET Framework.
105+
106+
Unlike PE files, mixing native and managed code is not a goal.
107+
108+
Lossless conversion from Webcil back to PE is not intended to be supported. The format is being
109+
documented in order to support diagnostic tooling and utilities such as decompilers, disassemblers,
110+
file identification utilities, dependency analyzers, etc.
111+

‎src/libraries/Microsoft.NET.WebAssembly.Webcil/Directory.Build.props‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<Project>
2+
<Import Project="..\Directory.Build.props" />
3+
<PropertyGroup>
4+
<IsShipping>true</IsShipping>
5+
<!-- this assembly should not produce a public package, rather it's meant to be shipped by the
6+
WasmAppBuilder task and the BrowserDebugProxy -->
7+
<IsShippingPackage>false</IsShippingPackage>
8+
<!-- This isn't a public API in a public package, don't ship documentation xml in the nugets that consume this assembly -->
9+
<GenerateDocumentationFile>false</GenerateDocumentationFile>
10+
</PropertyGroup>
11+
</Project>

‎src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Common/IsExternalInit.cs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
namespace System.Runtime.CompilerServices
5+
{
6+
internal sealed class IsExternalInit { }
7+
}

‎src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Microsoft.NET.WebAssembly.Webcil.csproj‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
<PropertyGroup>
3+
<TargetFrameworks>$(NetCoreAppToolCurrent);$(NetFrameworkToolCurrent)</TargetFrameworks>
4+
<Description>Abstractions for modifying .NET webcil binary images</Description>
5+
<IncludeSymbols>true</IncludeSymbols>
6+
<Serviceable>true</Serviceable>
7+
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
8+
<CLSCompliant>false</CLSCompliant>
9+
</PropertyGroup>
10+
11+
<ItemGroup>
12+
<!-- we need to keep the version of System.Reflection.Metadata in sync with dotnet/msbuild and dotnet/sdk -->
13+
<PackageReference Include="System.Reflection.Metadata" Version="$(SystemReflectionMetadataVersion)" />
14+
<PackageReference Include="System.Collections.Immutable" Version="$(SystemCollectionsImmutableVersion)" />
15+
</ItemGroup>
16+
17+
<ItemGroup>
18+
<Compile Include="Webcil\**\*.cs" />
19+
</ItemGroup>
20+
21+
<ItemGroup Condition="'$(TargetFrameworkIdentifier)' == '.NETFramework'">
22+
<Compile Include="Common\IsExternalInit.cs" />
23+
</ItemGroup>
24+
</Project>

‎src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/Internal/Constants.cs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
namespace Microsoft.NET.WebAssembly.Webcil.Internal;
5+
6+
internal static unsafe class Constants
7+
{
8+
public const int WC_VERSION_MAJOR = 0;
9+
public const int WC_VERSION_MINOR = 0;
10+
}

‎src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebciHeader.cs‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
using System.Runtime.InteropServices;
5+
6+
namespace Microsoft.NET.WebAssembly.Webcil;
7+
8+
/// <summary>
9+
/// The header of a WebCIL file.
10+
/// </summary>
11+
///
12+
/// <remarks>
13+
/// The header is a subset of the PE, COFF and CLI headers that are needed by the mono runtime to load managed assemblies.
14+
/// </remarks>
15+
[StructLayout(LayoutKind.Sequential, Pack = 1)]
16+
public unsafe struct WebcilHeader
17+
{
18+
public fixed byte id[4]; // 'W' 'b' 'I' 'L'
19+
// 4 bytes
20+
public ushort version_major; // 0
21+
public ushort version_minor; // 0
22+
// 8 bytes
23+
24+
public ushort coff_sections;
25+
public ushort reserved0; // 0
26+
// 12 bytes
27+
public uint pe_cli_header_rva;
28+
public uint pe_cli_header_size;
29+
// 20 bytes
30+
public uint pe_debug_rva;
31+
public uint pe_debug_size;
32+
// 28 bytes
33+
}

0 commit comments

Comments
 (0)