Skip to content

Commit

Permalink
Move portable RID graph into runtime and clean-up (dotnet#92211)
Browse files Browse the repository at this point in the history
* Move portable RID graph into runtime and clean-up

1. Move portable RID graph into runtime
2. Allow updates to both the non-portable and portable RID graphs under
   source build.
3. Clean-up project and remove hacks

* Update README and delete test

* Fix RID graph update when the key already exists

* Update src/libraries/Microsoft.NETCore.Platforms/readme.md

Co-authored-by: Jan Kotas <jkotas@microsoft.com>

* Update src/libraries/Microsoft.NETCore.Platforms/readme.md

Co-authored-by: Andy Gocke <angocke@microsoft.com>

---------

Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: Andy Gocke <angocke@microsoft.com>
  • Loading branch information
3 people committed Sep 19, 2023
1 parent 1185d19 commit 3b9b4fd
Show file tree
Hide file tree
Showing 29 changed files with 581 additions and 14,556 deletions.
7 changes: 7 additions & 0 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -471,4 +471,11 @@
<CustomBeforeNoTargets>$(RepositoryEngineeringDir)NoTargetsSdk.BeforeTargets.targets</CustomBeforeNoTargets>
<CustomAfterTraversalTargets>$(RepositoryEngineeringDir)TraversalSdk.AfterTargets.targets</CustomAfterTraversalTargets>
</PropertyGroup>

<PropertyGroup>
<!-- Keep in sync with outputs defined in Microsoft.NETCore.Platforms.csproj. -->
<BundledRuntimeIdentifierGraphFile>$([MSBuild]::NormalizePath('$(ArtifactsBinDir)', 'Microsoft.NETCore.Platforms', 'runtime.json'))</BundledRuntimeIdentifierGraphFile>
<BundledRuntimeIdentifierGraphFile Condition="!Exists('$(BundledRuntimeIdentifierGraphFile)')">$([MSBuild]::NormalizePath('$(LibrariesProjectRoot)', 'Microsoft.NETCore.Platforms', 'src', 'runtime.json'))</BundledRuntimeIdentifierGraphFile>
</PropertyGroup>

</Project>
1 change: 1 addition & 0 deletions Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -217,4 +217,5 @@
<InstallersRelativePath>Runtime/$(SharedFrameworkNugetVersion)/</InstallersRelativePath>
</PropertyGroup>
</Target>

</Project>
14 changes: 0 additions & 14 deletions eng/AvoidRestoreCycleOnSelfReference.targets

This file was deleted.

1 change: 0 additions & 1 deletion src/libraries/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@
<Import Project="$(RepositoryEngineeringDir)testing\coverage.targets" Condition="'$(EnableRunSettingsSupport)' == 'true' or '$(EnableCoverageSupport)' == 'true'" />

<Import Project="$(RepositoryEngineeringDir)illink.targets" Condition="'$(IsSourceProject)' == 'true' or '$(ExplicitlyImportCustomILLinkTargets)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)AvoidRestoreCycleOnSelfReference.targets" Condition="'$(AvoidRestoreCycleOnSelfReference)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)nativeSanitizers.targets" />

<ItemGroup Condition="'$(UseTargetFrameworkPackage)' != 'false'">
Expand Down
Original file line number Diff line number Diff line change
@@ -1,102 +1,24 @@
Microsoft Visual Studio Solution File, Format Version 12.00
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TestUtilities", "..\Common\tests\TestUtilities\TestUtilities.csproj", "{39BCA125-321F-490F-AD4E-28DCB4406969}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.NETCore.Platforms", "src\Microsoft.NETCore.Platforms.csproj", "{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.NETCore.Platforms.Tests", "tests\Microsoft.NETCore.Platforms.Tests.csproj", "{0C60F372-5C73-4BFA-9B91-5659C88F9750}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LibraryImportGenerator", "..\System.Runtime.InteropServices\gen\LibraryImportGenerator\LibraryImportGenerator.csproj", "{45449066-3A31-43E5-B705-20D667080A23}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Interop.SourceGeneration", "..\System.Runtime.InteropServices\gen\Microsoft.Interop.SourceGeneration\Microsoft.Interop.SourceGeneration.csproj", "{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ILLink.CodeFixProvider", "..\..\tools\illink\src\ILLink.CodeFix\ILLink.CodeFixProvider.csproj", "{BF52D0B1-1AD8-49DB-B910-9F9D7D41292A}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ILLink.RoslynAnalyzer", "..\..\tools\illink\src\ILLink.RoslynAnalyzer\ILLink.RoslynAnalyzer.csproj", "{BEFED0BD-AD56-4FAC-B9A7-CCE8BB5F8768}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ILLink.Tasks", "..\..\tools\illink\src\ILLink.Tasks\ILLink.Tasks.csproj", "{41387647-9500-448C-B021-B4837947EF62}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Mono.Linker", "..\..\tools\illink\src\linker\Mono.Linker.csproj", "{281E9ED5-5F16-475D-B50B-501633FCC8C6}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Mono.Linker", "..\..\tools\illink\src\linker\ref\Mono.Linker.csproj", "{20346A2D-06DD-4E3E-96D4-C9BF32697AE3}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{E4827496-6F39-4CA0-8F4A-ACDE9DFEBE5C}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{F96FBD24-3BB3-4D02-9884-4D90F94DD3C0}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "gen", "gen", "{E8010E1D-FDAF-481D-AA34-3B115B667E4B}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "gen", "tools\gen", "{2B7F6369-F9CA-493A-A105-B7B7B697A31F}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "tools\src", "{A527D4DF-4D57-40AE-B63B-1945F1D4FF3F}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "ref", "tools\ref", "{EB2B9277-DB89-4554-8499-C3240163E875}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tools", "tools", "{0F4ABB64-2357-49FB-9BD3-8FEFC2A650E6}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{39BCA125-321F-490F-AD4E-28DCB4406969}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{39BCA125-321F-490F-AD4E-28DCB4406969}.Debug|Any CPU.Build.0 = Debug|Any CPU
{39BCA125-321F-490F-AD4E-28DCB4406969}.Release|Any CPU.ActiveCfg = Release|Any CPU
{39BCA125-321F-490F-AD4E-28DCB4406969}.Release|Any CPU.Build.0 = Release|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Debug|Any CPU.Build.0 = Debug|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Release|Any CPU.ActiveCfg = Release|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Release|Any CPU.Build.0 = Release|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Release|Any CPU.ActiveCfg = Release|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Release|Any CPU.Build.0 = Release|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Debug|Any CPU.Build.0 = Debug|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Release|Any CPU.ActiveCfg = Release|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Release|Any CPU.Build.0 = Release|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Release|Any CPU.Build.0 = Release|Any CPU
{BF52D0B1-1AD8-49DB-B910-9F9D7D41292A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{BF52D0B1-1AD8-49DB-B910-9F9D7D41292A}.Debug|Any CPU.Build.0 = Debug|Any CPU
{BF52D0B1-1AD8-49DB-B910-9F9D7D41292A}.Release|Any CPU.ActiveCfg = Release|Any CPU
{BF52D0B1-1AD8-49DB-B910-9F9D7D41292A}.Release|Any CPU.Build.0 = Release|Any CPU
{BEFED0BD-AD56-4FAC-B9A7-CCE8BB5F8768}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{BEFED0BD-AD56-4FAC-B9A7-CCE8BB5F8768}.Debug|Any CPU.Build.0 = Debug|Any CPU
{BEFED0BD-AD56-4FAC-B9A7-CCE8BB5F8768}.Release|Any CPU.ActiveCfg = Release|Any CPU
{BEFED0BD-AD56-4FAC-B9A7-CCE8BB5F8768}.Release|Any CPU.Build.0 = Release|Any CPU
{41387647-9500-448C-B021-B4837947EF62}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{41387647-9500-448C-B021-B4837947EF62}.Debug|Any CPU.Build.0 = Debug|Any CPU
{41387647-9500-448C-B021-B4837947EF62}.Release|Any CPU.ActiveCfg = Release|Any CPU
{41387647-9500-448C-B021-B4837947EF62}.Release|Any CPU.Build.0 = Release|Any CPU
{281E9ED5-5F16-475D-B50B-501633FCC8C6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{281E9ED5-5F16-475D-B50B-501633FCC8C6}.Debug|Any CPU.Build.0 = Debug|Any CPU
{281E9ED5-5F16-475D-B50B-501633FCC8C6}.Release|Any CPU.ActiveCfg = Release|Any CPU
{281E9ED5-5F16-475D-B50B-501633FCC8C6}.Release|Any CPU.Build.0 = Release|Any CPU
{20346A2D-06DD-4E3E-96D4-C9BF32697AE3}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{20346A2D-06DD-4E3E-96D4-C9BF32697AE3}.Debug|Any CPU.Build.0 = Debug|Any CPU
{20346A2D-06DD-4E3E-96D4-C9BF32697AE3}.Release|Any CPU.ActiveCfg = Release|Any CPU
{20346A2D-06DD-4E3E-96D4-C9BF32697AE3}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{39BCA125-321F-490F-AD4E-28DCB4406969} = {E4827496-6F39-4CA0-8F4A-ACDE9DFEBE5C}
{0C60F372-5C73-4BFA-9B91-5659C88F9750} = {E4827496-6F39-4CA0-8F4A-ACDE9DFEBE5C}
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11} = {F96FBD24-3BB3-4D02-9884-4D90F94DD3C0}
{45449066-3A31-43E5-B705-20D667080A23} = {E8010E1D-FDAF-481D-AA34-3B115B667E4B}
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B} = {E8010E1D-FDAF-481D-AA34-3B115B667E4B}
{BF52D0B1-1AD8-49DB-B910-9F9D7D41292A} = {2B7F6369-F9CA-493A-A105-B7B7B697A31F}
{BEFED0BD-AD56-4FAC-B9A7-CCE8BB5F8768} = {2B7F6369-F9CA-493A-A105-B7B7B697A31F}
{2B7F6369-F9CA-493A-A105-B7B7B697A31F} = {0F4ABB64-2357-49FB-9BD3-8FEFC2A650E6}
{41387647-9500-448C-B021-B4837947EF62} = {A527D4DF-4D57-40AE-B63B-1945F1D4FF3F}
{281E9ED5-5F16-475D-B50B-501633FCC8C6} = {A527D4DF-4D57-40AE-B63B-1945F1D4FF3F}
{A527D4DF-4D57-40AE-B63B-1945F1D4FF3F} = {0F4ABB64-2357-49FB-9BD3-8FEFC2A650E6}
{20346A2D-06DD-4E3E-96D4-C9BF32697AE3} = {EB2B9277-DB89-4554-8499-C3240163E875}
{EB2B9277-DB89-4554-8499-C3240163E875} = {0F4ABB64-2357-49FB-9BD3-8FEFC2A650E6}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {E946A528-C3E7-48EC-AD6D-AE84ED2B11AC}
Expand Down
80 changes: 2 additions & 78 deletions src/libraries/Microsoft.NETCore.Platforms/readme.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Runtime IDs
The package `Microsoft.NETCore.Platforms` defines the runtime identifiers (RIDs) used by .NET packages to represent runtime-specific assets in NuGet packages.
The `Microsoft.NETCore.Platforms` transport package contains the portable and non-portable runtime identifier graph files for redistribution in the dotnet/sdk repository.

## What is a RID?
A RID is an opaque string that identifies a platform. RIDs have relationships to other RIDs by "importing" the other RID. In that way a RID is a directed graph of compatible RIDs.
Expand Down Expand Up @@ -76,80 +76,4 @@ runtimes/win/lib/netstandard1.0/foo.dll
When resolving for netstandard1.5/win7-x64 will select `lib/netstandard1.5/foo.dll` for the compile asset and `runtimes/win/lib/netstandard1.0/foo.dll` for the runtime asset.

## Adding new RIDs

### Why do I need to add a new RID?
NuGet's extensibility mechanism for platform-specific assets requires a RID be defined for any platform that needs assets specific to that platform. Unlike TFMs, which have a known relationship in NuGet (eg net4.5 is compatible with net4.0), RIDs are opaque strings which NuGet knows nothing about. The definition and relationship of RIDs comes solely from the `runtime.json` files within the root of the packages referenced by the project.
As such, whenever we want to put a new RID in a project.json in order to get assets specific for that RID we have to define the rid in some package. Typically that package is `Microsoft.NETCore.Platforms` if the RID is "official". If you'd like to prototype you can put the RID in any other package and so long as that package is referenced you can use that RID.

### Do I really need to add a new RID?
If you're prototyping on a platform that is compatible with an existing platform then you can reuse the RID for that existing platform. New RIDs are only needed when an asset needs to be different on a particular platform.

`Microsoft.NETCore.Platforms` attempts to define all RIDs that packages may need, and as such will define RIDs for platforms that we don't actually cross compile for. This is to support higher-level packages, 3rd party packages, that may need to cross-compile for that RID.

### Adding a new OS
Add a new `RuntimeGroup` item in `runtimeGroups.props`.

For example:
```xml
<RuntimeGroup Include="myLinuxDistro">
<Parent>linux</Parent>
<Architectures>x86;x64;arm</Architectures>
<Versions>42.0;43.0</Versions>
</RuntimeGroup>
```

This will create a new RID for `myLinuxDistro` where `myLinuxDistro` should be the string used for the `ID=` value in the `/etc/os-release` file.

Whenever modifying the `runtimeGroups.props` make sure to pack the project via the `dotnet pack` command and inspect if the generated package contains the desired changes.

RuntimeGroup items have the following format:
- `Identity`: the base string for the RID, without version architecture, or qualifiers.
- `Parent`: the base string for the parent of this RID. This RID will be imported by the baseRID, architecture-specific, and qualifier-specific RIDs (with the latter two appending appropriate architecture and qualifiers).
- `Versions`: A list of strings delimited by semi-colons that represent the versions for this RID.
- `TreatVersionsAsCompatible`: Default is true. When true, version-specific RIDs will import the previous version-specific RID in the Versions list, with the first version importing the version-less RID. When false all version-specific RIDs will import the version-less RID (bypassing previous version-specific RIDs)
- `OmitVersionDelimiter`: Default is false. When true no characters will separate the base RID and version (EG: win7). When false a '.' will separate the base RID and version (EG: osx.10.12).
- `ApplyVersionsToParent`: Default is false. When true, version-specific RIDs will import version-specific Parent RIDs similar to is done for architecture and qualifier (see Parent above).
- `Architectures`: A list of strings delimited by semi-colons that represent the architectures for this RID.
- `AdditionalQualifiers`: A list of strings delimited by semi-colons that represent the additional qualifiers for this RID. Additional qualifers do not stack, each only applies to the qualifier-less RIDs (so as not to cause combinatorial exponential growth of RIDs).

### Adding a new version to an existing OS
Find the existing `RuntimeGroup` in `runtimeGroups.props` and add the version to the list of `Versions`, separated by a semi-colon.

If the version you are adding needs to be treated as not-compatible with previous versions and the `RuntimeGroup` has not set `TreatVersionsAsCompatible`=`false` then you may create a new `RuntimeGroup` to represent the new compatibility band.

### Checking your work
After making a change to `runtimeGroups.props` you can examine the resulting changes in `runtime.json` and `runtime.compatibility.json`.

`runtime.json` is the graph representation of the RIDs and is what ships in the package.

`runtime.compatibility.json` is a flattened version of the graph that shows the RID precedence for each RID in the graph.

### Version compatibility
Version compatibility is represented through imports. If a platform is considered compatible with another version of the same platform, or a specific version of another platform, then it can import that platform. This permits packages to reuse assets that were built for the imported platform on the compatible platform. Compatibility here is a bit nebulous because inevitably different platforms will have observable differences that can cause compatibility problems. For the purposes of RIDs we'll try to represent compatibility as versions of a platform that are explicitly advertised as being compatible with a previous version and/or another platform and don't have any known broad breaking changes. It is usually better to opt to treat platforms as compatible since that enables the scenario of building an asset for a particular version and using that in future versions, otherwise you force people to cross-compile for all future versions the moment they target a specific version.

## Appendix : details of RID graph generation

### Naming convention
We use the following convention in all newly-defined RIDs. Some RIDs (win7-x64, win8-x64) predate this convention and don't follow it, but all new RIDs should follow it.
`[os name].[version]-[architecture]-[additional qualifiers]`, for example `osx.10.10-x64` or `ubuntu.14.04-x64`
- `[os name]` can contain any characters other than `.`
- `[version]` can contain any characters other than `-`. Typically a numeric version like 14.04 or 10.0.
- `[architecture]` can contain any characters other than `-`. Typically: `x86`, `x64`, `arm`, `arm64`
- `[additional qualifiers]` can be things like `aot`. Used to further differentiate different platforms.

For all of these we strive to make them something that can be uniquely discoverable at runtime, so that a RID may be computed from an executing application. As such these properties should be derivable from `/etc/os-release` or similar platform APIs / data.

### Import convention
Imports should be used when the added RID is considered compatible with an existing RID.

1. Architecture-specific RIDs should first import the architecture-less RID. EG: `osx.10.11-x64` should first import `osx.10.11`.
2. Architecture-specific RIDs that are compatible with a previous version RID for the same OS should then import the previous version, architecture specific RID. EG: `osx.10.11-x64` should then import `osx.10.10-x64`. If there is no earlier compatible/supported version, then a versionless RID should be imported. EG: `osx.10.10-x64` should import `osx-x64`.
3. Architecture-less RIDs that are compatible with a previous version RID for the same OS should then import the previous version, architecture neutral RID. EG: `osx.10.11` should import `osx.10.10`. If there is no earlier compatible/supported version, then a versionless RID should be imported. EG: `osx.10.10` should import `osx`.
4. Version-less RIDs should import an OS category. EG: `osx-x64` should import `unix-x64`, `osx` should import `unix`.

### Advanced RuntimeGroup metadata
The following options can be used under special circumstances but break the normal precedence rules we try to establish by generating the RID graph from common logic. These options make it possible to create a RID fallback chain that doesn't match the rest of the RIDs and therefore is hard for developers/package authors to reason about. Only use these options for cases where you know what you are doing and have carefully reviewed the resulting RID fallbacks using the CompatibliltyMap.

- `OmitRIDs`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will not be referenced nor defined.
- `OmitRIDDefinitions`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will not be defined by this RuntimeGroup, but will be referenced: useful in case some other RuntimeGroup (or runtime.json template) defines them.
- `OmitRIDReferences`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will be defined but not referenced by this RuntimeGroup.
The RID graphs should be only updated with new base OSes, architectures, or C standard libraries. The RID graphs shouldn't be updated with new OS flavor- and version-specific RIDs anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID.
Loading

0 comments on commit 3b9b4fd

Please sign in to comment.