Use versions repo tooling from BuildTools

[dagood]

This changes the auto-upgrades to use a single PR for all dependency upgrades (per branch), updating an already-opened PR when possible. (For more info see dotnet/corefx#10602.)

Here's an example upgrade: dagood#2.

This should let us enable auto-upgrade without dealing with PR spam.

/cc @wtgodbe @gkhanna79

---

Edit: This also adds full-version package dependency verification, so there are no more regexes. Check dotnet/corefx#10868 for some details on that. I've let the tool update dagood#2 to show how it changes the new file.

[gkhanna79]

Would this need to be updated when the change is moved to release branch in future? If so, can we set it only if not already set, enabling it to be set via build system too?

[dagood]

It would, but I don't see the value in setting it via the build system:

1. The plan is to use this for dependency validation, so everyone working on the branch needs the same value. (And I don't think there's a reliable way for dev builds to determine "current branch" without checking something into source.)
2. Even before that goes into action, currently the value is only used to upgrade dependencies, and it isn't used during builds.
3. The logic below is simply $(BaseDotNetBuildInfoUrl)[...]/$(DependencyBranch) in this PR, but it will be customized for different branches. E.g. dev/api would still depend on master for External, but dev/api for corefx and coreclr. (On the corefx side, once coreclr starts building dev/api, the corefx dev/api branch needs to start using that as its dependency. (If that's actually planned.))

I can't think of an alternative that doesn't require some property checked into source that is different based on branch.

[wtgodbe]

Awesome! Are there other places that invoke UpdateDependencies.ps1 that we'd need to modify to account for removing it?

[dagood]

The only place that should be using it is the Maestro subscription, which I'll change like I did for corefx in dotnet/versions#48. If I remember correctly I may have to be more creative with how I invoke it in coreclr because of the different run commands. I'll make sure I have a workable PR up on the versions repo before merging this into coreclr.

[dagood]

I added changes from dotnet/corefx#10868 so this uses full-version verification now, as well as full-version upgrade. I'm working on a decent command to add to the versions repo subscription.

@gkhanna79 Could you take a look at my response on the thread about setting dependency branch in the build system? In short, I don't see a reason to set it with a build parameter.

[gkhanna79]

@dagood Sounds good to me.

[dagood]

@dotnet-bot test this please

[dagood]

I'd thought those CI errors were because of a buildtools upgrade, but merging to a point where someone else has already upgraded, the errors are still happening. They look like this:

14:38:24 Tools are already initialized.
14:38:24 Running: D:\j\workspace\x86_lb_checke---9877589b\Tools\dotnetcli\dotnet.exe D:\j\workspace\x86_lb_checke---9877589b\Tools\run.exe build -Project=D:\j\workspace\x86_lb_checke---9877589b\bin\obj\Windows_NT.x86.Checked\install.vcxproj -MsBuildLog=/flp:Verbosity=normal;LogFile="D:\j\workspace\x86_lb_checke---9877589b\bin\Logs\CoreCLR_Windows_NT__x86__Checked.log" -MsBuildWrn=/flp1:WarningsOnly;LogFile="D:\j\workspace\x86_lb_checke---9877589b\bin\Logs\CoreCLR_Windows_NT__x86__Checked.wrn" -MsBuildErr=/flp2:ErrorsOnly;LogFile="D:\j\workspace\x86_lb_checke---9877589b\bin\Logs\CoreCLR_Windows_NT__x86__Checked.err" -configuration=Checked  -BuildOS=Windows_NT -BuildType=Checked -BuildArch=x86
14:38:24 Running: D:\j\workspace\x86_lb_checke---9877589b\Tools\msbuild.cmd   /nologo /verbosity:minimal /clp:Summary  D:\j\workspace\x86_lb_checke---9877589b\bin\obj\Windows_NT.x86.Checked\install.vcxproj /p:__BuildType=Checked /p:__BuildArch=x86 /p:__BuildOS=Windows_NT /p:Configuration=Checked  /flp:Verbosity=normal;LogFile=D:\j\workspace\x86_lb_checke---9877589b\bin\Logs\CoreCLR_Windows_NT__x86__Checked.log  /flp1:WarningsOnly;LogFile=D:\j\workspace\x86_lb_checke---9877589b\bin\Logs\CoreCLR_Windows_NT__x86__Checked.wrn  /flp2:ErrorsOnly;LogFile=D:\j\workspace\x86_lb_checke---9877589b\bin\Logs\CoreCLR_Windows_NT__x86__Checked.err /maxcpucount
14:38:25 
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\CodeAnalysis\Microsoft.CodeAnalysis.targets(219,5): error MSB4175: The task factory "CodeTaskFactory" could not be loaded from the assembly "C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Build.Tasks.Core.dll". Could not find file 'C:\Users\dotnet-bot\AppData\Local\Temp\lxwmyqzb.dll'. [D:\j\workspace\x86_lb_checke---9877589b\bin\obj\Windows_NT.x86.Checked\ZERO_CHECK.vcxproj]

http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/master/job/x86_lb_checked_windows_nt_prtest/3172/
http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/master/job/x86_ryujit_checked_windows_nt_prtest/3216/

It looks like other PRs haven't had this issue, but it's happened twice here with the same affected builds each time. Any ideas on this, or who would know what to look for?

[wtgodbe]

'lxwmyqzb.dll' seems kinda suspect, is that a real thing?

[gkhanna79]

@mmitche Does this sound like the CodeFactory infra issue you were investigating?

[mmitche]

Yes

[gkhanna79]

Whats the workaround/solution that @dagood could use?

[mmitche]

For now, rerunning the legs:

@dotnet-bot test Windows_NT x86 legacy_backend Checked Build and Test
@dotnet-bot test Windows_NT x86 ryujit Checked Build and Test

[dagood]

Third time's the charm! Thanks, merging.