Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Azure DevOps] Windows-2016 environment removal postponed until July 31, 2022 #5403

Closed
1 of 8 tasks
miketimofeev opened this issue Apr 14, 2022 · 45 comments
Closed
1 of 8 tasks

Comments

@miketimofeev
Copy link
Contributor

miketimofeev commented Apr 14, 2022

Breaking changes

Windows 2016 hosted runners were scheduled for full removal on April 1, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until July 31, 2022.

To raise awareness of the removal, we will have rolling 24-hour periods where the image will be unavailable for customers. During these "brownout" periods, customers will see an error message indicating that the image is slated for removal on July 31, 2022. The schedule for those brownout periods is as follows for Azure DevOps:

  • Monday April 18, 00:00 UTC to April 19, 00:00 UTC
  • Tuesday April 26, 00:00 UTC to April 27, 00:00 UTC
  • Wednesday May 4, 00:00 UTC to May 5, 00:00 UTC
  • Thursday May 12, 00:00 UTC to May 13, 00:00 UTC
  • Friday May 20, 00:00 UTC to May 21, 00:00 UTC
  • Monday May 23, 00:00 UTC to May 24, 00:00 UTC
  • Tuesday May 31, 00:00 UTC to June 1, 00:00 UTC
  • Wednesday June 8, 00:00 UTC to June 9, 00:00 UTC
  • Thursday June 16, 00:00 UTC to June 17, 00:00 UTC
  • Friday June 24, 00:00 UTC to June 25, 00:00 UTC
  • Monday June 27, 00:00 UTC to June 28, 00:00 UTC
  • Friday July 22, 00:00 UTC to July 23, 00:00 UTC
  • Wednesday July 27, 00:00 UTC to July 28, 00:00 UTC
  • Full removal - July 31, 15:00 UTC

See the original announcement at #4312

Target date

July 31, 2022

The motivation for the changes

Windows Server 2016 Active support ends on 11 Jan 2022 and Windows Server 2022 VM image is going out of beta later this year.
As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure, the Windows 2016 virtual environment will be removed from GitHub Actions and Azure DevOps.

Possible impact

Workflows targeting the Windows 2016 image will fail.

Virtual environments affected

  • Ubuntu 18.04
  • Ubuntu 20.04
  • macOS 10.15
  • macOS 11
  • macOS 12
  • Windows Server 2016
  • Windows Server 2019
  • Windows Server 2022

Mitigation ways

Change your workflows to use windows-latest, windows-2022, or windows-2019

@kningyi
Copy link

kningyi commented Apr 20, 2022

Noob here, our yml config is already using vmImage: 'windows-latest' but we're still seeing the error. What should we do?

image
image

@miketimofeev
Copy link
Contributor Author

miketimofeev commented Apr 20, 2022

@kningyi please remove everything below the

pool:
    vmImage: 'windows-latest'

(name and demands) and you should be good to go

@gene-keys
Copy link

IMPORTANT NOTE FOR THOSE OF YOU STILL STRUGGLING WITH WIN2016 BROWNOUT NOTICE.

CHECK EVERYTHING!

NOT JUST YOUR PIPELINE CODE, CHECK PIPELINE TRIGGERS POOLS. CHECK RELEASES POOLS, CHECK ALL THE POOLS EVERYWHERE ON YOUR DEVOPS ACCOUNT. WHAT WILL HAPPEN IS YOU WILL EVENTUALLY FIND SOMETHING STILL SET TO USE WIN2016 POOL. ONCE YOU FIND ALL OF THEM YOUR ACCOUNT WILL NO LONGER GET THE BROWNOUT NOTICE EVEN DURING THE SCHEDULED TIMES. SO IF YOU ARE GETTING THE NOTICE ITS NOT YOUR WIN2017 POOL, OR WIN-LATEST IN THE YML, ITS SOMETHING ELSE,

AZURE NOTICES ARE DOING A TERRIBLE JOB ON THIS WITH THEIR WORDING AND WHERE THE NOTICE APPEARS. THE NOTICE MEANS YOUR AZURE DEVOPS ACCOUNT IS STILL USING WINDWOS POOL SOMEWHERE, YOU DID NOT CHECK EVEYWHERE.

@Aeroverra
Copy link

Aeroverra commented Apr 26, 2022

I have a release pipeline (the one in the Releases tab) that gets triggered after a pipeline from the Pipelines Tab that I use for deployment to azure. This deployment keeps failing because of the 2016 error.

In the normal build pipeline I was able to set the vmImage but I don't see any way to access the yaml or change the image in the release pipeline edit page. How can this be done? I have tried searching this up plenty of times with no luck.

Edit: lol I can't believe that alluded me for so long. In case anyone else is looking. When you edit the release pipeline its under the tasks tab and then the category "run on agent". Looks like 2019 was recently made the default finally.

majguo added a commit to Azure-Samples/liberty-aad-oidc that referenced this issue Apr 27, 2022
upgrade to `windows-latest` per instructions from actions/runner-images#5403
majguo added a commit to Azure-Samples/open-liberty-on-aks that referenced this issue Apr 27, 2022
upgrade to `windows-latest` per instructions from actions/runner-images#5403
@Ziya137200
Copy link

Noob here, our yml config is already using vmImage: 'windows-latest' but we're still seeing the error. What should we do?

image image

@slonopotamus
Copy link

As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure

Isn't Windows Server 2016 going to get security updates until 2027 and this "motivation" is simply misleading?

@eli-schwartz
Copy link

As discussed in #5238

During these "brownout" periods, customers will see an error message indicating that the image is slated for removal on June 30, 2022.

The brownout error message still says April 1 and points to an old ticket with out-of-date information.

@prajwalar97
Copy link

We are having a pipeline which runs on 2016 Agent Specification. After changing the Agent Specification to 2019 or windows Latest we are getting below Error
2022-04-21T15:06:27.3818037Z Project file contains ToolsVersion="15.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="4.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=291333.
error CS1519: Invalid token '(' in class, struct, or interface member declaration
error CS1519: Invalid token '.' in class, struct, or interface member declaration

@robertoandrade
Copy link

As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure

Isn't Windows Server 2016 going to get security updates until 2027 and this "motivation" is simply misleading?

I agree 100% with this. According to Microsoft's own documentation, Windows Server 2016 will continue to get security updates until Jan 12, 2027 when it will finally stop receiving it which would justify for Microsoft to remove support for it on pipelines or start these brownouts.

@geekzter
Copy link

geekzter commented May 9, 2022

Noob here, our yml config is already using vmImage: 'windows-latest' but we're still seeing the error. What should we do?
image image

@Ziya137200 @kningyi please remove the line (26) with the reference to pool 'Hosted VS2017'.

ferwe added a commit to ferwe/PSModuleDevelopment that referenced this issue May 9, 2022
As the windows 2016 hosted runners is deprecated, the best practice is to replace it with windows-latest.
Any new PSMDProject shouldn't use the 2016 host anymore.

See GitHub Issue:
actions/runner-images#5403

Windows 2016 hosted runners were scheduled for full removal on April 1, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until June 30, 2022.
@afelthe
Copy link

afelthe commented May 9, 2022

I keep getting this error in my release pipelines and cannot figure out where to go to fix it. I am not using YAML but CLI.

@mohitook
Copy link

I get this message the whole day. Could you move your maintenance window out of work-hours next time? We'd appreciate it.
image

Additionally our systems are not using any windows-2016 related element, so the error itself makes no sense...

@miketimofeev
Copy link
Contributor Author

miketimofeev commented May 12, 2022

@mohitook if the systems don't use then you won't see the message. Maybe some of your jobs are using windows-2016 as it was the default image some time ago?

ymzayek pushed a commit to ymzayek/nilearn that referenced this issue May 13, 2022
bthirion pushed a commit to nilearn/nilearn that referenced this issue May 16, 2022
@carvido1
Copy link

Hello.
Is this affecting also self hosted runners ? It is not clear to me

@miketimofeev
Copy link
Contributor Author

@rushfrisby what error do you have? Windows-2019, for example, has a reduced set of installed .Net frameworks:
4.7.2 & 4.8 vs 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 on windows-2016

@rushfrisby
Copy link

@miketimofeev
##[error]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(1221,5): Error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.1 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks

@al-cheb
Copy link
Contributor

al-cheb commented Jul 6, 2022

@rushfrisby, Could you bump .Net version to 4.6.2?

@JMA-Dv
Copy link

JMA-Dv commented Jul 6, 2022

@rushfrisby I ran against this issue.

  1. Try to point your project to .NET v4.8.
  2. Re-publish now with the v4.8 project. (if it doesn't fix it, go to step 3).
  3. In your .yaml pipeline check for the pool tag. and change the vmImage: 'windows-2016' to 'windows-2019' or 'windows-latest'
  4. Re-run your pipeline.

Those were my steps to correct the issue.

@DeeDeeG
Copy link

DeeDeeG commented Jul 7, 2022

EDIT: Okay, this comment can be disregarded. The problem was resolved for us. We resolved this by updating one of our native C/C++ dependencies. It builds and passes CI running on the new Windows image.

Original comment

We have a project that started experiencing CI errors in the new image, even when we installed older Visual Studio Build Tools with Chocolatey. So newer Visual Studio isn't the culprit. atom-community/atom#428

Note: Building on the new image caused the problem. We have been testing on the windows-2019 image for over a year. If we take the app built on the newer image (windows-2019) and test it on vs2017-win2016, it does not cause the tests to pass. I can only conclude building on the newer image is the problem.

Does anyone have any idea what else is different about the new CI image environment that would have caused this? windows-2019 vs vs2017-win2016? (Other than Visual Studio being newer?) Any hints of where to zero in our troubleshooting? Thank you.

Right now it's looking like a subtle difference of behavior in some native C++ addon code we build for a Node module or two. (Async code that never returns, if I understand correctly.)

@miketimofeev miketimofeev unpinned this issue Jul 8, 2022
@miketimofeev miketimofeev changed the title [Azure DevOps] Windows-2016 environment removal postponed until June 30, 2022 [Azure DevOps] Windows-2016 environment removal postponed until July 22, 2022 Jul 13, 2022
@miketimofeev
Copy link
Contributor Author

The new retirement date is set — July, 22

@miketimofeev miketimofeev changed the title [Azure DevOps] Windows-2016 environment removal postponed until July 22, 2022 [Azure DevOps] Windows-2016 environment removal postponed until July 31, 2022 Jul 20, 2022
@miketimofeev
Copy link
Contributor Author

The retirement date is shifted to July, 31

@MelchiorBrandtCorstiusAtABNAMRO

LOL, can you guys please, please, please stop shifting the retirement date :) every time we need to tell our developers to stop using Win2016 as Agent type, and each time you guys postpone the deadline slowly people start using it again (I know their own fault), and we need to start warning them again, but we (and also Microsoft) are loosing our credibility on when they are actually decommissioned.

@miketimofeev
Copy link
Contributor Author

@melchiorbc sorry for the inconvenience, hopefully, this is the last one 🤞

@miketimofeev
Copy link
Contributor Author

Two more brownout dates are set:

  • Friday July 22, 00:00 UTC to July 23, 00:00 UTC
  • Wednesday July 27, 00:00 UTC to July 28, 00:00 UTC

@miketimofeev
Copy link
Contributor Author

Windows 2016 environment is finally deprecated in ADO.

artivis added a commit to artivis/manif that referenced this issue Aug 2, 2022
eli-schwartz added a commit to eli-schwartz/meson that referenced this issue Aug 2, 2022
…is deleted

The Windows 2016 images have been deprecated for a while now and
regularly browning out. There's no viable replacement for testing that
we can generate a usable `--backend=vs2017` project, so we cannot
migrate this to anything else.

The deprecated images are finally fully removed. See
actions/runner-images#5403

And now we get universal red CI, we cannot just wait for the brownout to
end to get a tiny little bit of final testing.

Simply remove the jobs so that we can tell if all the CI that actually
runs, is passing.
jpakkane pushed a commit to mesonbuild/meson that referenced this issue Aug 2, 2022
…is deleted

The Windows 2016 images have been deprecated for a while now and
regularly browning out. There's no viable replacement for testing that
we can generate a usable `--backend=vs2017` project, so we cannot
migrate this to anything else.

The deprecated images are finally fully removed. See
actions/runner-images#5403

And now we get universal red CI, we cannot just wait for the brownout to
end to get a tiny little bit of final testing.

Simply remove the jobs so that we can tell if all the CI that actually
runs, is passing.
nirbheek pushed a commit to mesonbuild/meson that referenced this issue Aug 8, 2022
…is deleted

The Windows 2016 images have been deprecated for a while now and
regularly browning out. There's no viable replacement for testing that
we can generate a usable `--backend=vs2017` project, so we cannot
migrate this to anything else.

The deprecated images are finally fully removed. See
actions/runner-images#5403

And now we get universal red CI, we cannot just wait for the brownout to
end to get a tiny little bit of final testing.

Simply remove the jobs so that we can tell if all the CI that actually
runs, is passing.
ax3l added a commit to openPMD/openPMD-api that referenced this issue Aug 9, 2022
```
[error]The Windows 2016 environment is deprecated, consider switching to windows-2019 or windows-2022 (windows-latest). For more details, see actions/runner-images#5403
[error]The remote provider was unable to process the request.
```

Also increase a macOS timeout.
@ddeveza
Copy link

ddeveza commented Aug 10, 2022

image
I already updated the agent spec but still encountering the error of 2016 deprecation.
image

@Bondarenko1990
Copy link

Windows 2016 hosted runners were scheduled for full removal on April 1, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until July 31, 2022.

same error. I use windows-2019

@MelchiorBrandtCorstiusAtABNAMRO

Windows 2016 environment is finally deprecated in ADO.

Hi Mike, are you sure of this? Up to today we still see jobs using the "vs2017-win2016" Microsoft Hosted Agent image.

@miketimofeev
Copy link
Contributor Author

@melchiorbc could you share a few links, please? I don't need access to pipelines, will take a look at telemetry by those urls

@MelchiorBrandtCorstiusAtABNAMRO

@miketimofeev I checked and I do see now the jobs trying to use 'vs2017-win2016' do get cancelled by Microsoft stating this is a deprecated environment, only people are still able to pick the 'vs2017-win2016' in their pipelines from the dropdown with VM types (which is confusing).
image

@miketimofeev
Copy link
Contributor Author

@melchiorbc this is fixed now, sorry for the inconvenience

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests