Use Remembered Arguments for a Package During Upgrades

[ferventcoder]

Following #358 (and related to #357), we capture the arguments passed during install/upgrade. Now we need to pass them to the upgrades automatically.

cc @SeanKilleen

Notes

This is in preview in 0.10.4 and will be turned to 'on' automatically in a future release. We are going to be continually making it better and won't turn it on by default until it is ready. If you want to turn it on and start using it, once you have 0.10.4 installed, run
choco feature enable -n useRememberedArgumentsForUpgrades.

You can also do this per command with --use-remembered-arguments. You can also turn it off per command with --ignore-remembered-arguments.

A couple of important notes:

1. Remembered arguments would be from the last version, so if you upgraded, those original install arguments did not carry up the chain unless you passed them at the command line during upgrade. There is probably a manual way to go and get those and put them in the latest version. Plainspeak: This means if you install and have upgraded since - the only arguments looked at are what you passed to the upgraded version. This is expected.
2. [Security] The .arguments file where these are stored will be completely encrypted from now on. This is a security mechanism in case there is any sensitive data being passed on the command line. This is by design.
3. This is not yet capturing arguments that only apply to the licensed editions of Chocolatey, but we expect to capture that in the future. This will be handled in a future release. useRememberedArgumentsForUpgrades feature does not remember C4B/Pro specific install arguments #1547
4. If you passed along a password for install in versions less than 0.10.4, it encrypted only that value and this will not unencrypt that (so it will be passed incorrectly). If this was a working feature in the past, this would be a breaking change. There is a workaround - you can override the value it picks up by passing the new password on the command line with the upgrade command. This is not the best thing, but since we have a workaround, we probably won't move forward with a fix for this (that is unless someone else really feels strongly about this and files an issue + provides a fix with specs).
5. This is not, and likely will never be, able to remember and pass through a custom source used for installation. Those are set very, very early in the process. This is open for discussion, but the workaround is to ensure your default sources contain everything you need.
6. Currently the saved values will override what is passed on the command line for the values it overrides. This is sub-optimal. However it is tricky. We'll post an issue that will resolve that in a future release. If you need to ignore those arguments passed through, be sure to pass --ignore-remembered-arguments to the upgrade command. This will be fixed in a future release.

Examples

Remembered Arguments lost during upgrade all examples (number 1 above)

For 1 above, say you are on 0.10.3. You install notepadplusplus --forceX86. Then you upgrade choco upgrade all and notepadplusplus gets upgraded. Then you install 0.10.4 and upgrade again choco upgrade notepadplusplus. It will have lost --forcex86 during the upgrade all so it won't be there on the upgrade with remembered arguments.

Now say you are on 0.10.4. choco install notepadplusplus --forcex86. When you upgrade every version it will keep --forcex86 unless you pass --ignore-remembered-arguments and do not pass it as part of the upgrade arguments.

[aaccioly]

Hi @ferventcoder, is this enabled by default now or do I still need to run choco feature enable -n useRememberedArgumentsForUpgrades manually to enable it for all packages?

[ferventcoder]

@aaccioly not enabled by default. You can always check with choco feature list. Also subscribe to chocolatey-announce so you will get a notice when it is turned on by default.

[ferventcoder]

Keep in mind there is a known issue and reproing it or determining the cause is quite difficult - #1443

[bcurran3]

This needs to be re-opened.
REF: #1547

[CollinChaffin]

This need to be reopened. Latest versions for some time now actually if you perform a choco upgrade -y ALL what happens on mult machines I can reproduce easily is that if ONE single package in that list had specific install arguments to remember on upgrade, that arg is literally being passed FOREVER to EVERY single MSI as even the logs show. In my case, Notepadplusplus install was installed with this arg: cinst --pre -y notepadplusplus.install -ia '/allowAppDataPluginsLoading=true' and now every package upgrade forever on this machine runs this and here is an error I just got from a MSI that actually does NOT simply ignore the switch unknown to it:

ERROR: Running ["C:\Windows\System32\msiexec.exe" /i "C:\ProgramData\chocolatey\lib\p
andoc\tools\pandoc-2.4-windows-i386.msi" /quiet /allowAppDataPluginsLoading=true] was
 not successful. Exit code was '1639'. Exit code indicates the following: The command
 line arguments passed to the MSI are incorrect. If you passed in additional argument
s, please adjust. Otherwise followup with the package maintainer(s) to get this fixed
......Truncated

Note the NotepadPlusPlus MSI switch being passed to Pandoc which obviously does not nor ever has used that switch unique only to NPP, and the main chocolatey log also even shows all the successful upgrades which also wind up getting passed this switch from a single package remembered args. If this occurs with a switch that IS actually used by other packages, it will be catastrophic.

EDIT: This does appear to be connected to this issue and the option useRememberedArgumentsForUpgrades which is enabled on every up to date Chocolatey install I have. Note this is quoted from the Chocolatey.config file:

<feature name="useRememberedArgumentsForUpgrades" enabled="true" setExplicitly="true" description="Use Remembered Arguments For Upgrades - When running upgrades, use arguments for upgrade that were used for installation ('remembered'). This is helpful when running upgrade for all packages. Available in 0.10.4+. This is considered in preview for 0.10.4 and will be flipped to on by default in a future release." />

Even this comment of This is helpful when running upgrade for all packages is troubling as it is the OPPOSITE of helpful at least in it's present form where a single package like NPP that has a very unique installer switch requirement is then being applied to ALL package upgrades when the ALL param is used.

[CollinChaffin]

@bcurran3 @ferventcoder Hey guys can this be re-opened please? See my comment above and let me know if this is already a fix underway or whether you need additional details above what I already posted. Thanks!

[ferventcoder]

@CollinChaffin We don't reopen something that has been completed. You are seeing a bug with it, which means a new issue. This is by the way corrected in the beta. 347f09f

[CollinChaffin]

@ferventcoder Then why with today's beta is it STILL adding old remembered arguments from a single package I installed over a year ago? Where are remembered arguments stored so I can clear this out and if what you are saying is correct I'm still not clear why I need to do so.

EDIT: And understand that whey I say it adds an old argument - it adds it for HUNDREDS of package upgrades, when it applies to ONE.

For anyone else having this issue and wanting to blank out old remembered arguments, looks like they are stored in .arguments but went from plain text to encrypted/encoded so other than totally disabling all remembered args globally the only way is to decrypt (machine key) each newer .argument file to verify.

[CollinChaffin]

@ferventcoder I just realized this is exactly what you called a MAJOR BUG in issue #1443 - same issue. 400 package upgrades will carry a single argument (like the mentioned 32/64 bit snafoo when NPP arg was carried to VLC and other packages). In my case, I can show a log with today's beta applying a switch of 'ADD_CMAKE_TO_PATH=System' (a switch that obviously applies only to CMAKE) over and over with a cup ALL which on top of just looking silly, is actually causing damage when packages that vomit on that unknown switch enough that they simply fail to upgrade.

It looks like issue #1443 is still at least listed as OPEN and at some point someone understood the severity and I know everyone is busy but your comments that it was a major bug in that issue were made over a year ago, and so I was bringing it up again because it's still not fixed and is in fact severe. Let me know if there is anything else I can test to help.

[CollinChaffin]

As further example, with Chocolatey v0.10.12-beta-3-gb65f586, here is the (relevant) log section upgrading my Git install a week ago showing the bad argument from the CMAKE package install actually being used (more info below):

2018-12-11 22:39:25,916 47288 [WARN ] - 
You have git v2.19.2 installed. Version 2.20.0 is available based on your source(s).
2018-12-11 22:39:25,918 47288 [DEBUG] - git - Adding remembered arguments for upgrade:  --prerelease --install-arguments="'ADD_CMAKE_TO_PATH=System'" --allow-downgrade --cache-location="'C:\Temp\chocolatey'" --use-system-powershell
2018-12-11 22:39:25,923 47288 [DEBUG] - Backing up existing git prior to operation.
2018-12-11 22:39:25,924 47288 [DEBUG] - Moving 'C:\ProgramData\chocolatey\lib\git'
 to 'C:\ProgramData\chocolatey\lib-bkp\git'
2018-12-11 22:39:27,925 47288 [DEBUG] - Attempting to create directory "C:\ProgramData\chocolatey\lib\git".
2018-12-11 22:39:28,799 47288 [DEBUG] - Attempting to copy "C:\ProgramData\chocolatey\lib-bkp\git\git.nupkg"
 to "C:\ProgramData\chocolatey\lib\git\git.nupkg".

Futher, when I decrypt the C:\ProgramData\chocolatey\.chocolatey\git.2.19.2\.arguments file it matches the above log with:

λ  DecryptChocoArgs .\.arguments

 --prerelease --install-arguments="'ADD_CMAKE_TO_PATH=System'" --allow-downgrade --cache-location="'C:\Temp\chocolatey'" --use-system-powershell

But then I got curious since this became very visible when I began reporting it above (November time frame)....and it does appear this may be a result of one or two VERY BAD BUG builds of Choco that somehow picked up ONE package arg and re-embedded it on ALL other packages in the same CUP operation - to then be carried forward as THEIR OWN args as long as that choco global feature is enabled. This was the cause of my initial posts above back in November. The proof:

 C:\ProgramData\chocolatey\.chocolatey  
λ cd git.2.13.0

 C:\ProgramData\chocolatey\.chocolatey\git.2.13.0  
λ  DecryptChocoArgs .\.arguments
 --cache-location="'C:\Temp\chocolatey'"

 C:\ProgramData\chocolatey\.chocolatey\git.2.13.0  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.13.0  
λ cd git.2.13.1

 C:\ProgramData\chocolatey\.chocolatey\git.2.13.1  
λ  DecryptChocoArgs .\.arguments
 --allow-downgrade --cache-location="'C:\Temp\chocolatey'"

 C:\ProgramData\chocolatey\.chocolatey\git.2.13.1  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.13.1  
λ cd git.2.14.1

 C:\ProgramData\chocolatey\.chocolatey\git.2.14.1  
λ  DecryptChocoArgs .\.arguments
 --cache-location="'C:\Temp\chocolatey'"

 C:\ProgramData\chocolatey\.chocolatey\git.2.14.1  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.14.1  
λ cd git.2.16.2

 C:\ProgramData\chocolatey\.chocolatey\git.2.16.2  
λ  DecryptChocoArgs .\.arguments
 --prerelease --allow-downgrade --cache-location="'C:\Temp\chocolatey'"
 
 C:\ProgramData\chocolatey\.chocolatey\git.2.16.2  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.16.2  
λ cd git.2.16.3

 C:\ProgramData\chocolatey\.chocolatey\git.2.16.3  
λ  DecryptChocoArgs .\.arguments
 --prerelease --allow-downgrade --cache-location="'C:\Temp\chocolatey'"

 C:\ProgramData\chocolatey\.chocolatey\git.2.16.3  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.16.3  
λ cd git.2.19.0

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.0  
λ  DecryptChocoArgs .\.arguments
 --prerelease --allow-downgrade --cache-location="'C:\Temp\chocolatey'"

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.0  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.0  
λ cd git.2.19.1

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.1  
λ  DecryptChocoArgs .\.arguments
 --prerelease --install-arguments="'ADD_CMAKE_TO_PATH=System'" --allow-downgrade --cache-location="'C:\
Temp\chocolatey'" --use-system-powershell

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.1  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.1  
λ cd git.2.19.2
 
 C:\ProgramData\chocolatey\.chocolatey\git.2.19.2  
λ  DecryptChocoArgs .\.arguments
 --prerelease --install-arguments="'ADD_CMAKE_TO_PATH=System'" --allow-downgrade --cache-location="'C:\
Temp\chocolatey'" --use-system-powershell

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.2  
λ cd ..

 C:\ProgramData\chocolatey\.chocolatey\git.2.19.2  
λ cd git.2.20.0

 C:\ProgramData\chocolatey\.chocolatey\git.2.20.0  
λ  DecryptChocoArgs .\.arguments
 --prerelease --install-arguments="'ADD_CMAKE_TO_PATH=System'" --allow-downgrade --cache-location="'C:\Temp\chocolatey'" --use-system-powershell

 C:\ProgramData\chocolatey\.chocolatey\git.2.20.0  
λ
 

EDIT: Just re-ran the scan ran in chunks first time wanted to verify had entire log of all versions tested

So as you can see, once this occurred it embedded the CMAKE package args into the Git package and continues to carry forward - forever. Now the question for which I believe I probably can guess the answer is - is there any way to undo this damage to every affected package's arg files? Since I myself had to decrypt the arg file I'm guessing I'm out of luck but wondering if there is any way I've missed with choco.exe - or - can a manual forced upgrade of each and every package using the corrected args also stick? I have not yet tested but that question is more for others affected with only a few packages as I'd be better off writing a utility to fix the arg files before going that route give the sheer number of packages I have that are affected. :( @ferventcoder Rob at least you can see now first hand what I was referring to and hopefully this makes more sense as to the severity.

[ferventcoder]

If you do a manual forced upgrade, it would replace with whatever args you last ran with.

It would be great if there was a way we could (in a more automated way), verify the upgrade args and let a user say, no, look further back and kind of work until the user gives it a heads up. I think this conversation should really be happening over on #1443 though.

[ppvnf]

is this gonna be enabled by default or the idea was dropped out?

[TheCakeIsNaOH]

@ppvnf it still is expected to be enabled by default at some point, but there are still open issues with this feature, specifically #2886 and #2761

Likely it would not be enabled until after there is at least one release with no known or found issues with it.

[ppvnf]

That makes sense. Thank you

[gabriel-vanca]

Any updates on this?

[pauby]

@gabriel-vanca This issue was closed as completed in 2017. If you're looking for an update on this comment, then this has not been abled by default and there are no immediate plans to do so. We will look at it again during the next release cycle.

[gabriel-vanca]

@ppvnf it still is expected to be enabled by default at some point, but there are still open issues with this feature, specifically #2886 and #2761

Likely it would not be enabled until after there is at least one release with no known or found issues with it.

Apologies, I know it was closed, but the issues mentioned here are still open.