[RFC] First draft of deleted file check

[wilsonge]

Pull Request for some of Issue #16458 .

Summary of Changes

Reworks our deleted file script onto something more rudimentary however that takes into account our npm and composer deps are no longer stored in the repo.

Things left to do

• Make the folder paths dynamic at minimum - better to use git to build them ourselves
• Don't trash any installation removed files
• Don't trash search folders or files
• Reverse order the folders list so we delete the nested directories first

Testing Instructions

This script has been used to generate the existing deleted files list - which are already usable for update - but has had a few modifications since.

Basically in the build create a tmp folder grab a copy of two joomla versions - the 3.x version goes into the a joomla390 folder and the 4.x into joomla400 folder (i picked 3.9.13 and 4.0.0 alpha 12) to generate the list in this PR). Then run the script - it should match what's in the com_admin script

Documentation Changes Required

n/a

[mbabker]

You definitely want the --from and --to params from my version of this script here, as you're going to want this script to be able to work without someone needing to arbitrarily set up multiple git clones on their own to run it. Then, copy the logic from the build script to run the git archive command to build your two working directories and do any of the setup steps needed (i.e. NPM and Composer plus cleanup).

The --from param, you'll probably want to restrict to real version numbers (tags) instead of any valid commit reference like it does now, because you're going to need it to behave differently based on what version you're doing the comparison from (3.x doesn't have build steps, 4.x does). I think it's safe to say this version of the script will only be run when doing a comparison to a commit in the 4.x tree, so that restriction isn't needed for the --to param.

[wilsonge]

The problem with git archive is that it gives me all my dev deps as well which I then somehow need to unstage after I've run composer install and npm install. I think we actually need to somehow split the build script out a bit because I need some of the common parts like the files/directories to ignore there

[wilsonge]

Pushed a new tweak up - a horrible version of doing a iterator filter (there must be a better way I can deal with the excluded folders other than the prepending which requires separate filters) - but at least it works. This means com_search and installation directories are excluded and there's a simple mechanism for this in the future.

I've also reversed the order of the directories so we delete subdirectories before parent directories

[wilsonge]

Just used this to update the latest script.php in 4.0-dev so will be able to use that as a metric of how stable this script is beyond the requirement to be able to download the two branches (for now i just downloaded and unzipped the latest nightlies of 3.10 and 4.0)

[SharkyKZ]

How to use this script? I tried php deleted_file_check.php --from joomla390 --to joomla400 get Missing ending directory due to extra colon here

joomla-cms/build/deleted_file_check.php

Line 39 in 9d216a2

$options = getopt('', array('from:', 'to::'));

.

[wilsonge]

php build/deleted_file_check.php --from=/Users/george/Downloads/Joomla_3.10.0-dev-Development-Full_Package --to=/Users/george/Downloads/Joomla_4.0.0-beta1-dev-Development-Full_Package

That's what I did - where the directories are the two nightlies.

I haven't changed that line from the working J3 version :/ so is this script working for you there?

[SharkyKZ]

That works, thanks. I haven't used this script before.

Should templates be deleted? They can contain custom overrides and assets.

[wilsonge]

Currently the intention was yes. Because we’re not shipping bootstrap 2 in j4 so they will be broken by default. However I’m willing to listen on it. If people don’t like that decision it’s easy to add it to the exclusion list

[mbabker]

Currently the intention was yes. Because we’re not shipping bootstrap 2 in j4 so they will be broken by default. However I’m willing to listen on it. If people don’t like that decision it’s easy to add it to the exclusion list

Lookup the 2.5 to 3.0 scripts. IIRC the admin template was fully uninstalled but the frontend template wasn't. The same should be done for 3.x to 4.0.

[wilsonge]

The 2.5 template didn’t break I don’t think because it had no framework under it? Not 100% on that though

[mbabker]

Protostar has its own CSS, it’s not using Bootstrap from the media directory. So that part isn’t busted. The JavaScript integrations probably are though, not much that can be done for that without a compatibility plugin overloading JHtmlBootstrap to restore BS2 (the same as my old BS3 demo plugin). Likewise layouts are busted because of the insistence of the default layouts being designed against the default frontend template and not being agnostic.

Still it’s probably be a bad idea to flat out wipe out someone’s template.

[brianteeman]

2.5 site and admin templates survived after the update

[richard67]

@wilsonge For me this is at least a release blocker due to following reason:

We want to have updateability from beta-n to beta-(n-1).

So latest with beta-2 we need it to be included into our build procedure so the script.php is up to date for each build.

If we want to go one step beyond and support updateability between nightly builds beginning with beta-1, then it even could be a beta-blocker.

Which of these 2 labels should I set? Feedback very welcome.

[wilsonge]

I've added an explicit release blocker - but the root issue it's partly solving is already a release blocker.

I don't think it's a beta blocker as we can already use this script - it's just about making it pretty to use

[richard67]

@wilsonge I think it should run automatically on every nightly build at the end.

[wilsonge]

It can not run automatically - its building array's to paste into script.php. unless we made these contents a standalone file - but that's out of scope of this PR. I'm manually putting the results into script.php right now (as was the case in 3.x too)