Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multiple TypeScript projects #16430

Merged
merged 28 commits into from
Apr 24, 2024

Conversation

nicolo-ribaudo
Copy link
Member

@nicolo-ribaudo nicolo-ribaudo commented Apr 14, 2024

Q                       A
Fixed Issues? Fixes #1, Fixes #2
Patch: Bug Fix?
Major: Breaking Change?
Minor: New Feature?
Tests Added + Pass? Yes
Documentation PR Link
Any Dependency Changes?
License MIT

With Project References, tsc is able to only type-check "projects" whose contents changed, without having to re-check the whole monorepo every time.

Usually, when using a monorepo, there is a 1:1 mapping between packages and TypeScript projects. In our case this doesn't work, because Project References do not support cycles. Instead, we generate one project per strongly connected component in our packages graph: we currently have only one SCC1, and its tsconfig.json is in packages/babel-helper-plugin-utils/tsconfig.json.

All the tsconfig.json files are not manually maintained (except for /tsconfig.base.json), but they are generated via node scripts/generators/tsconfig.js.

See #16430 (comment) for how this impacts our development workflows.

Old description

I also experimented with running TypeScript on multiple cores rather than in just one. tsc doesn't support it by default, but you can use node scripts/parallel-tsc/tsc.js . (or passing the path of any other project) to do it. It seems to be ~40% faster than plain tsc on a MacBook M2 with 8 cores (but for some reason only 6 are being used).

This PR makes running a full type checking significantly slower (this will affect CI), but it makes incremental builds significantly faster (up to 10x, depending on which package is modified, and this will affect local development). This are the results I'm getting now (but I was getting different results this morning, unless I hallucinate them):

  • without this PR, rm -rf dts tsconfig.tsbuildinfo && yarn tsc -b . takes 31s (both with and without skipLibCheck)
  • with this PR, make clean-ts && yarn tsc -b . takes 5m 22s
  • with this PR and skipLibCheck: true, make clean-ts && yarn tsc -b . takes 1min 53s
  • with this PR, make clean-ts && node scripts/parallel-tsc/tsc.js . takes 2m 42s
  • with this PR and skipLibCheck: true, make clean-ts && node scripts/parallel-tsc/tsc.js . takes 1min 16s

I believe the reason skipLibCheck: true significantly improves performance is that it lets TS avoid re-validating all the .d.ts files it generated for the intermediate projects. I will mark this PR as ready once we can get the skipLibCheck: true results while still validating our manually-written .d.td files.


@jakebailey Here is the branch :) Please feel free to ask any question that might help you, if you are going to take a look

Footnotes

  1. (@babel/helper-plugin-utils <> @babel/code-frame <> @babel/compat-data <> @babel/core <> @babel/generator <> @babel/helper-check-duplicate-nodes <> @babel/helper-compilation-targets <> @babel/helper-environment-visitor <> @babel/helper-fixtures <> @babel/helper-function-name <> @babel/helper-hoist-variables <> @babel/helper-module-imports <> @babel/helper-module-transforms <> @babel/helper-plugin-test-runner <> @babel/helper-simple-access <> @babel/helper-split-export-declaration <> @babel/helper-string-parser <> @babel/helper-transform-fixture-test-runner <> @babel/helper-validator-identifier <> @babel/helper-validator-option <> @babel/helpers <> @babel/highlight <> @babel/parser <> @babel/template <> @babel/traverse <> @babel/types, with 26 packages out of ~150, but I'm looking into reducing its size by splitting some packages across multiple projects to reduce cycles).

@babel-bot
Copy link
Collaborator

babel-bot commented Apr 14, 2024

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/56759

@nicolo-ribaudo
Copy link
Member Author

nicolo-ribaudo commented Apr 18, 2024

The last commit (d826ec0, Make types faster) significantly improves the situation by:

  • generating some giant object types at build time rather than at type checking time
  • replacing some type aliases with interface (that now we can do because the object shapes are statically known)
Before After
make clean-ts && yarn tsc -b ., skipLibCheck: false 5m 10s 3m 38s
make clean-ts && yarn tsc -b ., skipLibCheck: true 1m 53s 24s
make clean-ts && node scripts/parallel-tsc/tsc.js ., skipLibCheck: false 1m 9s 59s
make clean-ts && node scripts/parallel-tsc/tsc.js ., skipLibCheck: true 41s 22s

Before this PR, tsc . with no cache and skipLibCheck: false takes 31s.

@nicolo-ribaudo

This comment was marked as outdated.

@nicolo-ribaudo

This comment was marked as outdated.

@nicolo-ribaudo

This comment was marked as outdated.

@nicolo-ribaudo
Copy link
Member Author

Type checking now runs with skipLibCheck: true, except for in the root project which:

  • uses skipLibCheck: false
  • checks that all the generated .d.ts files are valid

This also caught some errors in the .d.ts files (658b19c and e2ebc5e), that TS was not reporting because it was assuming that the .d.ts files were valid.

It takes ~30 seconds to run, which is acceptable.

@nicolo-ribaudo
Copy link
Member Author

nicolo-ribaudo commented Apr 22, 2024

yarn eslint packages --format codeframe is crashing with OOM, and even when I set --max-old-space-size=8196 it takes 1m22s. This is after running make tscheck, so that all the .d.ts and .tsbuildinfo files are pre-generated.

cc @bradzacher @JoshuaKGoldberg I assume that this is just because we have 140 projects, but I'm pinging you just in case you want to take a look :) Linting is already slow for us, so this is not a blocker. It's weird however that when linting the same code, it breaks when linting it as multiple projects but works when using a single giant one.


EDIT NODE_OPTIONS="--max-old-space-size=8196" yarn eslint scripts benchmark packages codemods eslint '*.{js,cjs,mjs,ts}' --format codeframe also OOMs 😢

EDIT2: NODE_OPTIONS="--max-old-space-size=8196" yarn eslint packages --format codeframe works, I'll just split ESLint in multiple commands that are run one by one

@jakebailey
Copy link

I was going to suggest EXPERIMENTAL_projectService, but it looks like you've already set that... That should have helped with the OOM (and maybe it does but is not enough). 🙁

Given ESLint doesn't tell plugins when it's done with files, perhaps this is a case of every single file being left open. I tried to take a pprof profile of the run to see where all the memory is going, but that segfaults...

@nicolo-ribaudo
Copy link
Member Author

NODE_OPTIONS="--max-old-space-size=12294" yarn eslint scripts benchmark packages codemods eslint '*.{js,cjs,mjs,ts}' --format codeframe is enough, but it's surprising that when using project references the required memory goes from the default 1.5GB to 12GB 😬

Given ESLint doesn't tell plugins when it's done with files, perhaps this is a case of every single file being left open.

This shouldn't be affected by using multiple projects, right?

@jakebailey
Copy link

jakebailey commented Apr 22, 2024

This shouldn't be affected by using multiple projects, right?

It would; each project is its own "thing" internally, so the project service is actually spinning up multiple projects (same as the editor) as files are opened by ts-eslint. But if they're never closed, it's kinda like if you opened every file in the editor and never closed them again. But within reason, that shouldn't be so massively different than tsc -b either, since it also has to load everything... but the ProjectService is heavier than build mode is.

I'm also checking to see if this is the forceConsistentCasingInFileNames bug, which would be awesome (for us) since we'd finally have an OSS repro... EDIT: Nope, doesn't seem like it.

@nicolo-ribaudo nicolo-ribaudo marked this pull request as ready for review April 22, 2024 17:29
@nicolo-ribaudo
Copy link
Member Author

nicolo-ribaudo commented Apr 22, 2024

This PR is ready for review!

Some info:

  • a complete type checking takes about 30s both in this PR and on main
  • full type checking + linting is now a bit slower (1m20s on main, 1m40s with this PR)
    • make lint now also does type checking to ensure that all the TS projects are pre-compiled, but it will only perform an incremental compilation
  • make tscheck now also validates all the generated .d.ts files, to make sure that there wasn't a suppressed error that made it way to .d.ts. This will be helpful when we publish .d.ts files
    • this means that the manually-written .d.ts file will be checked as part of this step, and we can in other cases run with skipLibCheck: true
  • make tscheck is not exactly the same as yarn tsc -b ., because it also copies manually-written .d.ts files into the dts folder. It needs to be done interleaved with type checking, so that they all all there when checking the .d.ts folder. Unfortunately TSC doesn't support this (see Suggestion: "emitDts" option for exporting handwritten declarations  microsoft/TypeScript#39231)
  • When editing a file locally, incremental type checking now works. For example, if I add let x: string = 2 to packages/babel-helper-module-transforms/src/index.ts, yarn tsc -b . only takes 2.1s 🚀
  • As a rule of thumb, use yarn tsc -b . when doing an incremental check, and make tscheck when you are not sure (or when you need some manually-written .d.ts to be re-copied)

Also, I suggest reviewing filtering away JSON files since the diff is big and GitHub's UI struggles.

@@ -1,5 +1,3 @@
/// <reference path="../../../lib/third-party-libs.d.ts" />

import type { Token as JSToken, JSXToken } from "js-tokens";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was unnecessary and causing an invalid .d.ts file

type === "UpdateExpression"
? NodeDescriptions.UpdateExpression[String(prefix) as "true" | "false"]
: NodeDescriptions[type];
const toNodeDescription = (node: NodeWithDescription) =>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was causing an invalid .d.ts file, because TS drops the @ts-expect-error

@liuxingbaoyu
Copy link
Member

I'm not sure if having Make types faster even without using Project References will be enough, but now that we've done that it's good to merge it. :)
lint is a bit strangely slow in this case, but it's acceptable for me.
This may affect the eslint-vscode extension, if you are using it you might want to take a look,
Also this might unlock strictNullChecks, which I remember would OOM in the past, which is a bonus!

@nicolo-ribaudo
Copy link
Member Author

eslint-vscode is not affected by the OOM problem because it only runs on the currently open files :)

@nicolo-ribaudo
Copy link
Member Author

Also this might unlock strictNullChecks, which I remember would OOM in the past, which is a bonus!

We can now also migrate to strictNullChecks project-by-project, rather than having to do everything at the same time!

@liuxingbaoyu
Copy link
Member

liuxingbaoyu commented Apr 22, 2024

Will just running make lint work? There seems to be some failure on my local machine. 🤔

dts/packages/babel-parser/src/plugins/typescript/index.d.ts:414:150 - error TS2536: Type '"type"' cannot be used to index type 'T_19'.

414         finishImportSpecifier<T_19 extends N.ImportSpecifier | N.ImportDefaultSpecifier | N.ImportNamespaceSpecifier>(specifier: Undone<T_19>, type: T_19["type"], bindingType?: BindingFlag): T_19;
                                                                                                                                                         ~~~~~~~~~~~~

dts/packages/babel-parser/src/plugins/typescript/index.d.ts:448:101 - error TS2344: Type 'T_20' does not satisfy the constraint 'Node'.

448         finishCallExpression<T_20 extends N.CallExpression | N.OptionalCallExpression>(node: Undone<T_20>, optional: boolean): T_20;
                                                                                                        ~~~~

dts/packages/babel-parser/src/plugins/typescript/index.d.ts:643:41 - error TS2307: Cannot find module '../../typings' or its corresponding type declarations.

643         expectPlugin(pluginName: import("../../typings").Plugin, loc?: Position): true;
                                            ~~~~~~~~~~~~~~~

dts/packages/babel-parser/src/plugins/typescript/index.d.ts:644:45 - error TS2307: Cannot find module '../../typings' or its corresponding type declarations.

644         expectOnePlugin(pluginNames: import("../../typings").Plugin[]): void;
                                                ~~~~~~~~~~~~~~~

dts/packages/babel-parser/src/plugins/typescript/index.d.ts:673:234 - error TS2307: Cannot find module '../../typings' or its corresponding type declarations.

673         getPluginOption<PluginName extends "decorators" | "flow" | "typescript" | "pipelineOperator" | "importAttributes" | "estree" | "moduleAttributes" | "optionalChainingAssign" | "recordAndTuple", OptionName extends keyof import("../../typings").PluginOptions<PluginName>>(plugin: PluginName, name: OptionName): import("../../typings").PluginOptions<PluginName>[OptionName];
                                                                                                                                                            
                                                                                 ~~~~~~~~~~~~~~~

dts/packages/babel-parser/src/plugins/typescript/index.d.ts:673:324 - error TS2307: Cannot find module '../../typings' or its corresponding type declarations.

673         getPluginOption<PluginName extends "decorators" | "flow" | "typescript" | "pipelineOperator" | "importAttributes" | "estree" | "moduleAttributes" | "optionalChainingAssign" | "recordAndTuple", OptionName extends keyof import("../../typings").PluginOptions<PluginName>>(plugin: PluginName, name: OptionName): import("../../typings").PluginOptions<PluginName>[OptionName];
                                                                                                                                                            
                                                                                                                                                            
               ~~~~~~~~~~~~~~~

dts/packages/babel-preset-env/src/index.d.ts:2:30 - error TS2307: Cannot find module './types.ts' or its corresponding type declarations.

2 import type { Options } from "./types.ts";
                               ~~~~~~~~~~~~

dts/packages/babel-preset-env/src/normalize-options.d.ts:2:74 - error TS2307: Cannot find module './types.ts' or its corresponding type declarations.       

2 import type { BuiltInsOption, CorejsOption, ModuleOption, Options } from "./types.ts";
                                                                           ~~~~~~~~~~~~

dts/packages/babel-preset-env/src/normalize-options.d.ts:27:27 - error TS2307: Cannot find module '../../babel-helper-compilation-targets/src/types' or its corresponding type declarations.

27         browsers?: import("../../babel-helper-compilation-targets/src/types").Browsers;
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

dts/packages/babel-standalone/src/index.d.ts:15:65 - error TS2307: Cannot find module '../../babel-preset-env/src/types' or its corresponding type declarations.

15     env: (api: import("@babel/core").PresetAPI, options: import("../../babel-preset-env/src/types").Options, dirname: string) => PresetObject;
                                                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Found 64 errors.

[140/140] ./ took 3.55s
Error:
command: C:\Program Files\nodejs\node.exe scripts/parallel-tsc/tsc.js .
code: 1
    at GM (F:\babel\Makefile.js:4:204642)
    at $M (F:\babel\Makefile.js:4:204802)
    at CM.tscheck (F:\babel\Makefile.js:4:209984)
    at j.target.<computed> [as tscheck] (F:\babel\Makefile.js:4:107414)
    at F:\babel\Makefile.js:4:107521
    at Array.forEach (<anonymous>)
    at Timeout._onTimeout (F:\babel\Makefile.js:4:107475)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
make: *** [Makefile:61: tscheck] Error 1

Will just running make lint work? There seems to be some failure on my local machine. 🤔

Seems to be related to glob on Windows, I'm investigating.
image

Update: Well, glob only supports / not \. :)

@bradzacher
Copy link
Contributor

Jake has it right - there's sadly no infra in place for eslint to tell us when it's done with a file. There's also nothing to differentiate the persistent usecase (IDEs) from the "single-run" usecase (CLIs) which means we have to try and guess and fall back to "keep everything just in case".

140 is a lot of packages so it's unsurprising that it's OOMing. Essentially you're hoping that TS will share things as much as possible. But in reality even with the project service TS has to create a bunch of ts.Programs and none of them can share types. So over the lifetime of the run you pay the duplicate cost of all the interdependencies. The project service dedupes the ts.SourceFiles but cannot dedupes the types.
Eg if you have A, B and C, and both B and C depend on A - then you end up with 3 copies of A's types by the end of the run. Hence OOM 💥

There's sadly nothing we can do about this. We've discussed with the ESLint team and submitted proposals but have been rejected thus far.

The best solution is to isolate things and spawn one eslint process per package. This way you physically isolate things and thus manually cleanup the memory when a package is done linting. I see you essentially did this already. Note that something like nx does this automatically heh IIRC.

I have an old prototype CLI which was essentially a wrapper around eslint and did said splitting across child processes (typescript-eslint/typescript-eslint#4359) which was designed to do this generically and showed some good results in testing. It might be time to revive that 🤔

tsconfig.json Outdated Show resolved Hide resolved
/* This file is automatically generated by scripts/generators/tsconfig.js */
{
"compilerOptions": {
"paths": {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, i'm not sure why you are using ts paths here and not npm workspaces with exports
also using lib like wireit might help with eslint issue

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I... don't know 😅 It's what we were already doing before this PR. I guess it might be because we don't mark type-only dependencies as dependencies of packages, so the node_modules/workspaces structure is not complete for what TS needs.

Copy link

@alyahmedaly alyahmedaly Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the node_modules/workspaces structure is not complete for what TS needs.

I didn't got this point, I think keeping what you have now but with using ts project refs and drop the ts-paths file makes more sense I guess...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoiding paths would be great if you can; doing so effectively lies to the compiler about module layouts. Sometimes that's okay (especially for bundled projects), but I'm sorta surprised that this wouldn't "just work" if this repo is already a monorepo with package linking.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remove paths TS complains that it cannot find the type definitions inside node_modules (probably because we store them in a separate directory) -- I'll keep using paths because TS is able to automatically map from the source files to their corresponding .d.ts.

Copy link
Member Author

@nicolo-ribaudo nicolo-ribaudo Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll experiment in a separate PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's worth noting that if you're not doing a --fix run then you can actually pass the ts.Program instance and reuse the types for the lint run.

https://typescript-eslint.io/packages/parser#programs

This would net you a modest speedup by skipping one "typecheck" cycle.
Note - you'd need to switch from invoking tsc via a child process to directly using the typescript APIs to do this though.
Thought it was worth mentioning in case you want speeeeed

@nicolo-ribaudo
Copy link
Member Author

@bradzacher Thanks for the explanation! Yes, I was indeed expecting typescript-eslint or the ts service to indeed share types across projects rather than re-computing them every time.

I also just realized that yes we are now spawning multiple processes, but we are doing it serially rather than in parallel. This might be easy way for us to save some time :)

scripts/generators/tsconfig.js Outdated Show resolved Hide resolved
scripts/generators/tsconfig.js Outdated Show resolved Hide resolved
@nicolo-ribaudo nicolo-ribaudo changed the title Use multiple TypeScript projects to support incremental type checking Use multiple TypeScript projects Apr 24, 2024
@nicolo-ribaudo nicolo-ribaudo merged commit ee48754 into babel:main Apr 24, 2024
50 checks passed
@nicolo-ribaudo nicolo-ribaudo deleted the ts-project-references branch April 24, 2024 13:11
@github-actions github-actions bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Jul 25, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated A closed issue/PR that is archived due to age. Recommended to make a new issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants