Design doc: new builder with explicit options

[stsewd]

I'm suggesting to remove everything basically :D maybe is it too much? Let me know what you think.
Some "big" decisions are:

• Should we stop setting some things from the html_context?
• Should we stop injecting the analytics and canonical url for the users?

Find more while reading the document!

Rendered version: https://docs--8103.org.readthedocs.build/en/8103/development/design/explicit-builders.html

[humitos]

Looks like a good start to me! There is a lot happening here, tho 😄

My notes:

• it seems we are changing completely where we put the effort to onboard people. Moving it from "magic" to "documentation guides"
• I'm fine having guides explaining how to do things, but I'd expect the user to have a nice experience with our platform without having to read them before importing the project
• without "the guessing magic", how do we expect to be the first build? will it succeed without reading all of those guides?
• how we will point the user to the correct problem and guide them towards the solution from the build detail's page when the first build fails?
• as a user, importing a project and seeing it immediately failing is terrible --we would need a super nice and decorated page explaining all the problems together with the solution at once
• failing the first build because .readthedocs.yaml does not exist, and then failing because python.sphinx.conf is not defined, and then because Sphinx is not in the requirements.txt, and then because another and another (over and over) config was mandatory and wasn't defined is not a good experience at all
• having a good "Getting Starting with Read the Docs" document with all the sane config we expect from the user being copy&past-able would help us to reduce the on-boarding barrier
• showing copy&past-able messages from build detail's page on failed builds will also help to improve the experience

I also think we want to guide users towards a "minimum level of docs quality" where all the docs hosted at Read the Docs have all of these features, and readers would expect them to exist (unless the author decided to disable them). This was my thought when I suggested #7870. In my head, using Read the Docs should automatically bump your documentation quality to the next level.

I think that everything that I'm saying here could be achieved with:

• sane default to make everything to Just Work ™️ as much as we can
• make those defaults easily override-able / disabled completely
• improve the on-boarding UX and failed builds UX
• pre-install packages we control (or make suggestions to install them)

Summarizing, I'm fine removing as much magic as we can, but we need to know that this magic helps a lot of new users to have a better on-boarding. However, it generates conflicts to more experienced users. We need to find a sane balance where both type of users can co-exist. Removing all of it expecting the users to read all our guides is not the answer here without working on UX, to me.

[humitos]

What's the purpose of installing Sphinx/MkDocs here?

[stsewd]

So users don't need to have a requirements file for this, and installing them with pip doesn't have the same problems as Conda, they can be easily overridden.

[humitos]

This conflicts in some way with the confusions listed in this document:

The Sphinx version (or from any other tool) isn't the one they expect.

The new builder requires a conf.py. So, they will need to have Sphinx installed locally and they have ran sphinx-quickstart. Then, when importing on Read the Docs, we are going to install a random version (potentially different than the one they have installed locally). In the end, we are not fixing this problem / removing this confusion / making this confusion clearer.

[humitos]

Another things that rings me a bell here are:

• we will need to deal with not installing Sphinx/MkDocs when we support new different doc tools (eg. Hugo, etc)
• Read the Docs behaves differently when using conda than when using pip. I don't think we have a good reason to explain that to users: "conda resolver is broken" is not a good one 😄

[stsewd]

If they don't have a requirements file, they aren't expecting any version at all. If they have sphinx==1.2.3 or sphinx (latest) and we install something different or isn't downgraded than that's a problem.

[humitos]

I'd say we should install only dependencies that are required at build time to make Read the Docs' features to work: readthedocs-sphinx-ext. All the other extra dependencies should be defined by the user. We can start with 0 extra dependencies, and add them later if we find they are required, but at this point, I wouldn't include them here.

[ericholscher]

FWIW, I assume readthedocs-sphinx-ext depends on Sphinx, but we can probably install it without.

I think in this builder we should not install sphinx or mkdocs, and require the user to understand what is happening. Hopefully they can eventually specify it easily in the rtd.yaml (install: sphinx) or similar, but I agree that having less magic to manage is good.

[humitos]

It seems like a good moment to implement the new theme context together with the new builder 👍 --whatever its structure would be, but having it under a namespace (readthedocs.v1.*) is key to me.

[ericholscher]

Agreed -- though we should also heavily minimize the data to start, treating it basically as an API 👍

[humitos]

It seems that this is not decided yet. The document proposes to remove (if possible) passing html_context or at least reduce it.

Considering that we are trying to avoid this magic, I'd propose to do a little research to understand if it's possible to not touch this variable from our side and remove it completely since it's a Sphinx-only feature. Also, because we are trying to remove ourselves from the build process.

(Note: I'm considering using an intermediate tool-agnostic metadata.yaml file for required context like this in a future builder)

[humitos]

This approach makes sense if we are going to support other building tools in the future. This will probably have some impact on big projects with many HTML files, since we will need to rewrite all of them.

[ericholscher]

Yea, I think adjusting the built HTML is the right way to handle this, so we can support non-Sphinx builders. In mkdocs we're just adding this to their setting:

readthedocs.org/readthedocs/doc_builder/backends/mkdocs.py

Lines 175 to 178 in 924699c

	user_config.setdefault('extra_javascript', []).extend(
	[js for js in extra_javascript_list if js not in user_config.get(
	'extra_javascript')]
	)

But it seems like explicitly doing it after build against all HTML is more extensible.

[ericholscher]

Breaking it out from the extension into the build process is probably correct here. 👍

[humitos]

The build should fail, IMO. We should show the problem in the build detail's page instead in a 404 page. We already known that the build is not in a good state and we should communicate that to the user.

[ericholscher]

Agreed, fail the build.

[humitos]

This is related to the sane defaults we talked about. I think this provides value to non-experienced users and we haven't had troubles with it. Also, this is one of the cases where an experienced user can override these configs easily. I'd like to have more like this one instead of less ones 😄

[ericholscher]

I have mixed feelings here. If we're doing almost nothing in our conf.py override, I do think there's value in trying to get that to nothing at all.

If the goal here is giving users options here and letting them configure it with guides, I think that is a reasonable solution for most cases, but we need to figure out where the line is.

I think getting to the point where we don't override the users conf.py at all, if possible, would be a huge win for simplifying our builds. I don't know if it's realistic though.

[humitos]

After talking to @agjohnson I changed my mind here. I prefer to do nothing in the config file and write a good guide with a copy&paste-ready code explaining this.

I think getting to the point where we don't override the users conf.py at all, if possible, would be a huge win for simplifying our builds. I don't know if it's realistic though.

Agree. This is where I'd like to end up as well after the new builder.

[humitos]

These are the kind of the features that "bump your documentation to the next quality level" just because you are using Read the Docs.

You don't have to worry about dealing with Google Analytics code, canonical URLs, sitemap, robots, etc

[stsewd]

Analytics is already an opt-in option, so I'd be ok with moving the canonical url to an option too (we hit problems with injecting the canonical url twice). I'm +1 on giving users more control over this kind of settings instead of removing them.

[humitos]

I'm fine moving them to options and I'd like to see them built outside the extension as a post-processing HTML feature instead.

[ericholscher]

This is a good start. We definitely need to think a bit more about the migration and final state we're hoping to get to.

[ericholscher]

Yea, I think adjusting the built HTML is the right way to handle this, so we can support non-Sphinx builders. In mkdocs we're just adding this to their setting:

readthedocs.org/readthedocs/doc_builder/backends/mkdocs.py

Lines 175 to 178 in 924699c

	user_config.setdefault('extra_javascript', []).extend(
	[js for js in extra_javascript_list if js not in user_config.get(
	'extra_javascript')]
	)

But it seems like explicitly doing it after build against all HTML is more extensible.

[ericholscher]

This makes sense to me 👍

[humitos]

I'm not sure if it makes sense to migrate the version warning banner to the new builder. We have had a lot of different opinions about how it should work and we ended up creating a Sphinx extension that allow users to customize different aspects of it.

I'd say that it would be better to remove it completely and either:

1. maintain the extension we already have --inviting mkdocs (and other doctools) community to build their own extension for this, or
2. implement the JS version at Read the Docs that allow customization (text, link to "latest" version, etc)

I think 1) is better if we want to go in the "provide integration points where the community can hook into". Besides, it's less work for us 😄

[ericholscher]

I have mixed feelings here. If we're doing almost nothing in our conf.py override, I do think there's value in trying to get that to nothing at all.

If the goal here is giving users options here and letting them configure it with guides, I think that is a reasonable solution for most cases, but we need to figure out where the line is.

I think getting to the point where we don't override the users conf.py at all, if possible, would be a huge win for simplifying our builds. I don't know if it's realistic though.

[ericholscher]

Should we get rid of web settings entirely for things that are in the config file?

[stsewd]

I think they are still useful when you want to build from an existing release, since you can't change that code to add a config file.

[humitos]

I think we should remove them and provide a nice way to support version aliases. So, in case you have to re-build an old tag (e.g. 2.1) you can create a new branch/tag (e.g.2.1.1) and keep the same URL and everything but with updated/fixed docs by using an alias for that new branch/tag.

[ericholscher]

This section could probably use a bit more detail. Migrating all our users is going to be a lot of work. There's a number of things we could do to help them migrate though.

[stsewd]

Wasn't sure how to expand this, I have updated it to put a list of things that could be done.

[ericholscher]

I do wonder about implemented this given the change to the YAML file. I think that seems like a natural chance to break compatibility. @stsewd what needs to happen to unblock this? I believe @humitos had some thoughts here, but I'm 👍 on moving forward with a very simple builder for sphinx.yaml if we can get everything ready in time.

[stsewd]

@ericholscher I guess all the conversations that aren't marked as resolved still need some decisions, but I put on the doc that we can implement this new builder under a feature flag, so we don't need to implement/decide everything in one single iteration/PR.

[ericholscher]

@stsewd gotcha -- then I think we can move forward towards implementation then, unless anyone objects. I'm 👍 on it.

[humitos]

I'd suggest making this new builder opt-in and use build.builder: 2 instead of a feature flag. I'm not sure we can deprecate and remove the builder that we currently have. I think it's a good product for .com users.

[humitos]

I believe @humitos had some thoughts here

I had a long conversation with Anthony last week about "the new builder" and I considered this PR as the next step in my proposal before reaching the final goal (*). So, I was 👍🏼 moving forward here. However, @agjohnson ask me if it may not make sense to move directly to the final step without this intermediate builder to avoid maintaining 3 builders in the future. Now, I'm not yet sure what's the best next step. I suggested Anthony talk to you @ericholscher explaining what we talked about before moving forward in case it helps to reduce the amount of work required here.

In any case, for this PR I'd expect the following before start implementing it:

• update the document with the conversation we had here
• make the new builder opt-in: set the builder in the config file using build.builder: 2 instead of a feature flag
• do not consider builder v1 (current one) as deprecated: it's a good product for .com users and I think we can't remove it completely
• try to migrate all features we can from readthedocs-sphinx-ext to post-processing HTML. I'd expect some features will require being re-written in Javascript
• research about the possibility to remove the injection of html_context

I'm happy to keep talking about this if needed. I think we should have very clear what's the final goal related to executing custom commands (via post_/pre_ (clone, install, build, etc) hooks as we talked before exposing this new builder to users. With this being clear to us, we can build a clear path to move forward in a compatible way without breaking user's builds.

(*) the final goal is to not require control on the build process (dependencies, commands, etc): RTD would only work over the generated HTML + expected metadata.yaml generated by the user at build time

[ericholscher]

However, @agjohnson ask me if it may not make sense to move directly to the final step without this intermediate builder to avoid maintaining 3 builders in the future

Maybe the thing that makes the most sense is just building a "v2" builder that literally does nothing other than run the sphinx & mkdocs commands, then we can build on top of that over time, but users who want that functionality can opt into it? I agree though, if we end up allowing custom build commands, at some point this isn't even a builder, it's just a thing that calls the users command and post-processes output. Maybe we should start with that and see how it works?

[humitos]

I'm fine creating a new builder that does nothing (do not install extra dependencies) if we can decouple as much as we can the work we do in conf.py.tmpl/mkdocs.yaml at the same time. Otherwise, this new builder will still require access at build time to do our magic. I'm fine leaving the readthedocs-sphinx-ext requirement and installation as a Sphinx extension for now because the user does not depend on it and we can remove it later, but I'd like to remove all the other magic from it (e.g. injecting html_context, latex configs, etc)

I want to make sure that we can guarantee ourselves forward compatibility with the future changes to reach our goal.

However, I don't see too much value right now by having a builder that does nothing but doesn't add functionality either without having a clear path and an idea for a contract defined. It's a good intermediate step towards the final goal, though. This is the main reason why I think it worth to keep discussing a little more before proceed building this and then realizing that we still have some magic in our build process that users depend on.

[ericholscher]

I'm fine creating a new builder that does nothing (do not install extra dependencies) if we can decouple as much as we can the work we do in conf.py.tmpl/mkdocs.yaml at the same time. Otherwise, this new builder will still require access at build time to do our magic. I'm fine leaving the readthedocs-sphinx-ext requirement and installation as a Sphinx extension for now because the user does not depend on it and we can remove it later, but I'd like to remove all the other magic from it (e.g. injecting html_context, latex configs, etc)

Right, this work is required to "do nothing" and post-process the output. We would hopefully remove all tool-specific integration on our side, if possible. Presumably passing in environment variables instead of editing html_context and documenting how to load them into your config/build? This would mean not touching the users conf.py or mkdocs.yml.

The hardest thing here is probably search, where we're injecting code to get data out of the Sphinx internals. We'll need to decide if that is worth the complexity, or move that code into something users can enable if they want as a third-party extension. (eg. pip install readthedocs-extras?)

I want to make sure that we can guarantee ourselves forward compatibility with the future changes to reach our goal.

Agreed. We should be clear on what those future goals are, and design for them when we're building the system.

However, I don't see too much value right now by having a builder that does nothing but doesn't add functionality either without having a clear path and an idea for a contract defined. It's a good intermediate step towards the final goal, though. This is the main reason why I think it worth to keep discussing a little more before proceed building this and then realizing that we still have some magic in our build process that users depend on.

Agreed. I think the biggest question in my mind is around allowing users to customize the build commands -- it seems we want to get there eventually but we aren't ready. However, we could pretty easily build a system that allows customization of a subset of commands while still being able to eventually support that.

My thinking:

• a "v1" of this project doesn't install any dependencies or tool-specific code, post-processing output where needed (eg. for injecting the RTD css/js, etc.), but doesn't allow users to customize the build process yet. This probably isn't super useful, but is the required rearchitecting we need to get the build system where we want it. Some users will prefer this anyway, even without any customization.
• a "v2" of this project, which gives users more control over the entire build process, if we decide that is something that we want to do. This could include pre/post install & build hooks, along with customizing the install & build commands themselves.

The v1 of this plan is basically what we're discussing in this PR, but we should have a good idea of what v2 might look like, so we can design it properly.

[humitos]

@ericholscher All you said makes sense to me 👍🏼

I went ahead today's morning and wrote in a document (see #8190) all I had in mind about this topic, what I talked to Anthony and consider Santos' PR as a good intermediate step, and also the roadmap conversations we already had. It's not ready for review yet, just a way to early communicate my concern about a clear path and how to make the correct steps in that direction.

I don't want to deviate the conversation we are having here to a new PR, but use the new PR as a proposal for a final goal to make the required changes on this PR to reach that goal --avoiding being blocked later due to a dependency that we cannot remove.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

Closing this PR as we already took the ideas from here and we are moving in a different direction. See #8190 for more information. Please, if you think there are still good ideas to implement here, open a new issue or comment into the new issues that are tracking build.commands and its surroundings.