rustdoc: run on plain Markdown files & replace the use of pandoc for guides/tutorial/manual etc. #12747

huonw · 2014-03-07T03:51:27Z

This gives Rustdoc the ability to render our guides, tutorial and manual and modifies the those documents (minor modifications) and makefiles to achieve this.

See commits for more details (especially the makefile rewrite).

huonw · 2014-03-07T03:54:51Z

This is mainly just to get a review on the implementation, it still can't generate a table of contents (working on that now).

cc @alexcrichton

alexcrichton · 2014-03-07T08:15:15Z

src/librustdoc/html/markdown.rs

@@ -266,6 +265,15 @@ pub fn find_testable_code(doc: &str, tests: &mut ::test::Collector) {
            })
        }
    }
+    extern fn header(_ob: *buf, text: *buf, level: libc::c_int, opaque: *libc::c_void) {
+        unsafe {
+            vec::raw::buf_as_slice((*text).data, (*text).size as uint, |text| {


This may be null (at least sundown checks for it), so it's probably worth checking for null up front.

I also think that str::raw gives some better usage than vec => str::from_utf8 (I only realized this much later after writing this code).

I don't understand "some better usage"? Using this avoids allocations, unlike most functions in str::raw (I think)?

Ah I just found that there were functions that didn't require a closure/extra level of indentation, and it goes straight to a string from a buffer. Not really necessary though, I wouldn't be worried about allocations at all at locations like this.

alexcrichton · 2014-03-07T08:19:56Z

Ah, I see, this is a work in progress. This seems fine to me, I'd love to drop the pandoc/node deps!

huonw · 2014-03-07T08:22:42Z

Oh, erm... I forgot to add the new markdown.rs file itself... how embarassing.

huonw · 2014-03-07T08:25:10Z

Added. The patch probably makes more sense now...

alexcrichton · 2014-03-07T08:27:22Z

src/librustdoc/markdown.rs

+
+fn load_string(input: &Path) -> io::IoResult<Option<~str>> {
+    let mut f = try!(io::File::open(input));
+    let d = try!(f.read_to_end());


You could even combine this:

Ok(str::from_utf8_owned(try!(File::open(input).read_to_end())))

(not that it's a whole lot clearer, more just combining the two try! statements.

alexcrichton · 2014-03-07T08:29:55Z

Aha, this does indeed make more sense!

This looks pretty awesome, could we migrate to using this now?

adrientetar · 2014-03-07T11:43:50Z

I can do the Makefile changes and cleanups if you don't want to do it.

adrientetar · 2014-03-07T11:59:11Z

src/librustdoc/markdown.rs

+
+    let err = write!(
+        &mut out,
+        r#"<!doctype html>


nit: DOCTYPE should be uppercase (doesn't really matter)

huonw · 2014-03-07T12:15:40Z

Thanks for the offer, but I'll do the makefile stuff.

huonw · 2014-03-07T15:26:05Z

Pushed table of contents creation.

Generated docs: http://huonw.github.io/rust/build/doc/index.html (I haven't yet adjusted the stylesheet other than the syntax highlighting, so some edge cases look peculiar).

I met a snag with the makefiles/tutorials/guides: if we convert them into the format that sundown understands, we're moving away from what pandoc understands, and so we lose the ability to go directly to pdf and epub from Markdown.

(i'll address review and rebase later... going to sleep now.)

adrientetar · 2014-03-07T15:51:30Z

We could probably write a Python script that would transcript the files back to pandoc. What are the differences exactly?

alexcrichton · 2014-03-07T17:41:25Z

This continues to look awesome, nice work!

I wouldn't be too worried about losing the pdf/epub formats. I would think that a pdf could be generated by printing the HTML page (with an appropriate stylesheet)

huonw · 2014-03-07T21:50:55Z

We could probably write a Python script that would transcript the files back to pandoc. What are the differences exactly?

Basically just ~~~ {.lang .lang} (pandoc) vs. ~~~lang,lang (sundown), so yeah, should be relatively easy to script the change.

brson · 2014-03-08T02:23:28Z

I personally do not think a PDF tutorial and manual is that important, though I know others disagree. I would be ok with dropping them (perhaps temporarily) for the sake of progress, but also with the conversion script and retaining the optional pandoc dep.

adrientetar · 2014-03-08T06:25:10Z

@huonw Those are equivalent. In my rust-tuts repo I use ~~~~rust which works fine (works also when you paste snippets on GitHub for example).
@brson +1. (@alexcrichton we have a –relatively basic– print stylesheet modifier, this won't make results anywhere near LaTeX through)

huonw · 2014-03-08T07:18:59Z

@adrientetar It's not quite equivalent... it seems that sundown doesn't actually support multiple languages (so my ~~~lang,lang example above was wrong). I may experiment with just splitting on commas or something in sundown. (This is important because we have things like ~~~ {.ebnf .notation} and (should have) ~~~ {.c .notrust}.)

And, writing the multiple langauges/classes as ~~~lang,lang completely breaks pandocs rendering.

I'm editing the make files so that docs.mk is similar to crates.mk, so theoretically supporting a python script to change lang,lang to {.lang .lang} should be super easy.

(On the C note above: this change loses us the C syntax highlighting in the FFI guide.)

huonw · 2014-03-08T08:03:43Z

Also, btw, it seems sundown has some support for creating tables of contents; however I think the Rust version I've written here is better... at least; it's obvious how to use it, is more adjustable and gives the same output as pandoc (unlike the sundown one, afaict). I'm happy to try to use the C one if someone particularly hates the duplication, but ❤️ pure Rust from me.

huonw · 2014-03-08T10:13:46Z

So, anyway, I've been having the dumb a lot in this PR: I just discovered that sundown does support the {.foo .bar} syntax, so we can support pandoc at no cost.

/me writes "try thinking before typing" in big letters across his computer screen

adrientetar · 2014-03-08T13:04:48Z

Cool then. Maybe this notation is required when you have multiple class names.

huonw · 2014-03-08T13:13:39Z

Yeah, that's exactly it.

huonw · 2014-03-08T15:00:04Z

r?

Now switches HTML rendering over to rustdoc with a rather large makefile rewrite.

adrientetar · 2014-03-08T16:04:49Z

I think you forgot to change the code highlighting in src/doc/rust.css, sorry I did not catch that before.

adrientetar · 2014-03-08T21:46:45Z

mk/docs.mk

+# RUSTDOC_DEPS_xyz are extra dependencies for the rustdoc invocation
+# on xyz
+#
+# L10N_LANGS are the languages for which the crates have been


have been translated?

huonw · 2014-03-09T00:32:08Z

r?

alexcrichton · 2014-03-09T01:21:59Z

This looks fantastic, nice work!

r=me with minor things here and there addressed. The only big one which I think needs to be fixed is making sure RPATH_VAR is set when running rustdoc

huonw · 2014-03-09T01:57:19Z

Comments reviewed; but the last commit is debatable and worth a review itself...

This theoretically gives rustdoc the ability to render our guides, tutorial and manual (not in practice, since the files themselves need to be adjusted slightly to use Sundown-compatible functionality). Fixes rust-lang#11392.

markdown files. This means that # Foo ## Bar # Baz ### Qux ## Quz Gets a TOC like 1 Foo 1.1 Bar 2 Baz 2.0.1 Qux 2.1 Quz This functionality is only used when rendering a single markdown file, never on an individual module, although it could very feasibly be extended to allow modules to opt-in to a table of contents (std::fmt comes to mind).

This avoids having to include JS in the guide/tutorial/manual pages just to get the headers being links. The on-hover behaviour showing the little section marker § is preserved, because that gives a useful hint that the heading is a link.

The manual, tutorial and guides need the feature gates quite often, unfortunately, so this is the low-cost path to migrating to use rustdoc. This is only activated for pure-Markdown files. Preferably this would be avoided: rust-lang#12773

The changes are basically just because rustdoc runs tests/rendering on more snippets by default (i.e. everything without a `notrust` tag), and not anything significant.

This converts it to be very similar to crates.mk, with a single list of the documentation items creating all the necessary bits and pieces. Changes include: - rustdoc is used to render HTML & test standalone docs - documentation building now obeys NO_REBUILD=1 - testing standalone docs now obeys NO_REBUILD=1 - L10N is slightly less broken (in particular, it shares dependencies and code with the rest of the code) - PDFs can be built for all documentation items, not just tutorial and manual - removes the obsolete & unused extract-tests.py script - adjust the CSS for standalone docs to use the rustdoc syntax highlighting

E.g. this stops check-...-doc rules for `rustdoc.md` and `librustdoc` from stamping on each other, so that they are correctly built and tested. (Previously only the rustdoc crate was tested.)

This is meant to be compiling a crate, but the crate_id attribute seems to be upsetting it if the attribute is actually on the crate. I.e. this makes this test compile by putting the crate_id attribute on a function and so it's ignored. Such a hack. :(

parsing limitations. Sundown parses ``` ~~~ as a valid codeblock (i.e. mismatching delimiters), which made using rustdoc on its own documentation impossible (since it used nested codeblocks to demonstrate how testable codesnippets worked). This modifies those snippets so that they're delimited by indentation, but this then means they're tested by `rustdoc --test` & rendered as Rust code (because there's no way to add `notrust` to indentation-delimited code blocks). A comment is added to stop the compiler reading the text too closely, but this unfortunately has to be visible in the final docs, since that's the text on which the highlighting happens.

This restores the old behaviour (as compared to building PDF versions of all standalone docs), because some of the guides use unicode characters, which seems to make pdftex unhappy.

This gives Rustdoc the ability to render our guides, tutorial and manual and modifies the those documents (minor modifications) and makefiles to achieve this. See commits for more details (especially the makefile rewrite).

@alexcrichton

- remove `node.js` dep., it has no effect as of #12747 (1) - switch between LaTeX compilers, some cleanups - CSS: fixup the print stylesheet, refactor highlighting code (2) (1): `prep.js` outputs its own HTML directives, which `pandoc` cannot recognize when converting the document into LaTeX (this is why the PDF docs have never been highlighted as of now). Note that if we were to add the `.rust` class to snippets, we could probably use pandoc's native highlighting capatibilities i.e. Kate ([here is](http://adrientetar.github.io/rust-tuts/tutorial/tutorial.pdf) an example of that). (2): the only real highlighting change is for lifetimes which are now brown instead of red, the rest is just refactor of twos shades of red that look the same. Also I made numbers highlighting for src in rustdoc a tint more clear so that it is less bothering. @alexcrichton, @huonw Closes #9873. Closes #12788.

…r=jonas-schievink fix: Update 1.63 proc macro ABI to match rustc This updates us to the ABI used by rustc 1.63.0-beta.5, which will likely be the ABI of the next stable Rust release. It should also work on nightly (for now, but future changes won't be supported until the rustc version is bumped). cc rust-lang/rust-analyzer#12600

…xendoo Type safe CLI implementation for clippy-dev Use the derive feature of `clap` to generate CLI of clippy-dev. Adding new commands will be easier in the future and we get better compile time checking through exhaustive matching. --- I think I tested everything locally. But I would appreciate if the reviewer could go over it again, so that everything keeps working. changelog: none

alexcrichton reviewed Mar 7, 2014
View reviewed changes

adrientetar reviewed Mar 7, 2014
View reviewed changes

adrientetar reviewed Mar 8, 2014
View reviewed changes

huonw mentioned this pull request Mar 9, 2014

rustdoc should not automatically insert feature gates for testing code snippets in plain markdown files #12773

Closed

huonw mentioned this pull request Mar 9, 2014

Add an option to not run rustdoc blocks #12777

Merged

huonw added 8 commits March 9, 2014 19:29

rustdoc: run on plain Markdown files.

69b8ef8

This theoretically gives rustdoc the ability to render our guides, tutorial and manual (not in practice, since the files themselves need to be adjusted slightly to use Sundown-compatible functionality). Fixes rust-lang#11392.

rustdoc: hardcode each header as a link.

7a70ec1

This avoids having to include JS in the guide/tutorial/manual pages just to get the headers being links. The on-hover behaviour showing the little section marker § is preserved, because that gives a useful hint that the heading is a link.

docs: adjust code blocks to pass with rustdoc.

2d7d7e5

The changes are basically just because rustdoc runs tests/rendering on more snippets by default (i.e. everything without a `notrust` tag), and not anything significant.

mk: rename check-...-doc-<crate> to check-...-doc-crate-<crate>.

3c4ff1b

E.g. this stops check-...-doc rules for `rustdoc.md` and `librustdoc` from stamping on each other, so that they are correctly built and tested. (Previously only the rustdoc crate was tested.)

huonw added 2 commits March 9, 2014 20:59

mk: only build PDFs of the manual and tutorial.

b4815ad

This restores the old behaviour (as compared to building PDF versions of all standalone docs), because some of the guides use unicode characters, which seems to make pdftex unhappy.

bors closed this Mar 9, 2014

adrientetar mentioned this pull request Mar 9, 2014

doc: cleanups #12783

Merged

huonw deleted the rustdoc-markdown branch June 27, 2014 06:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustdoc: run on plain Markdown files & replace the use of pandoc for guides/tutorial/manual etc. #12747

rustdoc: run on plain Markdown files & replace the use of pandoc for guides/tutorial/manual etc. #12747

huonw commented Mar 7, 2014

huonw commented Mar 7, 2014

alexcrichton Mar 7, 2014

huonw Mar 7, 2014

alexcrichton Mar 7, 2014

alexcrichton commented Mar 7, 2014

huonw commented Mar 7, 2014

huonw commented Mar 7, 2014

alexcrichton Mar 7, 2014

alexcrichton commented Mar 7, 2014

adrientetar commented Mar 7, 2014

adrientetar Mar 7, 2014

huonw commented Mar 7, 2014

huonw commented Mar 7, 2014

adrientetar commented Mar 7, 2014

alexcrichton commented Mar 7, 2014

huonw commented Mar 7, 2014

brson commented Mar 8, 2014

adrientetar commented Mar 8, 2014

huonw commented Mar 8, 2014

huonw commented Mar 8, 2014

huonw commented Mar 8, 2014

adrientetar commented Mar 8, 2014

huonw commented Mar 8, 2014

huonw commented Mar 8, 2014

adrientetar commented Mar 8, 2014

adrientetar Mar 8, 2014

huonw commented Mar 9, 2014

alexcrichton commented Mar 9, 2014

huonw commented Mar 9, 2014

rustdoc: run on plain Markdown files & replace the use of pandoc for guides/tutorial/manual etc. #12747

rustdoc: run on plain Markdown files & replace the use of pandoc for guides/tutorial/manual etc. #12747

Conversation

huonw commented Mar 7, 2014

huonw commented Mar 7, 2014

alexcrichton Mar 7, 2014

Choose a reason for hiding this comment

huonw Mar 7, 2014

Choose a reason for hiding this comment

alexcrichton Mar 7, 2014

Choose a reason for hiding this comment

alexcrichton commented Mar 7, 2014

huonw commented Mar 7, 2014

huonw commented Mar 7, 2014

alexcrichton Mar 7, 2014

Choose a reason for hiding this comment

alexcrichton commented Mar 7, 2014

adrientetar commented Mar 7, 2014

adrientetar Mar 7, 2014

Choose a reason for hiding this comment

huonw commented Mar 7, 2014

huonw commented Mar 7, 2014

adrientetar commented Mar 7, 2014

alexcrichton commented Mar 7, 2014

huonw commented Mar 7, 2014

brson commented Mar 8, 2014

adrientetar commented Mar 8, 2014

huonw commented Mar 8, 2014

huonw commented Mar 8, 2014

huonw commented Mar 8, 2014

adrientetar commented Mar 8, 2014

huonw commented Mar 8, 2014

huonw commented Mar 8, 2014

adrientetar commented Mar 8, 2014

adrientetar Mar 8, 2014

Choose a reason for hiding this comment

huonw commented Mar 9, 2014

alexcrichton commented Mar 9, 2014

huonw commented Mar 9, 2014