Several markdown files concatted into one html file?

[dmca-glasgow]

Hello,

Firstly, thanks for the massive amount of work that has gone into all these packages! I've been working with Unified for about a week now and it's been a pleasure.

I've been tasked with creating a build tool which processes existing Markdown files into HTML. These Markdown files are sections which should be joined to form one HTML file. I've not found any documentation or packages accommodating this, and trying shoehorn it myself has caused a couple problems.

My plan was to parse and run each Markdown file individually (to report any warnings and errors with correct positional info), combine the syntax trees, then compile. However I've been struggling to find a pattern that works using the Unified API.

For example this works for one Markdown vfile:

import embedImages from 'remark-embed-images'; // <-- this is part of the problem

const processor = unified()
  .use(remarkParse)
  .use(embedImages)
  .use(myPluginA) 
  .use(myPluginB) 
  .use(myPluginC) 
  .use(remark2rehype)
  .use(rehypeDocument)
  .use(rehypeStringify);

processor.process(vfile, (err, file) => {
  report(err || file)
  console.log(String(file))
})

When I try to split it up a bit:

const parsed = processor.parse(vfile);
const transformed = await processor.run(parsed);
const compiled = processor.stringify(transformed);

1. I get the error:

(node:15709) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at validateString (internal/validators.js:120:11)
    at Object.resolve (path.js:980:7)
    at visitor (/.../node_modules/remark-embed-images/index.js:30:24)

file.dirname is undefined in remark-embed-images here, due to VFile info getting stripped by the parse step.

2. I'm not sure how I can extract vfile messages and report them using this approach.

One solution that does work is to process the files individually (without using rehypeDocument), concat the html strings, then reparse using unified().use(rehypeParse).use(doc).process() but I'm hoping to avoid compiling and reparsing.

Am I barking up the wrong tree? Any pointers on how to do this would be very helpful!

Cheers.

[wooorm]

you’re close! you can pass the vfile to parse, but also to run (and stringify), as the second parameter!

But indeed, unified is a pipeline for a single file, so here we’d need two:

const firstProcessor = unified()
  .use(remarkParse)
  .use(embedImages)
  // ..
  .use(remark2rehype)

// .parse and .run all markdown files, to get `hasts` (an array of HTML ASTs).

// Make that into a single HTML document
var hast = {type: 'root', children: hasts.flatMap(d => d.children)}

const secondProcessor = unified()
  // Use rehhype plugins that run on the whole document
  // ..
  .use(rehypeStringify)

// .run and .stringify the single `hast`. (or only stringify if there are no rehype plugins which should run on the whole)

[wooorm]

And thanks for the kind words, David!

[dmca-glasgow]

Thank you, that was the missing link!

I've come across another anomaly I don't quite understand...

With this code I get 2 lint warnings:

import styleGuide from 'remark-preset-lint-markdown-style-guide';

const processor1 = unified()
  .use(markdown)
  .use(styleGuide);

const parsed = processor1.parse(vfile);
const transformed = await processor1.run(parsed, vfile);
console.log(vfile.messages);

[
  [../fixture/week1/test.Rmd:1:1: Incorrect extension: use `md`] {
    reason: 'Incorrect extension: use `md`',
    line: 1,
    column: 1,
    location: { start: [Object], end: [Object] },
    source: 'remark-lint',
    ruleId: 'file-extension',
    file: '../fixture/week1/test.Rmd',
    fatal: false
  },
  [../fixture/week1/test.Rmd:10:165: Line must be at most 80 characters] {
    reason: 'Line must be at most 80 characters',
    line: 10,
    column: 165,
    location: { start: [Object], end: [Object] },
    source: 'remark-lint',
    ruleId: 'maximum-line-length',
    file: '../fixture/week1/test.Rmd',
    fatal: false
  }
]

But when remark-rehype is added to the chain I only get one:

const processor1 = unified()
  .use(markdown)
  .use(styleGuide)
  .use(remark2rehype);
console.log(vfile.messages);

[
  [../fixture/week1/test.Rmd:1:1: Incorrect extension: use `md`] {
    reason: 'Incorrect extension: use `md`',
    line: 1,
    column: 1,
    location: { start: [Object], end: [Object] },
    source: 'remark-lint',
    ruleId: 'file-extension',
    file: '../fixture/week1/test.Rmd',
    fatal: false
  }
]

Is this a bug, or is it due to the tree now being different so the lint rule no longer applies?

Thanks.

[wooorm]

Hmm, this is complex. remark-lint, the actual package (https://github.com/remarkjs/remark-lint/blob/main/packages/remark-lint/index.js), needs to run after all plugins. What it does is ignore some messages. And it needs to run last because for example a table of contents is generated by a different plugin, and it shouldn’t complain about that.

This can only be nicely done by running .use() with a different plugin inside it, so that it will always run last.

So that last plugin is running on the hast, which seems to be causing the line length warning to go away (the file extenion is unrelated to the contents of a file, so it doesn’t care about a different AST)

It’s interesting, I (or anyone else apparently) really combined remark-lint with going to rehype.

[dmca-glasgow]

It was intuitive that linting is run last given the output above, so I've created a remark processor and a rehype processor. I can report after the remark processor.

This separation of concerns has turned out to be an advantage in my project anyway - I can build a useful tree with processor1, work on it, then compile with processor2.

Thanks for all the help!

[wooorm]

Perfect! Glad that works, because I’d be unsure how to solve it otherwise 😅

All the best!