use markdown for the documentation

[gokcehan]

cc #1018

This PR changes the documentation to use markdown instead of go doc.

---

Online markdown

Man page conversion (to be lf.1):

curl -sSf https://raw.githubusercontent.com/gokcehan/lf/doc-md/doc.1 | man -l -

Plain text conversion (to be embedded in the binary):

curl -sSf https://raw.githubusercontent.com/gokcehan/lf/doc-md/doc.txt | less

Rendering the markdown with glow (optional):

curl -sSf https://raw.githubusercontent.com/gokcehan/lf/doc-md/doc.md | glow -p

or just:

glow -p https://raw.githubusercontent.com/gokcehan/lf/doc-md/doc.md

---

Our current documentation system has become a pain point and I have been thinking of renovating it for a while. I want to make some mental notes here to help with the decision for a replacement documentation system. Feel free to join the discussion.

Our current documentation system has the following advantages/disadvantages:

1. (Advantage) The documentation is available as a man page.
2. (Disadvantage) gen/man.sh is a hack and it has quirks.
3. (Disadvantage) gen/man.sh does not work in Windows.
4. (Advantage) Users don't need to install anything specific for go generate besides a POSIX shell.
5. (Disadvantage) It is a little unusual and confusing for contributors to run go generate after a doc update.
6. (Advantage) The documentation is automatically available online.
7. (Disadvantage) pkg.go.dev is mostly aimed for libraries and it is unsuitable for programs like lf.
8. (Disadvantage) The online documentation is not available for different versions (requires adopting the semantic versioning).
9. (Disadvantage) Go doc syntax is limiting and it is unsuitable for our documentation.
10. (Disadvantage) Go doc syntax is not commonly known by contributors.
11. (Advantage) The documentation is available in the binary.
12. (Disadvantage) lf -doc is not easily readable in a terminal.

An alternative documentation system should ideally keep all or most of the advantages while also getting rid of some of the disadvantages. I'm currently thinking of simply switching to markdown for the documentation. Some notes below:

1. There are converters (e.g. pandoc) which can convert markdown to man page.
2. Fixed.
3. Fixed.
4. It requires installing a markdown converter. However, we can instead consider generating the documentation only before our releases and avoid asking contributors to run go generate. I can then try to generate the documentation before a release myself, but ideally we could also consider setting up an automatic workflow for that (e.g. pandoc-action-example might work, not sure.)
5. Fixed, see above point.
6. Github automatically renders markdown.
7. Fixed.
8. Fixed, as it should be possible to have links to specific versions of the documentation in Github.
9. Fixed.
10. Fixed.
11. We can still embed the markdown documentation in the binary.
12. Fixed, by being able to optionally use terminal markdown renderers in a custom doc command (e.g. glow).

I'm currently in favor of using GFM to provide us enough features to work with. This is an experimental draft patch today to see what we can do about improving the current content in the documentation.

---

TODO

• Try the man page conversion of pandoc
• Try the plain text conversion of pandoc
• Embed the plain text conversion in the binary
• Try to set a workflow for automatic pandoc to man page and plain text conversions (not sure if it is possible to commit artifacts to the repo)
• Compare alternatives with pandoc

[joelim-work]

I think this is a great idea! The idea of using https://pkg.go.dev/ to host documentation has always seemed strange to me given that lf isn't a library.

A few points of discussion from me:

• From what I have read online, the way in which pandoc converts from Markdown to man pages is somewhat primitive (e.g. H1 headings map directly to .SH macros), so an idiomatic Markdown file may not convert to an idiomatic man page. You may have to format the Markdown file in such a way that it converts neatly to a man page.
• I'm not sure if it's a good idea to have lf -doc output raw Markdown, it forces users to either put up with the additional syntax, or install a renderer like glow. I think it might be better to have pandoc convert the Markdown documentation to plain text form and use that for lf -doc.
• For advanced users who do want to view the Markdown documentation using glow, I guess it would be possible to write a script that downloads (and optionally caches) the file from GitHub (perhaps from the Releases section), and then pipes the content into glow.

Otherwise I don't really have much more to add - this is a welcome change for both contributors and maintainers.

[ilyagr]

There are several projects that have Markdown-like syntax that is designed to be converted to a man page. One example is https://github.com/rtomayko/ronn. I like it, except for the fact that it requires installing Ruby. (Update: Also, it seems to be minimally maintained) There are others, I will add them to the thread if/when I remember what they are.

Ranger uses https://perldoc.perl.org/pod2man which is Perl-y and not Markdown.

OTOH, pandoc is great, and we could compromise on how canonical the man pages are to get its other benefits. There's a template (not sure how perfect) at https://eddieantonio.ca/blog/2015/12/18/authoring-manpages-in-markdown-with-pandoc/.

[ilyagr]

Here's the other alternative I encountered that seemed interesting. It comes from the suckless world, which might be a good philosophy for this purpose. I'll update this if I remember which project I stumbled on used .scd files.

https://drewdevault.com/2018/05/13/scdoc.html
https://manned.org/scdoc.5
https://github.com/ddevault/scdoc/blob/master/scdoc.5.scd (might be old)

[gokcehan]

@joelim-work I tried changing our markdown to be more idiomatic as a man page. I also tried using the plain text output of pandoc for the embedding, which seems to work okay for this purpose I think. Rendering with glow seems possible as you suggested (examples added to the PR description).

@ilyagr I have looked at the alternatives you mentioned. For some reason, ronn seems to crash with a stack trace on my machine (or I just don't know how to use it properly). I also think it would be better to use pandoc instead to have explicit support for GFM over markdown. For example, tables might be useful for our configuration files section and elsewhere. pod2man and scdoc seems okay, but I think it would be better to use markdown as it should be more commonly known by contributors (point 10 in the description). My initial impression is that these tools are also mostly aimed to provide first class support for man pages. In our case, I think the online documentation should be considered the first class, followed by the man page and the plain text embed, since the man page is not availabe on Windows. I'm also personally excited about the opportunities to have proper "see also" notes throughout the online version of the documentation. The documentation is long enough such that I don't expect many people to read it from start to finish these days. For now, the only changes that I made to support the man page is to add the first three sections in the beginning and capitalizing the section names. If these changes are not desired, we can also try to do these in gen/doc.sh instead as before.

Lastly, I looked at the pandoc workflow, but I'm not sure if it is possible to add generated artifacts to the repository. For now, I think I will be generating the documentation manually on my machine before new releases.

I'm marking the PR as ready for review. Now that we are using markdown, there are many changes that can be done, but I think they can be done separately afterwards. I'm sure you and others have better ideas than me to make the documentation more useful and pretty.

[joelim-work]

Overall the PR looks good to me. I agree with leaving formatting improvements for future changes.

I think you might have forgotten to remove lf.1 and doc.go, as they are part of the old documentation method. I'm not sure if there's any benefit in keeping them? In addition, you might want to update the contributing guidelines, and also change all the documentation links (https://pkg.go.dev/) to point to the markdown file instead.

The only other concern I have is the idea of keeping automatically generated files (i.e. the man page and text forms) in source control. This is actually a slightly controversial topic, though I think it's fine if you want to manually generate and commit the files before each release. I considered not storing them in the repo, and instead only generating them during the release.yml workflow pipeline, but this also means that the documentation won't be available when building from source.

[gokcehan]

@joelim-work Sorry for the mess as I wasn't paying much attention. I think I have cleaned up a little bit. Note, lf.1 is now replaced with the new content so it remains accessible as man lf.

I'm also not too satisfied with the generated files in general. Not storing them in the repo sounds interesting. I will think about it when I find some time.

[joelim-work]

@gokcehan I suspect you might have to keep doc.txt in the repo, otherwise building from source using go install won't work. I don't know how necessary it is to keep doc.txt in sync with the source markdown at all times, but if you only care about this when creating a release, then I guess it might be appropriate to configure release.yml to generate the files and commit them (or send a pull request).

For lf.1, if you choose to remove it from the repo, it won't be part of the source code archives when creating a release, but you should be able to automatically generate it and add it as part of the release.

Anyway I think the changes here are fine for now, and already an improvement over the current implementation. I will approve the PR.

[gokcehan]

@joelim-work I have given this some more thoughts today, but I still couldn't make up my mind. There are two complications that prevents me to remove the generated files:

First, we need a doc.txt file for go install as you already mentioned. We could keep a dummy doc.txt file to avoid breaking the build, but then go install will neither have the doc nor the man page. We could arrange the workflow to send an automatic PR as you suggest (if it is possible), but then it is not too different than manually generating the files before the release, which will also include the latest generated file in the source code archive. And if we generate doc.txt already, we might as well generate lf.1 along the way.

Second, any such change in the repo structure will technically be a breaking change for package maintainers. So if we decide to go down this route, it might be a good idea to think about a deprecation strategy along with it, or otherwise most packages will be broken in the next release.

On the other side, it is not too difficult to generate the files before the release. The only difficulty is that we may forget to do so, but hopefully not. It might be useful to add a gen/release.sh file to the repo for this purpose, that can be used by any collaborator even if I'm not around.

All in all, I think I will just merge this patch for now, but feel free to continue the discussion here or elsewhere for a possible change in a separate PR.