From 9a3d04e941ea84d09c1c6a7890f1d9c5d6d014c1 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Thu, 21 Feb 2019 10:50:47 -0500 Subject: [PATCH 1/8] Asciidoctor: Support for "open in widget" This'll take a while to describe properly. Let's start with the goal first, talk about the AsciiDoc implementation, then move on to the Asciidoctor implementation, then talk about why we need a compatibility layer, then describe the compatibity layer, and finally round out this book of a commit message by describing some of the more esoteric usages of this that this change currently supports and our plan for dropping this support in the future. == The goal Certain snippets in Elastic's docs are special and we'd like to decorate them with buttons. These buttons allow opening the snippets in developer tools or transforming them into `cURL` commands. I'm calling all of the stuff that makes these buttons an "open in widget". == AsciiDoc implementation Because of AsciiDoc's fairly limited ability to customize it we triggered these snippets by adding special comments after the code blocks that we'd like to decorate that look like this: ``` [source,js] ---- GET / ---- // CONSOLE <----- This is the comment ``` We customize AsciiDoc to emit all comments as `` elements in the generated xml and then use custom xslt to recognize remarks that look like `CONSOLE` to trigger the widget. These buttons need the contents of the snippet, `GET /` in the example above, to be accessible in a file. We implement this by post-processing the generated html in the perl code to extra these snippets to files, rewriting the html that generates the widget to point to the file. == Asciidoctor implementation Asciidoctor feels strongly that comments shouldn't have semantic meaning. I'm on board with that. So, to trigger the widget in Asciidoctor you use: ``` [source,console] ---- GET / ---- ``` Note that the language, which was `js` in AsciiDoc is now `console`. This language is what triggers the widget. This feels good to me because all snippets that are in the "console" language really do want this decoration. And because the snippets really *aren't* javascript. They just often have json in them. We recognize these `:listing` blocks using a treepreprocessor and built the snippet file when processing the Asciidoctor AST. We also rewrite the docbook that is generated by these blocks to contain a link to the extract snippet. Finally, we also use custom xslt to extract that link and render the widget. The xslt is less hairy than the one used to find the ``s. == Why we need compatibility It'd be fairly simple to change `// CONSOLE` to `[source,console]` on a page by page basis, but we have thousands of uses of `// CONSOLE` across dozens of books and dozens of branches. In addition, AsciiDoc doesn't understand `[source,console]` which'd make changing these a thing that you'd have to do at the same time as you switched the book from AsciiDoc to Asciidoctor. Beyond *that*, Elasticsearch automatically extracts snippets with the `// CONSOLE` comment and turns them into tests. There are just too many moving parts for a hard cut over from `// CONSOLE` to `[source,console]`. So I built a compatiblity layer in Asciidoctor == How the compatibilty layer works We use two customizations in Asciidoctor to make the compatibility layer go, one that recognizes comments shaped like `// CONSOLE` and turns them into a literal `// CONSOLE` using the `pass` macro like so: `pass:[// CONSOLE]`. The next one picks up source blocks that are immediately followed by `pass:[// CONSOLE]` and switches the language to `console`. Something like this *begs* for away to tell the user "you have 1232 warnings that you have to fix". Like linting but for your docs. I'll be thinking more about this in the coming weeks. But for now there is not warning when you use the compatibility layer. == Esoteric forms of the "open in widget" Kibana's docs have snippets like: ``` [source,js] -------------------------------------------------- GET api/logstash/pipeline/hello-world -------------------------------------------------- // KIBANA ``` Note the `// KIBANA`. This is *like* `// CONSOLE` but it is for snippets that should be sent to Kibana's API instead of Elasticsearch's. It is debateable if `kibana` is really a different language than `console`, but it is at least a different dialect. Either way, Asciidoctor triggers the "open in widget" for these snippets with `[source,kibana]`. Older versions of the documentation have snippets like this: ``` [source,js] -------------------------------------------------- GET / -------------------------------------------------- // AUTOSENSE ``` These open the command in an older tool named "sense" instead of the developer console. The developer console "grew out of" sense but hasn't been called that for a long time. In any case, it exists for compatibility with old books. The Definitive Guide has snippets that look like: ``` [source,js] -------------------------------------------------- GET / -------------------------------------------------- // SENSE:a/path/to/some.json ``` These are snippets that render as `GET /` but when you open them in "sense" they have the contents of `a/path/to/some.json` which is supposed to be similar. This isn't widely used and I personally think it is super confusing, at least the way that we've implemented it now. So I log a warning whenever you try to do this which should prevent folks from doing it accidentally. == Follow ups This work begs for at least two follow up changes: 1. A warnings management system so old books can continue to use the backwards compatibility features but new books will be forced to use native Asciidoctor features. 2. More in depth integration testing. Right now we build README.asciidoc with AsciiDoc and Asciidoctor which is great, but it is difficult to assert that the results are "close enough" with the tools that we have. We can do better and this change makes it obvious that we must do better. --- Makefile | 14 +- README.asciidoc | 72 ++++---- lib/ES/Template.pm | 3 +- lib/ES/Util.pm | 8 +- .../elastic_compat_preprocessor/extension.rb | 26 +++ .../extension.rb | 62 ++++++- resources/asciidoctor/lib/extensions.rb | 2 + .../lib/open_in_widget/extension.rb | 109 ++++++++++++ .../spec/elastic_compat_preprocessor_spec.rb | 67 ++++++- .../elastic_compat_tree_processor_spec.rb | 24 +++ .../asciidoctor/spec/open_in_widget_spec.rb | 167 ++++++++++++++++++ .../asciidoctor/spec/snippets/snippet.console | 1 + .../asciidoctor/spec/snippets/snippet.kibana | 1 + .../asciidoctor/spec/snippets/snippet.sense | 1 + resources/website_common.xsl | 16 +- 15 files changed, 518 insertions(+), 55 deletions(-) create mode 100644 resources/asciidoctor/lib/open_in_widget/extension.rb create mode 100644 resources/asciidoctor/spec/open_in_widget_spec.rb create mode 100644 resources/asciidoctor/spec/snippets/snippet.console create mode 100644 resources/asciidoctor/spec/snippets/snippet.kibana create mode 100644 resources/asciidoctor/spec/snippets/snippet.sense diff --git a/Makefile b/Makefile index d28f8176e04c8..cb33df69c88c8 100644 --- a/Makefile +++ b/Makefile @@ -38,12 +38,20 @@ expected_files_check: /tmp/readme_asciidoc .PHONY: same_files_check same_files_check: /tmp/readme_asciidoc /tmp/readme_asciidoctor - # The `grep -v snippets` is a known issue to be resolved "soon" + # The `grep -v`s are a known issue to be resolved "soon" diff \ <(cd /tmp/readme_asciidoc && find * -type f | sort \ - | grep -v snippets/blocks \ + | grep -v snippets/ \ ) \ - <(cd /tmp/readme_asciidoctor && find * -type f | sort) + <(cd /tmp/readme_asciidoctor && find * -type f | sort \ + | grep -v snippets/ \ + ) + # Temporary assertions about the `grep -v`ed files that we'll + # generalize in a followup change. Asciidoc doesn't trim trailing + # newlines but AsciiDoctor does. + diff \ + <(sed 's/[[:space:]]*$$//' /tmp/readme_asciidoc/snippets/blocks/1.json) \ + /tmp/readme_asciidoctor/snippets/1.console /tmp/readme_asciidoc: ./build_docs.pl --in_standard_docker \ diff --git a/README.asciidoc b/README.asciidoc index 7c73c5b08bd7f..b8303b57893ae 100644 --- a/README.asciidoc +++ b/README.asciidoc @@ -1005,12 +1005,26 @@ footnote to a particular line of code: === View in Console Code blocks can be followed by a "View in Console" link which, when clicked, -will open the code snippet in Console. The snippet can either be taken directly -from the code block (`CONSOLE`), or be a link to a custom snippet. +will open the code snippet in Console. There are two ways to do this, the +"AsciiDoc" way and the "Asciidoctor" way. The "AsciiDoc" way is preferred in +the Elaticsearch repository because it can recognize it to make tests. The +"Asciidoctor" way is preferred in other books, but only if they are built with +"Asciidoctor". Try it first and if it works then use it. Otherwise, use the +"AsciiDoct" way. -.Code block with CONSOLE link +//// + +The tricks that we pull to make ascidoctor support // CONSOLE are windy +and force us to add subs=+macros when we render an asciidoc snippet. +We *don't* require that for normal snippets, just those that contain +asciidoc. + +//// + +.Code block with CONSOLE link (AsciiDoc way) ================================== -[source,asciidoc] +ifdef::asciidoctor[[source,asciidoc,subs=+macros]] +ifndef::asciidoctor[[source,asciidoc]] -- [source,js] ---------------------------------- @@ -1026,6 +1040,24 @@ GET /_search ================================== <1> The `// CONSOLE` line must follow immediately after the code block, before any callouts. +.Code block with CONSOLE link (Asciidoctor way) +================================== +[source,asciidoc] +-- +[source,console] +---------------------------------- +GET /_search +{ + "query": "foo bar" \<1> +} +---------------------------------- + +\<1> Here's the explanation +-- +================================== + +Both render as: + [source,js] ---------------------------------- GET /_search @@ -1051,38 +1083,6 @@ The local web browser can be stopped with `Ctrl-C`. ================================ -==== Custom Console snippets - -Sometimes you will want to show a small amount of code in the code block, but -to provide a full recreation in the Console snippet. In this case, you need to: - -* Save the snippet file in the `./snippets/` directory in the root docs directory. -* Under the code block, specify the name of the snippet file with -+ - // CONSOLE: path/to/snippet.json - -For instance, to add a custom snippet to the file `./one/two/three.asciidoc`, save the snippet -to `./snippets/one/two/three/example_1.json`, then add the `CONSOLE` link below the code block: - -.Code block with custom CONSOLE link -================================== -[source,asciidoc] --- -[source,js] ----------------------------------- -GET /_search -{ - "query": "foo bar" \<1> -} ----------------------------------- -// CONSOLE:one/two/three/example_1.json <1> - -\<1> Here's the explanation --- -<1> The path should not contain the initial `snippets` directory -================================== - - [[admon-blocks]] === Admonition blocks diff --git a/lib/ES/Template.pm b/lib/ES/Template.pm index 5343b860b5f59..2748da041b37e 100644 --- a/lib/ES/Template.pm +++ b/lib/ES/Template.pm @@ -47,6 +47,7 @@ sub apply { my $self = shift; my $dir = shift; my $lang = shift || die "No lang specified"; + my $asciidoctor = shift; my $map = $self->_map; @@ -61,7 +62,7 @@ sub apply { $contents =~ s/\s*$/\n/; # Extract AUTOSENSE snippets - $contents = $self->_autosense_snippets( $file, $contents ); + $contents = $self->_autosense_snippets( $file, $contents ) unless $asciidoctor; # Fill in template my @parts = @{ $self->_parts }; diff --git a/lib/ES/Util.pm b/lib/ES/Util.pm index 426bd4ef5a3ed..ba652da6c02bd 100644 --- a/lib/ES/Util.pm +++ b/lib/ES/Util.pm @@ -153,7 +153,7 @@ sub build_chunked { my ($chunk_dir) = grep { -d and /\.chunked$/ } $dest->children or die "Couldn't find chunk dir in <$dest>"; - finish_build( $index->parent, $chunk_dir, $lang ); + finish_build( $index->parent, $chunk_dir, $lang, $asciidoctor ); extract_toc_from_index($chunk_dir); for ( $chunk_dir->children ) { run( 'mv', $_, $dest ); @@ -285,7 +285,7 @@ sub build_single { or die "Couldn't rename <$src> to : $!"; } - finish_build( $index->parent, $dest, $lang ); + finish_build( $index->parent, $dest, $lang, $asciidoctor ); } #=================================== @@ -367,10 +367,10 @@ sub build_pdf { #=================================== sub finish_build { #=================================== - my ( $source, $dest, $lang ) = @_; + my ( $source, $dest, $lang, $asciidoctor ) = @_; # Apply template to HTML files - $Opts->{template}->apply( $dest, $lang ); + $Opts->{template}->apply( $dest, $lang, $asciidoctor ); my $snippets_dest = $dest->subdir('snippets'); my $snippets_src; diff --git a/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb b/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb index b991745b71dc2..406cc9df7ee24 100644 --- a/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb +++ b/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb @@ -89,6 +89,24 @@ # Because Asciidoc permits these mismatches but asciidoctor does not. We'll # emit a warning because, permitted or not, they are bad style. # +# With the help of ElasticCompatTreeProcessor turns +# [source,js] +# ---- +# foo +# ---- +# // CONSOLE +# +# Into +# [source,console] +# ---- +# foo +# ---- +# Because Elastic has thousands of these constructs but Asciidoctor feels +# strongly that comments should not convey meaning. This is a totally +# reasonable stance and we should migrate away from these comments in new +# docs when it is possible. But for now we have to support the comments as +# well. +# class ElasticCompatPreprocessor < Asciidoctor::Extensions::Preprocessor include Asciidoctor::Logging @@ -140,6 +158,7 @@ def reader.process_line(line) @code_block_start = line end end + supported = 'added|coming|deprecated' # First convert the "block" version of these macros. We convert them # to block macros because they are at the start of the line.... @@ -147,6 +166,13 @@ def reader.process_line(line) # Then convert the "inline" version of these macros. We convert them # to inline macros because they are *not* at the start of the line.... line&.gsub!(/(#{supported})\[([^\]]*)\]/, '\1:[\2]') + # Transform Elastic's traditional comment based marking for + # AUTOSENSE/KIBANA/CONSOLE snippets into a marker that we can pick + # up during tree processing to turn the snippet into a marked up + # CONSOLE snippet. Asciidoctor really doesn't recommend this sort of + # thing but we have thousands of them and it'll take us some time to + # stop doing it. + line&.gsub!(%r{//\s*(?:AUTOSENSE|KIBANA|CONSOLE|SENSE:[^\n<]+)}, 'pass:[\0]') end end reader diff --git a/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb b/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb index 5cc4785426836..fb651cec75c59 100644 --- a/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb +++ b/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb @@ -23,14 +23,62 @@ # <1> The count of categories that were matched # <2> The categories retrieved # +# Turns +# [source,js] +# -------------------------------------------------- +# GET / <1> +# -------------------------------------------------- +# pass:[// CONSOLE] +# <1> The count of categories that were matched +# <2> The categories retrieved +# +# Into +# [source,console] +# -------------------------------------------------- +# GET / <1> +# -------------------------------------------------- +# <1> The count of categories that were matched +# <2> The categories retrieved +# class ElasticCompatTreeProcessor < TreeProcessorScaffold + include Asciidoctor::Logging + def process_block(block) - if block.context == :listing && block.style == "source" && - block.subs.include?(:specialcharacters) == false - # callouts have to come *after* special characters - had_callouts = block.subs.delete(:callouts) - block.subs << :specialcharacters - block.subs << :callouts if had_callouts - end + return unless block.context == :listing && block.style == 'source' + + process_subs block + process_lang_override block + end + + def process_subs(block) + return if block.subs.include? :specialcharacters + + # callouts have to come *after* special characters + had_callouts = block.subs.delete(:callouts) + block.subs << :specialcharacters + block.subs << :callouts if had_callouts + end + + LANG_MAPPING = { + 'AUTOSENSE' => 'sense', + 'CONSOLE' => 'console', + 'KIBANA' => 'kibana', + 'SENSE' => 'sense', + } + + def process_lang_override(block) + next_block = block.next_adjacent_block + return unless next_block && next_block.context == :paragraph + return unless next_block.source =~ %r{pass:\[//\s*([^:\]]+)(?::\s*([^\]]+))?\]} + + lang = LANG_MAPPING[$1] + snippet = $2 + return unless lang # Not a language we handle + + block.set_attr 'language', lang + block.set_attr 'snippet', snippet + + block.parent.blocks.delete next_block + block.parent.reindex_sections end end diff --git a/resources/asciidoctor/lib/extensions.rb b/resources/asciidoctor/lib/extensions.rb index 14535ada322de..cce7b3c2c5c50 100644 --- a/resources/asciidoctor/lib/extensions.rb +++ b/resources/asciidoctor/lib/extensions.rb @@ -7,6 +7,7 @@ require_relative 'elastic_compat_tree_processor/extension' require_relative 'elastic_compat_preprocessor/extension' require_relative 'elastic_include_tagged/extension' +require_relative 'open_in_widget/extension' Asciidoctor::Extensions.register ChangeAdmonition Asciidoctor::Extensions.register do @@ -18,5 +19,6 @@ treeprocessor CopyImages::CopyImages treeprocessor EditMe treeprocessor ElasticCompatTreeProcessor + treeprocessor OpenInWidget include_processor ElasticIncludeTagged end diff --git a/resources/asciidoctor/lib/open_in_widget/extension.rb b/resources/asciidoctor/lib/open_in_widget/extension.rb new file mode 100644 index 0000000000000..4dd93261b1b5e --- /dev/null +++ b/resources/asciidoctor/lib/open_in_widget/extension.rb @@ -0,0 +1,109 @@ +# frozen_string_literal: true + +require 'fileutils' + +require_relative '../scaffold.rb' + +## +# Extensions for enriching certain source blocks with "OPEN IN CONSOLE", +# "OPEN IN SENSE", "OPEN IN KIBANA", AND/OR "COPY_AS_CURL". +# +# Usage +# +# [source,console] +# --------- +# GET / +# --------- +# +# or +# +# [source,sense] +# --------- +# GET / +# --------- +# +# or +# +# [source,kibana] +# --------- +# GET / +# --------- +# +# or +# +# [source,sense,snippet=path/to/snippet.console] +# --------- +# GET / +# --------- +# +class OpenInWidget < TreeProcessorScaffold + include Asciidoctor::Logging + + CALLOUT_SCAN_RX = / ?#{Asciidoctor::CalloutScanRx}/ + + def process_block(block) + return unless block.context == :listing && block.style == 'source' + + lang = block.attr 'language' + return unless %w[console sense kibana].include? lang + + snippet = block.attr 'snippet' + if snippet + logger.warn message_with_context "reading snippets from a path makes the book harder to read", :source_location => block.source_location + # If you specify the snippet path then we should copy it into the + # destination directory so it is available for Kibana. + snippet_path = "snippets/#{snippet}" + normalized = block.normalize_system_path(snippet_path, block.document.base_dir) + if File.readable? normalized + copy_snippet block, normalized, snippet_path + else + logger.warn message_with_context "can't read snippet from #{normalized}", :source_location => block.source_location + end + else + # If you don't specify the snippet then we assign it a number and read + # if from the contents of the source listing, copying it to the + # destination directory so it is available for Kibana. + snippet_number = block.document.attr 'snippet_number', 1 + snippet = "#{snippet_number}.#{lang}" + block.document.set_attr 'snippet_number', snippet_number + 1 + + snippet_path = "snippets/#{snippet}" + source = block.source.gsub(CALLOUT_SCAN_RX, '') + "\n" + write_snippet block, source, snippet_path + end + block.set_attr 'snippet_link', "" + block.document.register :links, snippet_path + + def block.content + @attributes['snippet_link'] + super + end + end + + def copy_snippet(block, source, uri) + logger.info message_with_context "copying snippet #{source}", :source_location => block.source_location + copy_proc = block.document.attr 'copy_snippet' + if copy_proc + # Delegate to a proc for copying if one is defined. Used for testing. + copy_proc.call(uri, source) + else + destination = ::File.join block.document.options[:to_dir], uri + destination_dir = ::File.dirname destination + FileUtils.mkdir_p destination_dir + FileUtils.cp source, destination + end + end + + def write_snippet(block, snippet, uri) + logger.info message_with_context "writing snippet #{uri}", :source_location => block.source_location + write_proc = block.document.attr 'write_snippet' + if write_proc + # Delegate to a proc for copying if one is defined. Used for testing. + write_proc.call(uri, snippet) + else + destination = ::File.join block.document.options[:to_dir], uri + destination_dir = ::File.dirname destination + FileUtils.mkdir_p destination_dir + File.open(destination, 'w') { |file| file.write(snippet) } + end + end +end diff --git a/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb b/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb index e246261978824..f7812e77869a3 100644 --- a/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb +++ b/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb @@ -2,7 +2,9 @@ require 'change_admonition/extension' require 'elastic_compat_preprocessor/extension' +require 'elastic_compat_tree_processor/extension' require 'elastic_include_tagged/extension' +require 'open_in_widget/extension' require 'shared_examples/does_not_break_line_numbers' RSpec.describe ElasticCompatPreprocessor do @@ -11,6 +13,8 @@ Asciidoctor::Extensions.register do preprocessor ElasticCompatPreprocessor include_processor ElasticIncludeTagged + treeprocessor ElasticCompatTreeProcessor + treeprocessor OpenInWidget end end @@ -18,6 +22,8 @@ Asciidoctor::Extensions.unregister_all end + spec_dir = File.dirname(__FILE__) + include_examples "doesn't break line numbers" [ @@ -193,9 +199,9 @@ Example wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip.sha512 - shasum -a 512 -c elasticsearch-{version}.zip.sha512 <1> + shasum -a 512 -c elasticsearch-{version}.zip.sha512 unzip elasticsearch-{version}.zip - cd elasticsearch-{version}/ <2> + cd elasticsearch-{version}/ Compares the SHA of the downloaded .zip archive and the published checksum, which should output @@ -249,4 +255,61 @@ DOCBOOK expect(actual).to eq(expected.strip) end + + def stub_file_opts + return { + 'copy_snippet' => proc { |uri, source| }, + 'write_snippet' => proc { |uri, source| }, + } + end + + [ + %w[CONSOLE console], + %w[AUTOSENSE sense], + %w[KIBANA kibana], + ].each do |name, lang| + it "transforms #{name} comments into a listing with the #{lang} language" do + input = <<~ASCIIDOC + == Example + [source,js] + ---- + foo + ---- + // #{name} + ASCIIDOC + expected = <<~DOCBOOK + + Example + foo + + DOCBOOK + actual = convert input, stub_file_opts, eq( + "INFO: : line 3: writing snippet snippets/1.#{lang}" + ) + expect(actual).to eq(expected.strip) + end + end + + it "transforms SENSE comments into a listing with the SENSE language and a path" do + input = <<~ASCIIDOC + == Example + [source,js] + ---- + foo + ---- + // SENSE: snippet.sense + ASCIIDOC + expected = <<~DOCBOOK + + Example + foo + + DOCBOOK + warnings = <<~WARNINGS + WARN: : line 3: reading snippets from a path makes the book harder to read + INFO: : line 3: copying snippet #{spec_dir}/snippets/snippet.sense + WARNINGS + actual = convert input, stub_file_opts, eq(warnings.strip) + expect(actual).to eq(expected.strip) +end end diff --git a/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb b/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb index adf30c136cb24..9eec2101f4bbd 100644 --- a/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb +++ b/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb @@ -71,4 +71,28 @@ DOCBOOK expect(actual).to eq(expected.strip) end + + [ + %w[CONSOLE console], + %w[AUTOSENSE sense], + %w[KIBANA kibana], + ].each do |name, lang| + it "transforms legacy // #{name} commands into the #{lang} language" do + actual = convert <<~ASCIIDOC + == Example + [source,js] + ---- + GET / + ---- + pass:[// #{name}] + ASCIIDOC + expected = <<~DOCBOOK + + Example + GET / + + DOCBOOK + expect(actual).to eq(expected.strip) + end + end end diff --git a/resources/asciidoctor/spec/open_in_widget_spec.rb b/resources/asciidoctor/spec/open_in_widget_spec.rb new file mode 100644 index 0000000000000..3a9505110bbca --- /dev/null +++ b/resources/asciidoctor/spec/open_in_widget_spec.rb @@ -0,0 +1,167 @@ +# frozen_string_literal: true + +require 'open_in_widget/extension' + +RSpec.describe OpenInWidget do + before(:each) do + Asciidoctor::Extensions.register do + treeprocessor OpenInWidget + end + end + + after(:each) do + Asciidoctor::Extensions.unregister_all + end + + spec_dir = File.dirname(__FILE__) + + def stub_file_opts(result) + return { + 'copy_snippet' => proc { |uri, source| result << [uri, source] }, + 'write_snippet' => proc { |uri, snippet| result << [uri, snippet] }, + } + end + + %w[console sense kibana].each do |lang| + it "supports automatic snippet extraction with #{lang} language" do + input = <<~ASCIIDOC + == Example + [source,#{lang}] + ---- + GET / + ---- + ASCIIDOC + expected = <<~DOCBOOK + + Example + GET / + + DOCBOOK + file_opts = [] + actual = convert input, stub_file_opts(file_opts), eq( + "INFO: : line 3: writing snippet snippets/1.#{lang}" + ) + expect(actual).to eq(expected.strip) + expect(file_opts).to eq([ + ["snippets/1.#{lang}", "GET /\n"], + ]) + end + + it "supports automatic snippet extraction for many snippets with #{lang} language" do + input = <<~ASCIIDOC + == Example + [source,#{lang}] + ---- + GET / + ---- + + [source,#{lang}] + ---- + GET / + ---- + + [source,#{lang}] + ---- + GET / + ---- + + [source,#{lang}] + ---- + GET / + ---- + ASCIIDOC + expected = <<~DOCBOOK + + Example + GET / + GET / + GET / + GET / + + DOCBOOK + file_opts = [] + warnings = <<~WARNINGS + INFO: : line 3: writing snippet snippets/1.#{lang} + INFO: : line 8: writing snippet snippets/2.#{lang} + INFO: : line 13: writing snippet snippets/3.#{lang} + INFO: : line 18: writing snippet snippets/4.#{lang} + WARNINGS + actual = convert input, stub_file_opts(file_opts), eq(warnings.strip) + expect(actual).to eq(expected.strip) + expect(file_opts).to eq([ + ["snippets/1.#{lang}", "GET /\n"], + ["snippets/2.#{lang}", "GET /\n"], + ["snippets/3.#{lang}", "GET /\n"], + ["snippets/4.#{lang}", "GET /\n"], + ]) + end + + it "supports override snippet path with #{lang} language" do + input = <<~ASCIIDOC + == Example + [source,#{lang},snippet=snippet.#{lang}] + ---- + GET / + ---- + ASCIIDOC + expected = <<~DOCBOOK + + Example + GET / + + DOCBOOK + warnings = <<~WARNINGS + WARN: : line 3: reading snippets from a path makes the book harder to read + INFO: : line 3: copying snippet #{spec_dir}/snippets/snippet.#{lang} + WARNINGS + file_opts = [] + actual = convert input, stub_file_opts(file_opts), eq(warnings.strip) + expect(actual).to eq(expected.strip) + expect(file_opts).to eq([ + ["snippets/snippet.#{lang}", "#{spec_dir}/snippets/snippet.#{lang}"], + ]) + end + end + + it "strips callouts from written snippets" do + input = <<~ASCIIDOC + == Example + [source,console] + ---- + GET / <1> + + POST /foo/_doc/1 + { + "f1": "v1" <2> + } + ---- + ASCIIDOC + expected = <<~DOCBOOK + + Example + GET / + + POST /foo/_doc/1 + { + "f1": "v1" + } + + DOCBOOK + file_opts = [] + actual = convert input, stub_file_opts(file_opts), eq( + "INFO: : line 3: writing snippet snippets/1.console" + ) + expect(actual).to eq(expected.strip) + expected_snippet_body = <<~SNIPPET + GET / + + POST /foo/_doc/1 + { + "f1": "v1" + } + SNIPPET + expect(file_opts).to eq([ + ["snippets/1.console", expected_snippet_body], + ]) + end +end diff --git a/resources/asciidoctor/spec/snippets/snippet.console b/resources/asciidoctor/spec/snippets/snippet.console new file mode 100644 index 0000000000000..c6ee45096e44b --- /dev/null +++ b/resources/asciidoctor/spec/snippets/snippet.console @@ -0,0 +1 @@ +GET / diff --git a/resources/asciidoctor/spec/snippets/snippet.kibana b/resources/asciidoctor/spec/snippets/snippet.kibana new file mode 100644 index 0000000000000..06faf0330fc0d --- /dev/null +++ b/resources/asciidoctor/spec/snippets/snippet.kibana @@ -0,0 +1 @@ +GET /api/security/role/my_kibana_role diff --git a/resources/asciidoctor/spec/snippets/snippet.sense b/resources/asciidoctor/spec/snippets/snippet.sense new file mode 100644 index 0000000000000..c6ee45096e44b --- /dev/null +++ b/resources/asciidoctor/spec/snippets/snippet.sense @@ -0,0 +1 @@ +GET / diff --git a/resources/website_common.xsl b/resources/website_common.xsl index 1be0bbc7dad46..b357f872d8ba1 100644 --- a/resources/website_common.xsl +++ b/resources/website_common.xsl @@ -134,8 +134,19 @@
- +
+ + +
+ + + + + + @@ -331,7 +342,7 @@ - + @@ -353,6 +364,7 @@ + From b3fa0ec839e09fac509f2fa9491387e9e9a130b4 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Thu, 21 Feb 2019 16:27:52 -0500 Subject: [PATCH 2/8] Word --- README.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.asciidoc b/README.asciidoc index b8303b57893ae..834a7ab8838c8 100644 --- a/README.asciidoc +++ b/README.asciidoc @@ -1010,7 +1010,7 @@ will open the code snippet in Console. There are two ways to do this, the the Elaticsearch repository because it can recognize it to make tests. The "Asciidoctor" way is preferred in other books, but only if they are built with "Asciidoctor". Try it first and if it works then use it. Otherwise, use the -"AsciiDoct" way. +"AsciiDoc" way. //// From 702975ec5d9656dd29e1430e7d9adcf661472885 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Fri, 22 Feb 2019 17:09:46 -0500 Subject: [PATCH 3/8] Clean up after merge --- integtest/Makefile | 2 +- integtest/html_diff | 4 ++++ .../lib/elastic_compat_tree_processor/extension.rb | 2 +- resources/asciidoctor/lib/open_in_widget/extension.rb | 2 +- 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/integtest/Makefile b/integtest/Makefile index 7c10e4c666157..46dee0f06bbc4 100644 --- a/integtest/Makefile +++ b/integtest/Makefile @@ -48,7 +48,7 @@ readme_expected_files: /tmp/readme_asciidoc ) # The grep -v below are for known issues with asciidoctor for file in $$(cd /tmp/$*_asciidoc && find * -type f -name '*.html' \ - | grep -v 'blocks\|changes\|experimental\|multi-part'); do \ + | grep -v 'changes\|experimental\|multi-part'); do \ ./html_diff /tmp/$*_asciidoc/$$file /tmp/$*_asciidoctor/$$file; \ done diff --git a/integtest/html_diff b/integtest/html_diff index 3f90e255a44be..0a97d0356e5cf 100755 --- a/integtest/html_diff +++ b/integtest/html_diff @@ -31,6 +31,10 @@ def normalize_html(html): # Remove the zero width space that asciidoctor adds after each horizontal # ellipsis. They don't hurt anything but asciidoc doesn't make them html = html.replace('\u2026\u200b', '\u2026') + # We intentionally changed lang-js to lang-console because in Asciidoctor + # because that is more accurate + html = html.replace('"programlisting prettyprint lang-js"', + '"programlisting prettyprint lang-console"') # Temporary workaround for known issues html = html.replace('class="edit_me" href="/./', 'class="edit_me" href="') html = re.sub( diff --git a/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb b/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb index fb651cec75c59..70c3017b53880 100644 --- a/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb +++ b/resources/asciidoctor/lib/elastic_compat_tree_processor/extension.rb @@ -64,7 +64,7 @@ def process_subs(block) 'CONSOLE' => 'console', 'KIBANA' => 'kibana', 'SENSE' => 'sense', - } + }.freeze def process_lang_override(block) next_block = block.next_adjacent_block diff --git a/resources/asciidoctor/lib/open_in_widget/extension.rb b/resources/asciidoctor/lib/open_in_widget/extension.rb index 4dd93261b1b5e..90be43986c910 100644 --- a/resources/asciidoctor/lib/open_in_widget/extension.rb +++ b/resources/asciidoctor/lib/open_in_widget/extension.rb @@ -39,7 +39,7 @@ class OpenInWidget < TreeProcessorScaffold include Asciidoctor::Logging - CALLOUT_SCAN_RX = / ?#{Asciidoctor::CalloutScanRx}/ + CALLOUT_SCAN_RX = / ?#{Asciidoctor::CalloutScanRx}/.freeze def process_block(block) return unless block.context == :listing && block.style == 'source' From b74306c12f71d17ef1cf15df19904c2591a954f4 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Tue, 26 Feb 2019 10:11:58 -0500 Subject: [PATCH 4/8] Missing command --- .../asciidoctor/spec/elastic_compat_tree_processor_spec.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb b/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb index 9eec2101f4bbd..04ba3a3d23b6c 100644 --- a/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb +++ b/resources/asciidoctor/spec/elastic_compat_tree_processor_spec.rb @@ -76,15 +76,16 @@ %w[CONSOLE console], %w[AUTOSENSE sense], %w[KIBANA kibana], - ].each do |name, lang| - it "transforms legacy // #{name} commands into the #{lang} language" do + %w[SENSE:path/to/snippet.sense sense], + ].each do |command, lang| + it "transforms legacy // #{command} commands into the #{lang} language" do actual = convert <<~ASCIIDOC == Example [source,js] ---- GET / ---- - pass:[// #{name}] + pass:[// #{command}] ASCIIDOC expected = <<~DOCBOOK From e2722182060f161d7623b9cd8b66fe54a69b6cb3 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Tue, 26 Feb 2019 10:15:01 -0500 Subject: [PATCH 5/8] speeling --- resources/website_common.xsl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/resources/website_common.xsl b/resources/website_common.xsl index b357f872d8ba1..f549e2adc3993 100644 --- a/resources/website_common.xsl +++ b/resources/website_common.xsl @@ -136,7 +136,7 @@
- +
- + From cc8f3c7192da7b2a297c070be557a7540d340bb9 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Tue, 26 Feb 2019 14:15:25 -0500 Subject: [PATCH 6/8] Shift to error And shift the warning for being hard to read to after the copy. --- .../lib/open_in_widget/extension.rb | 4 ++-- .../spec/elastic_compat_preprocessor_spec.rb | 2 +- .../asciidoctor/spec/open_in_widget_spec.rb | 18 +++++++++++++++++- 3 files changed, 20 insertions(+), 4 deletions(-) diff --git a/resources/asciidoctor/lib/open_in_widget/extension.rb b/resources/asciidoctor/lib/open_in_widget/extension.rb index 90be43986c910..02c1651109688 100644 --- a/resources/asciidoctor/lib/open_in_widget/extension.rb +++ b/resources/asciidoctor/lib/open_in_widget/extension.rb @@ -49,15 +49,15 @@ def process_block(block) snippet = block.attr 'snippet' if snippet - logger.warn message_with_context "reading snippets from a path makes the book harder to read", :source_location => block.source_location # If you specify the snippet path then we should copy it into the # destination directory so it is available for Kibana. snippet_path = "snippets/#{snippet}" normalized = block.normalize_system_path(snippet_path, block.document.base_dir) if File.readable? normalized copy_snippet block, normalized, snippet_path + logger.warn message_with_context "reading snippets from a path makes the book harder to read", :source_location => block.source_location else - logger.warn message_with_context "can't read snippet from #{normalized}", :source_location => block.source_location + logger.error message_with_context "can't read snippet from #{normalized}", :source_location => block.source_location end else # If you don't specify the snippet then we assign it a number and read diff --git a/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb b/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb index 062a7b6837316..1be12568e8530 100644 --- a/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb +++ b/resources/asciidoctor/spec/elastic_compat_preprocessor_spec.rb @@ -369,8 +369,8 @@ def stub_file_opts DOCBOOK warnings = <<~WARNINGS - WARN: : line 3: reading snippets from a path makes the book harder to read INFO: : line 3: copying snippet #{spec_dir}/snippets/snippet.sense + WARN: : line 3: reading snippets from a path makes the book harder to read WARNINGS actual = convert input, stub_file_opts, eq(warnings.strip) expect(actual).to eq(expected.strip) diff --git a/resources/asciidoctor/spec/open_in_widget_spec.rb b/resources/asciidoctor/spec/open_in_widget_spec.rb index 3a9505110bbca..35765652677b3 100644 --- a/resources/asciidoctor/spec/open_in_widget_spec.rb +++ b/resources/asciidoctor/spec/open_in_widget_spec.rb @@ -111,8 +111,8 @@ def stub_file_opts(result) DOCBOOK warnings = <<~WARNINGS - WARN: : line 3: reading snippets from a path makes the book harder to read INFO: : line 3: copying snippet #{spec_dir}/snippets/snippet.#{lang} + WARN: : line 3: reading snippets from a path makes the book harder to read WARNINGS file_opts = [] actual = convert input, stub_file_opts(file_opts), eq(warnings.strip) @@ -121,6 +121,22 @@ def stub_file_opts(result) ["snippets/snippet.#{lang}", "#{spec_dir}/snippets/snippet.#{lang}"], ]) end + + it "logs an error if override snippet is missing with #{lang} language" do + input = <<~ASCIIDOC + == Example + [source,#{lang},snippet=missing.#{lang}] + ---- + GET / + ---- + ASCIIDOC + warnings = <<~WARNINGS + ERROR: : line 3: can't read snippet from #{spec_dir}/snippets/missing.#{lang} + WARNINGS + file_opts = [] + convert input, stub_file_opts(file_opts), eq(warnings.strip) + expect(file_opts).to eq([]) + end end it "strips callouts from written snippets" do From 15ba6d246c207de08bc471e0728b9b878dfb805f Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Thu, 7 Mar 2019 10:42:28 -0500 Subject: [PATCH 7/8] Test cleanups --- integtest/Makefile | 6 +++--- integtest/html_diff | 7 +++---- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/integtest/Makefile b/integtest/Makefile index 6588f7cd5c6b8..5b2b56684209c 100644 --- a/integtest/Makefile +++ b/integtest/Makefile @@ -63,15 +63,15 @@ experimental_expected_files: /tmp/experimental_asciidoc %_same_files: /tmp/%_asciidoc /tmp/%_asciidoctor diff \ <(cd /tmp/$*_asciidoc && find * -type f | sort \ - | grep -v snippets \ + | grep -v 'snippets/' \ ) \ <(cd /tmp/$*_asciidoctor && find * -type f | sort \ - | grep -v snippets \ + | grep -v 'snippets/' \ ) - # The grep -v below are for known issues with asciidoctor for file in $$(cd /tmp/$*_asciidoc && find * -type f -name '*.html'); do \ ./html_diff /tmp/$*_asciidoc/$$file /tmp/$*_asciidoctor/$$file; \ done + # TODO validate the snippets have the same contents even if the files aren't the same # Build the docs into the target define BD= diff --git a/integtest/html_diff b/integtest/html_diff index 0f71a1badd1f1..d1b0f939076b8 100755 --- a/integtest/html_diff +++ b/integtest/html_diff @@ -39,10 +39,9 @@ def normalize_html(html): # because that is more accurate html = html.replace('"programlisting prettyprint lang-js"', '"programlisting prettyprint lang-console"') - # Temporary workaround for known issues - html = re.sub( - r'(?m)^\s+
' - r'\s+
\n', '', html) + # The URL for the console snippets has changed + html = re.sub(r'data-snippet="[^"]+"', 'data-snippet="snippet"', html) + # Temporary work around for known issue html = html.replace('\\<1>', '<1>') return html From 922f3ec766feb4771e0c783c782bfaca3ca25c84 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Thu, 7 Mar 2019 10:50:56 -0500 Subject: [PATCH 8/8] Style --- .../lib/elastic_compat_preprocessor/extension.rb | 3 ++- resources/asciidoctor/lib/open_in_widget/extension.rb | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb b/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb index 14a23ba827cea..e101e5831062e 100644 --- a/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb +++ b/resources/asciidoctor/lib/elastic_compat_preprocessor/extension.rb @@ -113,6 +113,7 @@ class ElasticCompatPreprocessor < Asciidoctor::Extensions::Preprocessor INCLUDE_TAGGED_DIRECTIVE_RX = /^include-tagged::([^\[][^\[]*)\[(#{Asciidoctor::CC_ANY}+)?\]$/.freeze SOURCE_WITH_SUBS_RX = /^\["source", ?"[^"]+", ?subs="(#{Asciidoctor::CC_ANY}+)"\]$/.freeze CODE_BLOCK_RX = /^-----*$/.freeze + SNIPPET_RX = %r{//\s*(?:AUTOSENSE|KIBANA|CONSOLE|SENSE:[^\n<]+)}.freeze def process(_document, reader) reader.instance_variable_set :@in_attribute_only_block, false @@ -175,7 +176,7 @@ def reader.process_line(line) # CONSOLE snippet. Asciidoctor really doesn't recommend this sort of # thing but we have thousands of them and it'll take us some time to # stop doing it. - line&.gsub!(%r{//\s*(?:AUTOSENSE|KIBANA|CONSOLE|SENSE:[^\n<]+)}, 'pass:[\0]') + line&.gsub!(SNIPPET_RX, 'pass:[\0]') end end reader diff --git a/resources/asciidoctor/lib/open_in_widget/extension.rb b/resources/asciidoctor/lib/open_in_widget/extension.rb index 02c1651109688..cd5e676e0253f 100644 --- a/resources/asciidoctor/lib/open_in_widget/extension.rb +++ b/resources/asciidoctor/lib/open_in_widget/extension.rb @@ -61,8 +61,8 @@ def process_block(block) end else # If you don't specify the snippet then we assign it a number and read - # if from the contents of the source listing, copying it to the - # destination directory so it is available for Kibana. + # the contents of the source listing, copying it to the destination + # directory so it is available for Kibana. snippet_number = block.document.attr 'snippet_number', 1 snippet = "#{snippet_number}.#{lang}" block.document.set_attr 'snippet_number', snippet_number + 1 @@ -75,7 +75,7 @@ def process_block(block) block.document.register :links, snippet_path def block.content - @attributes['snippet_link'] + super + "#{@attributes['snippet_link']}#{super}" end end