Refactor with AST (#70)

* Refactor

Signed-off-by: Loong Dai <loong.dai@intel.com>

* fix CI

Signed-off-by: Loong Dai <loong.dai@intel.com>

* skip on Win

Signed-off-by: Loong Dai <loong.dai@intel.com>

* fix no imports

Signed-off-by: Loong Dai <loong.dai@intel.com>

Makefile

@@ -6,13 +6,13 @@ default: clean generate test build
 
 clean:
 	@echo BIN_OUTPUT: ${BIN_OUTPUT}
-	rm -rf dist/ cover.out
+	@rm -rf dist cover.out
 
 build: clean
-	go build -v -trimpath -o ${BIN_OUTPUT} .
+	@go build -v -trimpath -o ${BIN_OUTPUT} .
 
 test: clean
-	go test -v -cover ./...
+	@go test -v -cover ./...
 
 generate:
-	go generate ./...
+	@go generate ./...

README.md

@@ -4,6 +4,26 @@ GCI, a tool that controls golang package import order and makes it always determ
 
 The desired output format is highly configurable and allows for more custom formatting than `goimport` does.
 
+GCI considers a import block based on AST as below:
+```
+Doc
+Name Path Comment
+```
+All comments will keep as they were, except the independent comment blocks(line breaks before and after).
+
+GCI splits all import blocks into different sections, now support three section type:
+- standard: Golang official imports, like "fmt"
+- custom: Custom section, use full and the longest match(match full string first, if multiple matches, use the longest one)
+- default: All rest import blocks
+
+The priority is standard>custom>default, all sections sort alphabetically inside.
+
+All import blocks use one TAB(`\t`) as Indent.
+
+**Note**:
+
+`nolint` is hard to handle at section level, GCI will consider it as a single comment.
+
 ## Download
 
 ```shell
@@ -28,18 +48,14 @@ Aliases:
   print, output
 
 Flags:
-      --NoInlineComments           Drops inline comments while formatting
-      --NoPrefixComments           Drops comment lines above an import statement while formatting
-  -s, --Section strings            Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry.
-                                   Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
-                                   Def | Default - Contains all imports that could not be matched to another section type
-                                   NL | NewLine - Prints an empty line
-                                   Prefix(gitlab.com/myorg) | pkgPrefix(gitlab.com/myorg) - Groups all imports with the specified Prefix. Imports will be matched to the longest Prefix.
-                                   Std | Standard - Captures all standard packages if they do not match another section
-                                    (default [Standard,Default])
-  -x, --SectionSeparator strings   SectionSeparators are inserted between Sections (default [NewLine])
-      --SkipGeneratedFiles         Don't process generated files
-  -h, --help                       help for print
+  -d, --debug             Enables debug output from the formatter
+  -h, --help              help for print
+  -s, --section strings   Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry. The Section order is the same as below, default value is [Standard,Default].
+                          Std | Standard - Captures all standard packages if they do not match another section
+                          Prefix(github.com/daixiang0) | pkgPrefix(github.com/daixiang0) - Groups all imports with the specified Prefix. Imports will be matched to the longest Prefix.
+                          Def | Default - Contains all imports that could not be matched to another section type
+                          [DEPRECATED] Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
+                          [DEPRECATED] NL | NewLine - Prints an empty line
 ```
 
 ```shell
@@ -53,18 +69,14 @@ Aliases:
   write, overwrite
 
 Flags:
-      --NoInlineComments           Drops inline comments while formatting
-      --NoPrefixComments           Drops comment lines above an import statement while formatting
-  -s, --Section strings            Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry.
-                                   Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
-                                   Def | Default - Contains all imports that could not be matched to another section type
-                                   NL | NewLine - Prints an empty line
-                                   Prefix(gitlab.com/myorg) | pkgPrefix(gitlab.com/myorg) - Groups all imports with the specified Prefix. Imports will be matched to the longest Prefix.
-                                   Std | Standard - Captures all standard packages if they do not match another section
-                                    (default [Standard,Default])
-  -x, --SectionSeparator strings   SectionSeparators are inserted between Sections (default [NewLine])
-      --SkipGeneratedFiles         Don't process generated files
-  -h, --help                       help for write
+  -d, --debug             Enables debug output from the formatter
+  -h, --help              help for write
+  -s, --section strings   Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry. The Section order is the same as below, default value is [Standard,Default].
+                          Std | Standard - Captures all standard packages if they do not match another section
+                          Prefix(github.com/daixiang0) | pkgPrefix(github.com/daixiang0) - Groups all imports with the specified Prefix. Imports will be matched to the longest Prefix.
+                          Def | Default - Contains all imports that could not be matched to another section type
+                          [DEPRECATED] Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
+                          [DEPRECATED] NL | NewLine - Prints an empty line
 ```
 
 ```shell
@@ -75,19 +87,14 @@ Usage:
   gci diff path... [flags]
 
 Flags:
-      --NoInlineComments           Drops inline comments while formatting
-      --NoPrefixComments           Drops comment lines above an import statement while formatting
-  -s, --Section strings            Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry.
-                                   Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
-                                   Def | Default - Contains all imports that could not be matched to another section type
-                                   NL | NewLine - Prints an empty line
-                                   Prefix(gitlab.com/myorg) | pkgPrefix(gitlab.com/myorg) - Groups all imports with the specified Prefix. Imports will be matched to the longest Prefix.
-                                   Std | Standard - Captures all standard packages if they do not match another section
-                                    (default [Standard,Default])
-  -x, --SectionSeparator strings   SectionSeparators are inserted between Sections (default [NewLine])
-      --SkipGeneratedFiles         Don't process generated files
-  -d, --debug                      Enables debug output from the formatter
-  -h, --help                       help for diff
+  -d, --debug             Enables debug output from the formatter
+  -h, --help              help for diff
+  -s, --section strings   Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry. The Section order is the same as below, default value is [Standard,Default].
+                          Std | Standard - Captures all standard packages if they do not match another section
+                          Prefix(github.com/daixiang0) | pkgPrefix(github.com/daixiang0) - Groups all imports with the specified Prefix. Imports will be matched to the full and longest Prefix. All groups are in alphabetical order.
+                          Def | Default - Contains all imports that could not be matched to another section type
+                          [DEPRECATED] Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
+                          [DEPRECATED] NL | NewLine - Prints an empty line
 ```
 
 ### Old style
@@ -107,21 +114,21 @@ Flags:
 
 **Note**::
 
-The old style is only for local tests, `golangci-lint` uses new style.
+The old style is only for local tests, will be deprecated, please uses new style, `golangci-lint` uses new style as well.
 
 ## Examples
 
-Run `gci write --Section Standard --Section Default --Section "Prefix(github.com/daixiang0/gci)" main.go` and you will handle following cases:
+Run `gci write -s standard -s default -s "prefix(github.com/daixiang0/gci)" main.go` and you will handle following cases:
 
 ### simple case
 
 ```go
 package main
 import (
   "golang.org/x/tools"
-
+  
   "fmt"
-
+  
   "github.com/daixiang0/gci"
 )
 ```
@@ -131,11 +138,11 @@ to
 ```go
 package main
 import (
-  "fmt"
+    "fmt"
 
-  "golang.org/x/tools"
+    "github.com/daixiang0/gci"
 
-  "github.com/daixiang0/gci"
+    "golang.org/x/tools"
 )
 ```
 
@@ -157,12 +164,17 @@ package main
 import (
   "fmt"
 
-  go "github.com/golang"
-
   "github.com/daixiang0/gci"
+
+  go "github.com/golang"
 )
 ```
 
 ## TODO
 
+- Ensure only one blank between `Name` and `Path` in an import block
+- Ensure only one blank between `Path` and `Comment` in an import block
+- Format comments
 - Add more testcases
+- Support imports completion (please use `goimports` first then use GCI)
+- Optimize comments

cmd/gci/gcicommand.go

@@ -1,22 +1,18 @@
 package gci
 
 import (
-	"fmt"
-
 	"github.com/spf13/cobra"
 	"go.uber.org/zap/zapcore"
 
-	"github.com/daixiang0/gci/pkg/configuration"
-	"github.com/daixiang0/gci/pkg/constants"
-	"github.com/daixiang0/gci/pkg/gci"
-	sectionsPkg "github.com/daixiang0/gci/pkg/gci/sections"
+	"github.com/daixiang0/gci/pkg/config"
 	"github.com/daixiang0/gci/pkg/log"
+	"github.com/daixiang0/gci/pkg/section"
 )
 
-type processingFunc = func(args []string, gciCfg gci.GciConfiguration) error
+type processingFunc = func(args []string, gciCfg config.Config) error
 
 func (e *Executor) newGciCommand(use, short, long string, aliases []string, stdInSupport bool, processingFunc processingFunc) *cobra.Command {
-	var noInlineComments, noPrefixComments, debug, skipGeneratedFiles *bool
+	var noInlineComments, noPrefixComments, skipGenerated, debug *bool
 	var sectionStrings, sectionSeparatorStrings *[]string
 	cmd := cobra.Command{
 		Use:               use,
@@ -25,10 +21,13 @@ func (e *Executor) newGciCommand(use, short, long string, aliases []string, stdI
 		Long:              long,
 		ValidArgsFunction: goFileCompletion,
 		RunE: func(cmd *cobra.Command, args []string) error {
-			fmtCfg := configuration.FormatterConfiguration{*noInlineComments, *noPrefixComments, *debug}
-			gciCfg, err := gci.GciStringConfiguration{
-				fmtCfg, *sectionStrings, *sectionSeparatorStrings, *skipGeneratedFiles,
-			}.Parse()
+			fmtCfg := config.BoolConfig{
+				NoInlineComments: *noInlineComments,
+				NoPrefixComments: *noPrefixComments,
+				Debug:            *debug,
+				SkipGenerated:    *skipGenerated,
+			}
+			gciCfg, err := config.YamlConfig{Cfg: fmtCfg, SectionStrings: *sectionStrings, SectionSeparatorStrings: *sectionSeparatorStrings}.Parse()
 			if err != nil {
 				return err
 			}
@@ -45,18 +44,27 @@ func (e *Executor) newGciCommand(use, short, long string, aliases []string, stdI
 	// register command as subcommand
 	e.rootCmd.AddCommand(&cmd)
 
-	sectionHelp := "Sections define how inputs will be processed. " +
-		"Section names are case-insensitive and may contain parameters in (). " +
-		fmt.Sprintf("A section can contain a Prefix and a Suffix section which is delimited by %q. ", constants.SectionSeparator) +
-		"These sections can be used for formatting and will only be rendered if the main section contains an entry." +
-		"\n" +
-		sectionsPkg.SectionParserInst.SectionHelpTexts()
-	// add flags
 	debug = cmd.Flags().BoolP("debug", "d", false, "Enables debug output from the formatter")
+
+	sectionHelp := `Sections define how inputs will be processed. Section names are case-insensitive and may contain parameters in (). A section can contain a Prefix and a Suffix section which is delimited by ":". These sections can be used for formatting and will only be rendered if the main section contains an entry. The Section order is the same as below, default value is [Standard,Default].
+Std | Standard - Captures all standard packages if they do not match another section
+Prefix(github.com/daixiang0) | pkgPrefix(github.com/daixiang0) - Groups all imports with the specified Prefix. Imports will be matched to the longest Prefix.
+Def | Default - Contains all imports that could not be matched to another section type
+[DEPRECATED] Comment(your text here) | CommentLine(your text here) - Prints the specified indented comment
+[DEPRECATED] NL | NewLine - Prints an empty line`
+
+	skipGenerated = cmd.Flags().Bool("skip-generated", false, "Skip generated files")
+
+	sectionStrings = cmd.Flags().StringSliceP("section", "s", nil, sectionHelp)
+
+	// deprecated
 	noInlineComments = cmd.Flags().Bool("NoInlineComments", false, "Drops inline comments while formatting")
+	cmd.Flags().MarkDeprecated("NoInlineComments", "Drops inline comments while formatting")
 	noPrefixComments = cmd.Flags().Bool("NoPrefixComments", false, "Drops comment lines above an import statement while formatting")
-	sectionStrings = cmd.Flags().StringSliceP("Section", "s", gci.DefaultSections().String(), sectionHelp)
-	sectionSeparatorStrings = cmd.Flags().StringSliceP("SectionSeparator", "x", gci.DefaultSectionSeparators().String(), "SectionSeparators are inserted between Sections")
-	skipGeneratedFiles = cmd.Flags().Bool("SkipGeneratedFiles", false, "Don't process generated files")
+	cmd.Flags().MarkDeprecated("NoPrefixComments", "Drops inline comments while formatting")
+	sectionSeparatorStrings = cmd.Flags().StringSliceP("SectionSeparator", "x", section.DefaultSectionSeparators().String(), "SectionSeparators are inserted between Sections")
+	cmd.Flags().MarkDeprecated("SectionSeparator", "Drops inline comments while formatting")
+	cmd.Flags().MarkDeprecated("x", "Drops inline comments while formatting")
+
 	return &cmd
 }

cmd/gci/root.go

@@ -6,9 +6,10 @@ import (
 
 	"github.com/spf13/cobra"
 
-	"github.com/daixiang0/gci/pkg/configuration"
+	"github.com/daixiang0/gci/pkg/config"
 	"github.com/daixiang0/gci/pkg/gci"
 	"github.com/daixiang0/gci/pkg/log"
+	"github.com/daixiang0/gci/pkg/section"
 )
 
 type Executor struct {
@@ -59,9 +60,16 @@ func (e *Executor) runInCompatibilityMode(cmd *cobra.Command, args []string) err
 	}
 	// generate section specification from old localFlags format
 	sections := gci.LocalFlagsToSections(*e.localFlags)
-	sectionSeparators := gci.DefaultSectionSeparators()
-	cfg := gci.GciConfiguration{
-		configuration.FormatterConfiguration{false, false, false}, sections, sectionSeparators, false,
+	sectionSeparators := section.DefaultSectionSeparators()
+	cfg := config.Config{
+		BoolConfig: config.BoolConfig{
+			NoInlineComments: false,
+			NoPrefixComments: false,
+			Debug:            false,
+			SkipGenerated:    false,
+		},
+		Sections:          sections,
+		SectionSeparators: sectionSeparators,
 	}
 	if *e.writeMode {
 		return gci.WriteFormattedFiles(args, cfg)

go.mod

@@ -6,20 +6,23 @@ require (
 	github.com/hexops/gotextdiff v1.0.3
 	github.com/spf13/cobra v1.3.0
 	github.com/stretchr/testify v1.7.0
+	github.com/tj/assert v0.0.3
 	go.uber.org/zap v1.17.0
 	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
 	golang.org/x/tools v0.1.5
-	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
+	gopkg.in/yaml.v2 v2.4.0
 )
 
 require (
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
 	go.uber.org/atomic v1.7.0 // indirect
 	go.uber.org/multierr v1.6.0 // indirect
 	golang.org/x/mod v0.5.0 // indirect
 	golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect
 	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
 )

go.sum

@@ -278,8 +278,9 @@ github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FI
 github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
 github.com/pelletier/go-toml v1.9.4/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
-github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/sftp v1.10.1/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -325,6 +326,8 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
+github.com/tj/assert v0.0.3 h1:Df/BlaZ20mq6kuai7f5z2TvPFiwC3xaWJSDQNiIS3Rk=
+github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk=
 github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM=
 github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -764,6 +767,7 @@ gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
 gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=