Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ExecCommandWith for cmd.exe in Windows #1072

Closed
wants to merge 5 commits into from

Conversation

janlazo
Copy link
Contributor

@janlazo janlazo commented Oct 2, 2017

Close #1018

Run the command as is in cmd.exe with no parsing and escaping.
Explicity set cmd.SysProcAttr so execCommand does not escape the command.
Technically, the command should be escaped with ^ for special characters, including ".
This allows cmd.exe commands to be chained together.

See neovim/neovim#7343 (comment)

However, this requires a new shellescape function that is specific to one of the following:

  • interactive prompt
  • batchfile
  • command name

fzf#shellescape in the Vim plugin can handle only the batchfile.

Close junegunn#1018

Run the command as is in cmd.exe with no parsing and escaping.
Explicity set cmd.SysProcAttr so execCommand does not escape the command.
Technically, the command should be escaped with ^ for special characters, including ".
This allows cmd.exe commands to be chained together.

See neovim/neovim#7343 (comment)

However, this requires a new shellescape function that is specific to one of the following:
- interactive prompt
- batchfile
- command name
fzf#shellescape in the Vim plugin can handle only the batchfile.
@junegunn
Copy link
Owner

junegunn commented Oct 2, 2017

Thanks. First off, let's make it clear that fzf is a command-line program that can be used with or without Vim (many users of fzf do not even use Vim) and there is no way we can enforce anything on bind/preview expressions users specify. The specifics of fzf#shellecape should be out of the discussion here.

Having said that, I have a few comments.

  1. I see that you use "%s", so what happens if the command contains "s? Is it really okay? I'm specifically asking about this because placeholder expression {} is currently replaced to double-quoted string of the current line on Windows; see https://github.com/junegunn/fzf/blob/0.17.0/src/terminal.go#L1111-L1116
  2. Should we create a temporary batch file inside Go code instead if we eventually can't avoid non-trivial escaping of the command?

@janlazo
Copy link
Contributor Author

janlazo commented Oct 2, 2017

I mentioned fzf#shellescape because it is tested already in Windows and I'm interested in the internal s:shellesc_cmd. I like to rewrite s:shellesc_cmd in Go such that it works for the interactive prompt but I'm haven't gotten to Go regex yet. What users specify is mostly what they get because I don't know how to escape all the backslashes. Everything else is manageable.

As for your questions,

  1. It's okay to have multiple, uneven " as long as it works in the interactive prompt, cmd.exe is launched with shellcmdflag /s /c, and the command is wrapped in double quotes thanks to cmd.exe "intelligent" dequoting. Problem is how cmd.exe parses a command in the prompt. From previous experience, double-quoting works only if there are no nested double quotes and the last character is not a backslash. What's actually required is a double escape, one for each token and another for the entire command when passing to cmd.exe.
    Idea is to make it similar to sh -c 'user_command'.
    Instead, it will be cmd.exe /s /c ^"escaped_user_command^"
  2. Yes, but I don't think we'll get there yet. Unlike Vim, this is your project so we can adjust the escape code as we go.

For reference, I'm trying to avoid case 1, mentioned in cmd.exe /?, by forcing the case 2.

@junegunn
Copy link
Owner

junegunn commented Oct 2, 2017

What is "interactive prompt"? EDIT: Nevermind, I get it, you meant cmd session.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 2, 2017

I see what you mean now for the double-quoted string for the placeholder.
The entire command can break if there's another double quote after the placeholder.
This is a problem for piping and redirection if the placeholder is not the last token in the preview command.

I need a crude shellescape for the command after all.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 2, 2017

Besides quoteEntry, is there anything else that the placeholder string needs to be escaped correctly?

@janlazo
Copy link
Contributor Author

janlazo commented Oct 3, 2017

Is strings.replace in quoteEntry for shellwords.Parse? I thought strconv.Quote handled this already for " and \.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 3, 2017

I kept strconv.Quote for convenience in quoteEntry but it should explicitly escape the placeholder now in a cmd.exe session.

@junegunn
Copy link
Owner

junegunn commented Oct 3, 2017

I thought strconv.Quote handled this already for " and .

I think you're right. Also, it seems that it was a mistake to use strconv.Quote as it does more than just quoting the string.

According to https://golang.org/pkg/strconv/#Quote

The returned string uses Go escape sequences (\t, \n, \xFF, \u0100) for control characters and non-printable characters

I don't think we want this behavior in this context.

quoteEntry uses strconv.Quote in Windows but this escapes more than \ and "
Use strings.Replace to handle only \ and ".
    - double all backslashes
    - double-quote the entry, escaping all inner double quotes with backslash
src/terminal.go Outdated
return strconv.Quote(strings.Replace(entry, "\"", "\\\"", -1))
escaped := strings.Replace(entry, "\\", "\\\\", -1)
escaped = "\"" + strings.Replace(escaped, "\"", "\\\"", -1) + "\""
r, _ := regexp.Compile("[&|<>()@^%\"]")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use raw string literals (using backticks) in these lines to make them more readable.

https://golang.org/ref/spec#String_literals

@janlazo
Copy link
Contributor Author

janlazo commented Oct 3, 2017

What about parsing FZF_DEFAULT_OPTS?

words, _ := shellwords.Parse(os.Getenv("FZF_DEFAULT_OPTS"))

@junegunn
Copy link
Owner

junegunn commented Oct 3, 2017

What about it? I don't see how it's related to the current discussion.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 3, 2017

The parser is incompatible with cmd.exe or powershell.exe for non-trivial parsing so the user must set it to work on sh.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 3, 2017

Anyway, I can escape FZF_DEFAULT_COMMAND for cmd.exe. I couldn't customize it well for rg, sift, or powershell before because of shellwords.Parse.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 4, 2017

If there's no issues with the changes, then I will update the docs for --preview and FZF_DEFAULT_COMMAND.

@junegunn
Copy link
Owner

junegunn commented Oct 5, 2017

The parser is incompatible with cmd.exe or powershell.exe for non-trivial parsing so the user must set it to work on sh.

Yes, I'd like to leave this as an open issue, since 1. I believe it's rare to have non-trivial options in $FZF_DEFAULT_OPTS, 2. and it's easy to come up with a sh-compatible version of it.

then I will update the docs for --preview and FZF_DEFAULT_COMMAND

What needs to be mentioned?

@janlazo
Copy link
Contributor Author

janlazo commented Oct 5, 2017

What needs to be mentioned?

The user has to do the escaping themselves because fzf will run their command as is on cmd.exe, not SHELL. Updating the README is enough but fzf --help could mention it in Windows.

@junegunn
Copy link
Owner

junegunn commented Oct 6, 2017

The user has to do the escaping themselves

Can you give an example where escaping is needed?

their command as is on cmd.exe, not SHELL

Umm, is SHELL actually used on Windows? I didn't expect Windows users would think that fzf would ever use SHELL. Maybe I'm not following you correctly?

@janlazo
Copy link
Contributor Author

janlazo commented Oct 6, 2017

Can you give an example where escaping is needed?

Umm, is SHELL actually used on Windows?

fzf unsets TERM to run in sh (aliased to bash). SHELL is set to /usr/bin/bash.

@junegunn
Copy link
Owner

junegunn commented Oct 6, 2017

Thanks for the comment. But I was also curious how a command would look like after manual escaping. So the commands shown in https://github.com/junegunn/fzf/wiki/Windows#relative-filepaths will have to change?

fzf unsets TERM to run in sh (aliased to bash). SHELL is set to /usr/bin/bash.

This is the point I don't understand. /usr/bin/bash is not a valid path on Windows, is it? Is SHELL a thing on Windows?

@junegunn
Copy link
Owner

junegunn commented Oct 6, 2017

Are we discussing Cygwin environment?

@janlazo
Copy link
Contributor Author

janlazo commented Oct 6, 2017

So the commands shown in https://github.com/junegunn/fzf/wiki/Windows#relative-filepaths will have to change?

For cmd.exe only for consistency with this PR (ie. add /s flag and wrap the command with double quotes)
The rest have little/no escaping so they're unaffected.

Is SHELL a thing on Windows?

No, COMSPEC (or ComSpec) is but its value is almost always the absolute path of cmd.exe.

Are we discussing Cygwin environment?

For SHELL or running fzf on sh or bash.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 6, 2017

/usr/bin/bash is not a valid path on Windows, is it?

By default, no.
Forward slash is supported and the network drive is optional for paths within the current network drive.
If the user made a directory C:\usr\bin\, then cd /usr/bin is allowed.

@junegunn
Copy link
Owner

junegunn commented Oct 6, 2017

Thanks for the clarification and for being patient with me. So for the users running fzf on cmd.exe, there's no confusion, but Cygwin or bash users should be aware that SHELL is not used by fzf and FZF_DEFAULT_COMMAND should be written for cmd.exe.

You can briefly mention this on the README page or we can just add a link to the Windows wiki page.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 6, 2017

We can update the Windows wiki page and then add a link in the README and Cygwin wiki page to consolidate the explanation for why SHELL is ignored and how FZF_DEFAULT_COMMAND should be set.

I don't mind doing more work to clarify the changes introduced.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 7, 2017

On second thought, !abc! must be escaped as ^!abc^!. The Vim plugin needs the update as well.
Test case is a prompt asking a user to define their prompt as follows:

cmd /v:ON /s /c "set /p my_prompt= & fzf --prompt !my_prompt!"

Use case is a short for loop on the preview command

@junegunn
Copy link
Owner

junegunn commented Oct 7, 2017

We should probably add terminal_test.go and implement test cases for the Windows version of quoteEntry.

@janlazo
Copy link
Contributor Author

janlazo commented Oct 9, 2017

Will it run on Travis? I thought util.IsWindows is defined on compile time in src/util/util_windows.go

@junegunn
Copy link
Owner

junegunn commented Oct 9, 2017

util.IsWindows returns a static value, but we can extract the block as quoteEntryCmd and test it separately.

if util.IsWindows() {
  return quoteEntryCmd(entry)
} 
return ...

@junegunn
Copy link
Owner

Can you add a few test cases?

@janlazo
Copy link
Contributor Author

janlazo commented Oct 11, 2017

I can add unescaped commands from https://github.com/junegunn/fzf/wiki/Windows#relative-filepaths, some shell commands I used in neovim/neovim#7343, and basic statements in batchfiles and powershell scripts.

It helps if I can isolate these test cases into one file so I can test with go run quote_test.go.
This is what I have on a separate file to test quoteEntry on the echo command (cmd.exe) and echo.exe (cygwin). echo.exe should output exactly the original unescaped test case.

package main

import (
	"fmt"
	"os/exec"
	"syscall"
	"strings"
	"regexp"
)

func main() {
	tests := [...]string{
		"\"",
		"\\",
		"\\\"",
		"\"\\\\\\\"",
		"&|<>()@^%!",
		"%USERPROFILE%",
		"C:\\Program Files (x86)\\",
	}

	for _, v := range tests {
		escaped := Escape(v)
		out, _ := ExecCommandWith("echo " + escaped).Output()
		out2, _ := ExecCommandWith("\"echo\" " + escaped).Output()
		fmt.Printf("%s\n%s\n%s%s\n", v, escaped, out, out2)
	}
}

func Escape(entry string) string {
	escaped := strings.Replace(entry, `\`, `\\`, -1)
	escaped = `"` + strings.Replace(escaped, `"`, `\"`, -1) + `"`
	r, _ := regexp.Compile(`[&|<>()@^%!"]`)
	return r.ReplaceAllStringFunc(escaped, func (match string) string {
		return "^" + match
	})
}

func ExecCommandWith(command string) *exec.Cmd {
	cmd := exec.Command("cmd")
	cmd.SysProcAttr = &syscall.SysProcAttr{
	    HideWindow: false,
	    CmdLine: fmt.Sprintf(` /s /c "%s"`, command),
	    CreationFlags: 0,
	}
	return cmd
}

@junegunn
Copy link
Owner

junegunn commented Oct 11, 2017

Functional test cases for quoteEntry should suffice for now.

diff --git a/src/terminal.go b/src/terminal.go
index 1890a2a..98d837c 100644
--- a/src/terminal.go
+++ b/src/terminal.go
@@ -1103,14 +1103,18 @@ func keyMatch(key int, event tui.Event) bool {
 		event.Type == tui.Mouse && key == tui.DoubleClick && event.MouseEvent.Double
 }
 
+func quoteEntryCmd(entry string) string {
+	escaped := strings.Replace(entry, `\`, `\\`, -1)
+	escaped = `"` + strings.Replace(escaped, `"`, `\"`, -1) + `"`
+	r, _ := regexp.Compile(`[&|<>()@^%!"]`)
+	return r.ReplaceAllStringFunc(escaped, func(match string) string {
+		return "^" + match
+	})
+}
+
 func quoteEntry(entry string) string {
 	if util.IsWindows() {
-		escaped := strings.Replace(entry, `\`, `\\`, -1)
-		escaped = `"` + strings.Replace(escaped, `"`, `\"`, -1) + `"`
-		r, _ := regexp.Compile(`[&|<>()@^%!"]`)
-		return r.ReplaceAllStringFunc(escaped, func (match string) string {
-		    return "^" + match
-		})
+		return quoteEntryCmd(entry)
 	}
 	return "'" + strings.Replace(entry, "'", "'\\''", -1) + "'"
 }
diff --git a/src/terminal_test.go b/src/terminal_test.go
index d42d2b8..60f2b1a 100644
--- a/src/terminal_test.go
+++ b/src/terminal_test.go
@@ -91,3 +91,22 @@ func TestReplacePlaceholder(t *testing.T) {
 	result = replacePlaceholder("echo {}/{1}/{3}/{2..3}", true, Delimiter{regex: regex}, false, "query", items1)
 	check("echo '  foo'\\''bar baz'/'f'/'r b'/''\\''bar b'")
 }
+
+func TestQuoteEntryCmd(t *testing.T) {
+	tests := map[string]string{
+		`"`:                       `^"\^"^"`,
+		`\`:                       `^"\\^"`,
+		`\"`:                      `^"\\\^"^"`,
+		`"\\\"`:                   `^"\^"\\\\\\\^"^"`,
+		`&|<>()@^%!`:              `^"^&^|^<^>^(^)^@^^^%^!^"`,
+		`%USERPROFILE%`:           `^"^%USERPROFILE^%^"`,
+		`C:\Program Files (x86)\`: `^"C:\\Program Files ^(x86^)\\^"`,
+	}
+
+	for input, expected := range tests {
+		escaped := quoteEntryCmd(input)
+		if escaped != expected {
+			t.Errorf("Input: %s, expected: %s, actual %s", input, expected, escaped)
+		}
+	}
+}

This is all I need for now. Maybe you could add a few more test cases to the map? I'll leave it up to you.

junegunn pushed a commit that referenced this pull request Oct 14, 2017
Close #1018

Run the command as is in cmd.exe with no parsing and escaping.
Explicity set cmd.SysProcAttr so execCommand does not escape the command.
Technically, the command should be escaped with ^ for special characters,
including ". This allows cmd.exe commands to be chained together.

See neovim/neovim#7343 (comment)

This commit also updates quoteEntry to use strings.Replace instead of
strconv.Quote which escapes more than \ and ".
@junegunn
Copy link
Owner

Closed by c4185e8. I added the test cases to your commits and pushed to master. Let me know if there's anything else you want to address. Thanks!

@junegunn junegunn closed this Oct 14, 2017
@janlazo
Copy link
Contributor Author

janlazo commented Oct 14, 2017

I will send another PR to add more test cases from the wiki to the map. I was busy this week so I didn't have time to include your changes myself so thanks for closing this with your changes.

@junegunn can you update the README or wiki for me to mention SHELL issue for Bash/Cygwin users so they can set FZF_DEFAULT_COMMAND accordingly?

@junegunn
Copy link
Owner

No hurries. I already did update the wiki page, so please take a look.

https://github.com/junegunn/fzf/wiki/Windows

I'll also add a link to the page on the README page.

junegunn added a commit that referenced this pull request Oct 15, 2017
@janlazo
Copy link
Contributor Author

janlazo commented Oct 15, 2017

That's fine. Thanks you.

@janlazo janlazo deleted the go_windows_ExecCommand branch October 31, 2017 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants