Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Bug 7571 - set directory root for debugger.html #7594

Closed

Conversation

jainsneha23
Copy link
Contributor

Fixes #7571

Summary of Changes

  • fix for absolute paths

@jainsneha23
Copy link
Contributor Author

The code needs refactoring

Copy link
Contributor

@jasonLaster jasonLaster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test in addToTree.spec.js or another relevant file?

@jasonLaster
Copy link
Contributor

So, I believe we started using whatwg-url for this reason last month here.

https://www.npmjs.com/package/whatwg-url

I wonder if using this library would give us a more spec compliant URL parse that would help us...

@jainsneha23
Copy link
Contributor Author

So, I believe we started using whatwg-url for this reason last month here.

https://www.npmjs.com/package/whatwg-url

I wonder if using this library would give us a more spec compliant URL parse that would help us...

Thanks . Will look into this , if It can help. This solution is hacky and I need to refactor this to have more robust solution.

case "ng:":
case "webpack:":
const path = pathname.substring(2);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using whatwg-url will automatically do this. Can I introduce that module in src/utils/url.js ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we can try that

@jainsneha23
Copy link
Contributor Author

@jasonLaster This is better than what I did before. Also fixes the absolute path for https://w2x2oypmqk.codesandbox.io/

Still I am not completely satisfied here. The way to create directory has to change if the url is from webpack. Check chrome's webpack folder. It creates folders based on webpack paths. Check image attached. I think we need to re-think on webpack tree. What are your thoughts on this?

screenshot 2018-12-26 at 4 46 42 pm

Note: Will add unit tests for webpack, once I am done with the exact output that we want.

@jasonLaster
Copy link
Contributor

jasonLaster commented Dec 27, 2018

@jainsneha23 can you give some more examples of the differences between chrome and firefox? I'm not sure I follow...

Also, could you share the whatwg example, either by pushing it to this branch or by sharing another branch.

by the way, @loganfsmyth is our path expert and will be back on monday.

@jainsneha23
Copy link
Contributor Author

jainsneha23 commented Dec 27, 2018

@jasonLaster

Find the screeshot below on the difference on how node and whatwg handles URL differently.

image

With protocols ending with :// for webpack and ng, node URL doesn't seem to return correct path. There are extra slashes. So, using whatwg URL is better.

Also we have URLs like

  • webpack:///../../abc (2 level up than where the webpack config is)
  • webpack:////Users/abc (absolute path)
  • webpack:///abc/def (same directory as the webpack config)

The screenshot above shows how chrome creates folders with . .. for level up folders

But in our source-map, we trim all the dots. So, that is not achievable right now. But the path for absolute paths is fixed. I have to fix the unit test though. Will do it tomorrow.

May be we can think on should we handle the level up folders for webpack, and raise a separate issue for the same. As that will require changes in source-map as well.

Copy link
Contributor

@loganfsmyth loganfsmyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my questions on this are really more about the existing architecture, but the big question to me is, why aren't these source URLs already absolute? It seems like we should aim to have an absolute URL computed up front for all sources, and if we have that, the WHATWG URL API pretty much provides everything else we need, and we can avoid pretty much all of the logic in _getURL.

return {
...def,
path: pathname,
filename,
group: `${protocol}//`
group: `${protocol}//`,
isAbsolutePath: pathname.startsWith("//")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem right to me, if we're using WHATWG URL to process this, it should just be a single slash.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@loganfsmyth For webpack, we can have urls with double slash, and dots
Please find some webpack urls below:

webpack:///webpack/bootstrap fc4578a0bef29d2a63f3
webpack:///src/component/index.js
webpack:////Users/workspace/github/debugger.html/node_modules/react/index.js
webpack:///./utils/debug.js
webpack:///../node_modules/devtools-environment/test-flag.js

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In these examples, with whatwg-url, the pathname should be

  • /webpack/bootstrap fc4578a0bef29d2a63f3
  • /src/component/index.js
  • //Users/workspace/github/debugger.html/node_modules/react/index.js
  • /utils/debug.js
  • /node_modules/devtools-environment/test-flag.js

what is throwing me off here is these are already all absolute because they start with a single slash.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@loganfsmyth
Yes with whatwg-url the pathname are as you mentioned. What I meant by absolute was system absolute path.

If you see, all other paths are relative to the debugger.html folder. And only this path //Users/workspace/github/debugger.html/node_modules/react/index.js is system absolute path.

In webpack, the paths are file paths in system, and either they are relative to the path from where the webpack was run like the other 4, or absolute system path.

And this issue raised happened only for the system absolute path. I was also able to see the same issue in https://w2x2oypmqk.codesandbox.io/

Sorry for miscommunication. And we can have a quick chat on this today evening if you want.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think part of what is tripping me up is, aren't those examples just mistakes on the part of Webpack? All URLs like this are absolute. The fact that there is a double slash at the start of one of them just means that one of the path segments is an empty string. It seems like a mistake in whatever configuration generated that URL. It seems unreasonable to expect the devtools to try to interpret what is essentially bad data.

With URLs as

webpack:///webpack/bootstrap fc4578a0bef29d2a63f3
webpack:///src/component/index.js
webpack:////Users/workspace/github/debugger.html/node_modules/react/index.js
webpack:///./utils/debug.js
webpack:///../node_modules/devtools-environment/test-flag.js

I'd expect that the folder structure would be

Webpack (webpack://)
* "webpack"
  * "bootstrap fc4578a0bef29d2a63f3"
* "src"
  * "components"
    * "index.js"
* ""
  * "Users"
    * "workspace"
      * "github"
        * "debugger.html"
          * "node_modules"
            * "react"
              * "index.js"
* "utils"
  * "debug.js"
* "node_modules"
  * "devtools-enviroment"
    * "test-flag.js"

return {
...def,
path: pathname,
filename,
group: `${protocol}//`
group: `${protocol}//`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised that this doesn't include the hostname. How that we've got WHATWG URL, I'd have expected more of the logic in this function to collapse into

group: `${protocol}//${host || ""}`

@jasonLaster
Copy link
Contributor

@jainsneha23 I wonder how well whatwg does in general...

could you try running getURL without all of the special cases and share some findings?

diff --git a/src/utils/sources-tree/getURL.js b/src/utils/sources-tree/getURL.js
index 42e1f5227..055912b19 100644
--- a/src/utils/sources-tree/getURL.js
+++ b/src/utils/sources-tree/getURL.js
@@ -39,82 +39,7 @@ function _getURL(source: Source, defaultDomain: string): ParsedURL {

   const { pathname, protocol, host } = parse(url);
   const filename = getUnicodeUrlPath(getFilenameFromPath(pathname));
-
-  switch (protocol) {
-    case "javascript:":
-      // Ignore `javascript:` URLs for now
-      return def;
-
-    case "moz-extension:":
-    case "resource:":
-      return {
-        ...def,
-        path: pathname,
-        filename,
-        group: `${protocol}//${host || ""}`
-      };
-
-    case "webpack:":
-    case "ng:":
-      return {
-        ...def,
-        path: pathname,
-        filename,
-        group: `${protocol}//`
-      };
-
-    case "about:":
-      // An about page is a special case
-      return {
-        ...def,
-        path: "/",
-        filename,
-        group: url
-      };
-
-    case "data:":
-      return {
-        ...def,
-        path: "/",
-        group: NoDomain,
-        filename: url
-      };
-
-    case "":
-      if (pathname && pathname.startsWith("/")) {
-        // use file protocol for a URL like "/foo/bar.js"
-        return {
-          ...def,
-          path: pathname,
-          filename,
-          group: "file://"
-        };
-      } else if (!host) {
-        return {
-          ...def,
-          path: url,
-          group: defaultDomain || "",
-          filename
-        };
-      }
-      break;
-
-    case "http:":
-    case "https:":
-      return {
-        ...def,
-        path: pathname,
-        filename,
-        group: getUnicodeHostname(host)
-      };
-  }
-
-  return {
-    ...def,
-    path: pathname,
-    group: protocol ? `${protocol}//` : "",
-    filename
-  };
+  return { ...def, pathname, protocol, host, filename };
 }

@jasonLaster
Copy link
Contributor

By the way, as you can probably tell... this is somewhat of a scary change so it will likely move pretty slow as we try to sort out all of the different scenarios. Thanks (in advance) for your patience :)

@jainsneha23
Copy link
Contributor Author

jainsneha23 commented Jan 7, 2019

Here are some differences listed (we can ignore node url for now, as this happens on the browser). Also notice the difference in chrome and firefox. SO, better to use WHATWG for consistency

WHATWG URL:   const URL = require(whatwg-url).URL; new URL(urlstring)
NODE:   const URL = require(url); URL.parse(urlstring)
BROWSER:  new URL(urlstring)
URL USING WHATWG URL USING NODE URL USING CHROME URL USING FIREFOX NIGHTLY URL
javascript:alert(1) url: javascript:alert(1) url: javascript:alert(1) url: javascript:alert(1) url: javascript:alert(1)
  protocol: javascript: protocol: javascript: protocol: javascript: protocol: javascript:
  host: host: null host: host:
  pathname: alert(1) pathname: alert(1) pathname: alert(1) pathname: alert(1)
  filename: (index) filename: (index) filename: (index) filename: (index)
         
moz-extension:kasjdkasdjklasjdlk url: moz-extension:kasjdkasdjklasjdlk url: moz-extension:kasjdkasdjklasjdlk url: moz-extension:kasjdkasdjklasjdlk url: moz-extension:kasjdkasdjklasjdlk
  protocol: moz-extension: protocol: moz-extension: protocol: moz-extension: protocol: moz-extension:
  host: host: kasjdkasdjklasjdlk host: host: kasjdkasdjklasjdlk
  pathname: kasjdkasdjklasjdlk pathname: null pathname: kasjdkasdjklasjdlk pathname: /
  filename: (index) filename: filename: (index) filename: (index)
         
resource://gre/res/svg.css url: resource://gre/res/svg.css url: resource://gre/res/svg.css url: resource://gre/res/svg.css url: resource://gre/res/svg.css
  protocol: resource: protocol: resource: protocol: resource: protocol: resource:
  host: gre host: gre host: host: gre
  pathname: /res/svg.css pathname: /res/svg.css pathname: //gre/res/svg.css pathname: /res/svg.css
  filename: svg.css filename: svg.css filename: svg.css filename: svg.css
         
webpack:///abc/index.js url: webpack:///abc/index.js url: webpack:///abc/index.js url: webpack:///abc/index.js url: webpack:///abc/index.js
  protocol: webpack: protocol: webpack: protocol: webpack: protocol: webpack:
  host: host: host: host:
  pathname: /abc/index.js pathname: /abc/index.js pathname: ///abc/index.js pathname: ///abc/index.js
  filename: index.js filename: index.js filename: index.js filename: index.js
         
webpack:////abc/index.js url: webpack:////abc/index.js url: webpack:////abc/index.js url: webpack:////abc/index.js url: webpack:////abc/index.js
  protocol: webpack: protocol: webpack: protocol: webpack: protocol: webpack:
  host: host: host: host:
  pathname: //abc/index.js pathname: //abc/index.js pathname: ////abc/index.js pathname: ////abc/index.js
  filename: index.js filename: index.js filename: index.js filename: index.js
         
ng:///abc/index.js url: ng:///abc/index.js url: ng:///abc/index.js url: ng:///abc/index.js url: ng:///abc/index.js
  protocol: ng: protocol: ng: protocol: ng: protocol: ng:
  host: host: host: host:
  pathname: /abc/index.js pathname: /abc/index.js pathname: ///abc/index.js pathname: ///abc/index.js
  filename: index.js filename: index.js filename: index.js filename: index.js
         
about:config url: about:config url: about:config url: about:config url: about:config
  protocol: about: protocol: about: protocol: about: protocol: about:
  host: host: config host: host:
  pathname: config pathname: null pathname: config pathname: config
  filename: (index) filename: filename: (index) filename: (index)
         
http://firefox.com url: http://firefox.com url: http://firefox.com url: http://firefox.com url: http://firefox.com
  protocol: http: protocol: http: protocol: http: protocol: http:
  host: firefox.com host: firefox.com host: firefox.com host: firefox.com
  pathname: / pathname: / pathname: / pathname: /
  filename: (index) filename: (index) filename: (index) filename: (index)
         
https://firefox.com url: https://firefox.com url: https://firefox.com url: https://firefox.com url: https://firefox.com
  protocol: https: protocol: https: protocol: https: protocol: https:
  host: firefox.com host: firefox.com host: firefox.com host: firefox.com
  pathname: / pathname: / pathname: / pathname: /
  filename: (index) filename: (index) filename: (index) filename: (index)
         
data:text/html, url: data:text/html, url: data:text/html, url: data:text/html, url: data:text/html
  protocol: data: protocol: data: protocol: data: protocol: data:
  host: host: text host: host:
  pathname: text/html, pathname: /html,%20%3Chtml%20contenteditable%3E pathname: text/html, pathname: text/html
  filename: (index) filename: (index) filename: (index) filename: (index)
         
/foo/bar.js url: /foo/bar.js url: /foo/bar.js url: /foo/bar.js url: /foo/bar.js
  protocol: protocol: null protocol: protocol:
  host: host: null host: host:
  pathname: /foo/bar.js pathname: /foo/bar.js pathname: /foo/bar.js pathname: /foo/bar.js
  filename: bar.js filename: bar.js filename: bar.js filename: bar.js
         
//cdn.ajax.com/react.js url: //cdn.ajax.com/react.js url: //cdn.ajax.com/react.js url: //cdn.ajax.com/react.js url: //cdn.ajax.com/react.js
  protocol: protocol: null protocol: protocol:
  host: host: null host: host:
  pathname: //cdn.ajax.com/react.js pathname: //cdn.ajax.com/react.js pathname: //cdn.ajax.com/react.js pathname: //cdn.ajax.com/react.js
  filename: react.js filename: react.js filename: react.js filename: react.js
         
file://foo/bar.js url: file://foo/bar.js url: file://foo/bar.js url: file://foo/bar.js url: file://foo/bar.js
  protocol: file: protocol: file: protocol: file: protocol: file:
  host: foo host: foo host: foo host:
  pathname: /bar.js pathname: /bar.js pathname: /bar.js pathname: /bar.js
  filename: bar.js filename: bar.js filename: bar.js filename: bar.js
         
urn:isbn:9780141036144 url: urn:isbn:9780141036144 url: urn:isbn:9780141036144 url: urn:isbn:9780141036144 url: urn:isbn:9780141036144
  protocol: urn: protocol: urn: protocol: urn: protocol: urn:
  host: host: isbn:9780141036144 host: host:
  pathname: isbn:9780141036144 pathname: null pathname: isbn:9780141036144 pathname: isbn:9780141036144
  filename: (index) filename: filename: (index) filename: (index)
         
tel:+1-816-555-1212 url: tel:+1-816-555-1212 url: tel:+1-816-555-1212 url: tel:+1-816-555-1212 url: tel:+1-816-555-1212
  protocol: tel: protocol: tel: protocol: tel: protocol: tel:
  host: host: +1-816-555-1212 host: host:
  pathname: +1-816-555-1212 pathname: null pathname: +1-816-555-1212 pathname: +1-816-555-1212
  filename: (index) filename: filename: (index) filename: (index)
         
git@github.com:mdn/browser-compat-data.git url: git@github.com:mdn/browser-compat-data.git url: git@github.com:mdn/browser-compat-data.git url: git@github.com:mdn/browser-compat-data.git url: git@github.com:mdn/browser-compat-data.git
  protocol: protocol: null protocol: protocol:
  host: host: null host: host:
  pathname:git@github.com:mdn/browser-compat-data.git pathname: git@github.com:mdn/browser-compat-data.git pathname:git@github.com:mdn/browser-compat-data.git pathname:git@github.com:mdn/browser-compat-data.git
  filename: browser-compat-data.git filename: browser-compat-data.git filename: browser-compat-data.git filename: browser-compat-data.git
         
  url: ftp://example.org/resource.txt url: ftp://example.org/resource.txt url: ftp://example.org/resource.txt url: ftp://example.org/resource.txt
ftp://example.org/resource.txt protocol: ftp: protocol: ftp: protocol: ftp: protocol: ftp:
  host: example.org host: example.org host: example.org host: example.org
  pathname: /resource.txt pathname: /resource.txt pathname: /resource.txt pathname: /resource.txt
  filename: resource.txt filename: resource.txt filename: resource.txt filename: resource.txt
         
mailto:help@supercyberhelpdesk.info url: mailto:help@supercyberhelpdesk.info url: mailto:help@supercyberhelpdesk.info url: mailto:help@supercyberhelpdesk.info url: mailto:help@supercyberhelpdesk.info
  protocol: mailto: protocol: mailto: protocol: mailto: protocol: mailto:
  host: host: supercyberhelpdesk.info host: host:
  pathname:help@supercyberhelpdesk.info pathname: null pathname:help@supercyberhelpdesk.info pathname:help@supercyberhelpdesk.info
  filename:help@supercyberhelpdesk.info filename: filename:help@supercyberhelpdesk.info filename:help@supercyberhelpdesk.info

So, we will still need the special cases for javascript, about, data, and no protocol case (to handle file urls, and relative urls)

@@ -86,8 +86,9 @@ function traverseTree(
tree: TreeDirectory,
debuggeeHost: ?string
): TreeNode {
const parts = url.path.split("/").filter(p => p !== "");
parts.unshift(url.group);
const parts = url.path.split("/").filter((p, i) => p !== "" || i < 2);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a comment here as to what is going on.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@darkwing @loganfsmyth Do we really need this filter? Cannot we just render the path as is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that this filter seems unnecessary to me. My vote would be for

const parts = url.path.split("/");
if (parts[0] === "") parts.shift();
parts.unshift(url.group);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking over this again, I think we should just keep it simple and do

const parts = url.path.split("/");
parts.unshift(url.group);

and revisit after that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@loganfsmyth Unfortunately the snippet you cited creates the following:

tree

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, that's what I was worried about. Honestly, maybe the better approach would be to just leave the filter for now and change things more in a later pass?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@darkwing in davidwalsh.name, the empty folder you got was because of a path '/', which creates empty node. for this case, we can remove any trailing slash from a url.

otherwise this works fine. can you please have a look at my commit.

@darkwing
Copy link
Contributor

@loganfsmyth Since you were the sourcemaps legend, I'd like your take on this; i.e. if this is a viable solution.

@loganfsmyth loganfsmyth self-requested a review February 4, 2019 21:26
@darkwing
Copy link
Contributor

darkwing commented Feb 5, 2019

Seeing this with the current code:

ftreecurrent

@jasonLaster
Copy link
Contributor

@darkwing what is the test page?

@darkwing
Copy link
Contributor

darkwing commented Feb 5, 2019

My blog, of course :D

@darkwing
Copy link
Contributor

darkwing commented Feb 8, 2019

I played with this today and it's totally frustrating! I appreciate your efforts @jainsneha23!

  1. I tried adding a regex to remove multiple / (url.path.replace(/\/{2,}/g, "/").split("/").filter......) and that wasn't helpful.

  2. I commented out import { URL } from "whatwg-url"; from the url.js file and that didn't appear to make any visual change.

@jainsneha23
Copy link
Contributor Author

@jasonLaster @darkwing The branch is rebased with latest master, and unit tests are updates for files with no filename, webpack urls, webpack url with system absolute path. Its ready for review.

@jasonLaster
Copy link
Contributor

so... in terms of getting this library included in the debugger vendors...

There are a couple of steps:

  1. adding it to vendors.js
  2. updating copy-module.js which is in MC and not in github. This part means we will need to update MC first...
diff --git a/package.json b/package.json
index be1988695..77365dc2a 100644
--- a/package.json
+++ b/package.json
@@ -80,7 +80,8 @@
     "react-transition-group": "^2.2.1",
     "reselect": "^4.0.0",
     "svg-inline-react": "^3.0.0",
-    "wasmparser": "^0.7.0"
+    "wasmparser": "^0.7.0",
+    "whatwg-url": "..."
   },
   "private": true,
   "workspaces": [
diff --git a/src/vendors.js b/src/vendors.js
index 29660c0b2..bc7792280 100644
--- a/src/vendors.js
+++ b/src/vendors.js
@@ -28,6 +28,7 @@ import * as fuzzaldrinPlus from "fuzzaldrin-plus";
 import * as transition from "react-transition-group/Transition";
 import * as reactAriaComponentsTabs from "react-aria-components/src/tabs";
 import * as reselect from "reselect";
+import whatWg from "whatwg-url";

 // Modules imported without destructuring
 import classnames from "classnames";
@@ -55,5 +56,6 @@ export const vendored = {
   reselect,
   // Svg is required via relative paths, so the key is not imported path.
   // See .babel/transform-mc.js
-  Svg
+  Svg,
+  whatWg
 };

@loganfsmyth
Copy link
Contributor

Functionally this seems good to me. The hard remaining part is definitely getting this into mozilla-central itself.

@jasonLaster
Copy link
Contributor

I don't mind getting it into MC

@darkwing
Copy link
Contributor

I'd like to get some closure here -- @loganfsmyth @jasonLaster What's the difficulty in getting this into mc?

@jasonLaster
Copy link
Contributor

@darkwing I hope to do this next week

@jasonLaster
Copy link
Contributor

Hey @jainsneha23 i'm sorry for the delay, i'm finally getting around to landing this in firefox

https://phabricator.services.mozilla.com/D31313

and ufortunately i'm seeing the original issue with the debugger directory. Did you see this as well when working on it or was your version fixed?

@jainsneha23
Copy link
Contributor Author

@jasonLaster hey my version was fixed. Can you elaborate on this? Then I will check. Are you seeing the error after rebasing it with current master? Or my branch itself gives the error?

@jasonLaster
Copy link
Contributor

oh cool. Yeah, i copied your version in the phabricator branch. I would be really interested if this branch works when you rebase it...

@jainsneha23
Copy link
Contributor Author

Sure. I will rebase and check this weekend.

@jasonLaster
Copy link
Contributor

@jainsneha23
Copy link
Contributor Author

great!! thanks @jasonLaster @darkwing @loganfsmyth

@jainsneha23 jainsneha23 deleted the bug-7571-set-dir-root branch May 25, 2019 04:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[sources] Set directory root does not work for debugger.html
5 participants