Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle server side DESCRIBE #17

Closed
fruch opened this issue Mar 15, 2023 · 13 comments · Fixed by #88
Closed

handle server side DESCRIBE #17

fruch opened this issue Mar 15, 2023 · 13 comments · Fixed by #88
Assignees

Comments

@fruch
Copy link
Collaborator

fruch commented Mar 15, 2023

It was implemented on Scylla in:

scylladb/scylladb#11106

scylla-cqlsh has the code to support it, but for now it's disabled for scylla, we need to find a way to identify the server side is supported

few options:

  • based on scylla version
  • trying it on server side and fallback to client side
  • scylla to introduce a feature flags on a virtual table ?
@fruch fruch self-assigned this Mar 15, 2023
@raphaelsc
Copy link
Member

Bumped into this today.

@fruch
Copy link
Collaborator Author

fruch commented Apr 13, 2023

Bumped into this today.

What do you mean bumped, it was causing an issue ? or the describe didn't work as you expected ?

@fruch
Copy link
Collaborator Author

fruch commented Jun 19, 2023

@roydahan this is not exactly specific to our group, but I think whom could take it, would learn lots in the process.

@fruch
Copy link
Collaborator Author

fruch commented Jul 9, 2023

@avelanarius FYI this task is something I'm not getting to, and it sound like a something the driver team might suited to handle.

@raphaelsc
Copy link
Member

Bumped into this today.

What do you mean bumped, it was causing an issue ? or the describe didn't work as you expected ?

By bumped, I meant that cqlsh doesn't use server-side describe by default, meaning that I had to user other means to access it. There are new attributes being introduced to Scylla that cannot be seen today with cqlsh.

avelanarius added a commit to avelanarius/scylla-cqlsh that referenced this issue Jul 28, 2023
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version (since
it doesn't support all of its features), so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

Fixes scylladb#17
avelanarius added a commit to avelanarius/scylla-cqlsh that referenced this issue Jul 28, 2023
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version (since
it doesn't support all of its features), so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is 
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

Fixes scylladb#17
avelanarius added a commit to avelanarius/scylla-cqlsh that referenced this issue Jul 28, 2023
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is 
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
@avelanarius
Copy link

avelanarius commented Jul 28, 2023

few options:

  • based on scylla version
  • trying it on server side and fallback to client side
  • scylla to introduce a feature flags on a virtual table ?

I think "trying it on server side and fallback to client side" makes the most sense. I'm currently working on this issue (now only fixing tests which have hardcoded expected responses) and those are the reasons I listed:

The other possible solutions were rejected:

  • Based on Scylla version: would require ugly hard-coding of versions
  • Modifying Scylla to provide some indication that this feature is
    enabled: Scylla 5.2 is already released without it, by implementing
    it in another way, we'll get it out sooner
  • Do a trial "DESCRIBE" at the start of connection to detect if the
    server supports it: if cqlsh ever supported connecting to multiple
    nodes (right now it uses WhiteListRoundRobinPolicy) we would have
    to do the check on all of the nodes in case a rolling upgrade is
    currently occurring and some of the nodes don't support server-side
    DESCRIBE.

@avelanarius avelanarius assigned avelanarius and unassigned fruch Jul 28, 2023
@avelanarius
Copy link

Refs scylladb/scylladb#14895 - maybe I'll adjust the cqlsh tests to be more lenient towards missing whitespace, but it should be fixed in Scylla nevertheless.

@mykaul
Copy link
Contributor

mykaul commented Dec 25, 2023

We should prioritize this (for scylladb/scylladb#16482 , if we choose this path)

@roydahan
Copy link
Collaborator

@avelanarius I moved it back to you since I saw you wrote you were working on this issue.
I'm not sure if you meant that you finished and bump into tests failures or started from the tests and stopped there.
In any case, scylladb/scylladb#14895 is fixed now.

Let's get done with this one as well.

@avelanarius avelanarius assigned Lorak-mmk and unassigned avelanarius Feb 19, 2024
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 14, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is 
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 14, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is 
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 14, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is 
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 14, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a 
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is 
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 16, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 17, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 20, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 27, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 27, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
sylwiaszunejko pushed a commit to sylwiaszunejko/scylla-cqlsh that referenced this issue May 27, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes scylladb#17
@fruch fruch closed this as completed in #88 May 27, 2024
fruch pushed a commit that referenced this issue May 27, 2024
Since Scylla 5.2 (scylladb/scylladb@e6ffc22)
a support for server-side DESCRIBE was added. However, cqlsh did not
start to use this functionality, since it is only enabled if it detects
CQL version at least 4. Scylla did not increase this version number as
it doesn't support all of its features, so there is a need for a
different detection mechanism for server-side DESCRIBE.

This commit changes the behavior in do_describe: cqlsh will first try
to execute the server-side DESCRIBE. If this fails with SyntaxException,
meaning that Scylla doesn't support that command, cqlsh falls back to
the client-side DESCRIBE behavior.

The other possible solutions were rejected:
- Based on Scylla version: would require ugly hard-coding of versions
- Modifying Scylla to provide some indication that this feature is
  enabled: Scylla 5.2 is already released without it, by implementing
  it in another way, we'll get it out sooner
- Do a trial "DESCRIBE" at the start of connection to detect if the
  server supports it: if cqlsh ever supported connecting to multiple
  nodes (right now it uses WhiteListRoundRobinPolicy) we would have
  to do the check on all of the nodes in case a rolling upgrade is
  currently occurring and some of the nodes don't support server-side
  DESCRIBE.

The change was tested manually on a couple of last Scylla OSS, Scylla
Enterprise and Cassandra releases.

Fixes #17
@mykaul
Copy link
Contributor

mykaul commented Jul 21, 2024

@roydahan - this went in without tests. In general, I'm not sure where's the CI for cqlsh and where are its own tests.

@fruch
Copy link
Collaborator Author

fruch commented Jul 21, 2024

@roydahan - this went in without tests. In general, I'm not sure where's the CI for cqlsh and where are its own tests.

As part of the work in:
#88

@sylwiaszunejko tested it with dtest, there is a whole integration suite there for cqlsh, also covering describe.

Also the unit tests and use integration tests in cqlsh repo are covering, and were refactored during the work ion that PR.

@mykaul
Copy link
Contributor

mykaul commented Jul 22, 2024

@roydahan - this went in without tests. In general, I'm not sure where's the CI for cqlsh and where are its own tests.

As part of the work in: #88

@sylwiaszunejko tested it with dtest, there is a whole integration suite there for cqlsh, also covering describe.

Also the unit tests and use integration tests in cqlsh repo are covering, and were refactored during the work ion that PR.

Great - thanks. I could not find the cqlsh tests.

@Lorak-mmk
Copy link
Collaborator

@roydahan - this went in without tests. In general, I'm not sure where's the CI for cqlsh and where are its own tests.

As part of the work in: #88

@sylwiaszunejko tested it with dtest, there is a whole integration suite there for cqlsh, also covering describe.

Also the unit tests and use integration tests in cqlsh repo are covering, and were refactored during the work ion that PR.

If dtests are the only tests for clqsh, then perhaps cqlsh CI should run them automatically?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants