Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make configs aware of structured data layout #2090

Closed
mberhault opened this issue Aug 13, 2015 · 4 comments · Fixed by #2495
Closed

Make configs aware of structured data layout #2090

mberhault opened this issue Aug 13, 2015 · 4 comments · Fixed by #2495
Assignees
@mberhault
Milestone
Beta

Comments

@mberhault
Copy link
Contributor

Zone and accounting configs need to be aware of the structured data layout.
At the very least, we need table-level granularity for both.
@mberhault mberhault self-assigned this Aug 13, 2015
@petermattis
Copy link
Collaborator

We need to redesign the configs. Prefix maps make a lot less sense when the objects the user sees are database, table and user. I'd like to see accounting automatically performed at the database, table and user levels and for zones to be configurable at those levels.

@spencerkimball
Copy link
Member

👍 I think zone configs listed by key prefix should be phased out and added instead to database and table descriptors.

I think accounting configs can be removed entirely because the database / table structure already properly delimits the key prefixes which we'd want to keep accounting information over.

The permission configs are actively being replaced by @mberhault with the SQL privileges.

@thschroeter thschroeter mentioned this issue Aug 13, 2015
Closed
@mberhault
Copy link
Contributor Author

@thschroeter: fyi, this is purely about the location of the configs and their layout.
while they won't be in their current locations, but probably attached to the db/table descriptors (or similar), we'll still need to gossip them to avoid extra lookups everytime. Any work done on the gossiping part should be reasonably easy to adapt to other locations.

Still, there's a decent amount of thought to be put into this, so as of yet I have no idea what this will look like, just vague ideas. The first step of this will most likely be a quick RFC.

@thschroeter
Copy link
Contributor

@mberhault: thanks for the explanation.

@tamird tamird mentioned this issue Aug 14, 2015
Closed
@jess-edwards jess-edwards mentioned this issue Aug 17, 2015
Closed
78 tasks
@jess-edwards jess-edwards added this to the Beta milestone Aug 17, 2015
This was referenced Aug 21, 2015
Closed
Closed
Closed
@mberhault mberhault mentioned this issue Aug 24, 2015
Merged
@mberhault mberhault mentioned this issue Aug 24, 2015
Merged
mberhault pushed a commit that referenced this issue Aug 24, 2015
Remove user config.
42f4da1 
Work towards #2090.

* remove UserConfig proto
* remove admin users endpoint
* change cli/user commands to use basic sql statements to store the data
  in the system.users table (#2235)
mberhault pushed a commit that referenced this issue Aug 24, 2015
Create system.users table.
010e11d 
Work towards #2090.

This creates a new system table storing username/hashed-password
that is currently in the users config (removal will come next).

The main part of this change is applying custom permissions based
on the system table. descriptor and namespace are read-only,
but users needs to be modified.
mberhault pushed a commit that referenced this issue Aug 24, 2015
Remove accounting config
c0fa217 
Work towards #2090

Remove accounting config.
This includes removing the acct cli command since there is currently
nothing in the accounting config.
mberhault pushed a commit that referenced this issue Aug 25, 2015
Remove accounting config
0200c5d 
Work towards #2090

Remove accounting config/keys/endpoints/cli/etc...

There is currently nothing in the accounting config, and no exact plans.
When this emerges, we'll move it into the system database alongside
other configs.
This was referenced Aug 25, 2015
Merged
Closed
@mberhault mberhault mentioned this issue Sep 10, 2015
Merged
mberhault pushed a commit that referenced this issue Sep 10, 2015
Add Zones system table.
9e048b7 
Work towards: #2090

Additional tweaks:
* fix all tests that depend on the number of system databases/tables
  to use dynamic counts.
* rework the SystemDBSPan loading a bit. change checksum to sha1 and
  include keys.
mberhault pushed a commit that referenced this issue Sep 14, 2015
Use zone configs from system table.
d07f87a 
Fixes #2090 (make configs aware of structured layout)
Fixes #2179 (sync system DB across nodes)
Fixes #2272 (split at table boundaries)

This switches all zone configs from the old-school prefix-based one to
the system.zones table.

Basic summary:
* the systemdb span (all system tables) gets gossiped as
  config.SystemConfig
* config.SystemConfig has two basic methods: ComputeSplitKeys(start,
  end) and GetZoneConfigForKey(range start key)
* we split at every user object ID (database or table) below the
  currently allocated one
* if a table does not have a zone config, we use its databases's config.
* if a database does not have a zone config, we use the hard-coded
  default.

Some likely changes in the very near future:
* pre-build list of actual zones from existing tables (not databases)
  and only split on those.
* allow setting the system span config
* allow setting the non-structured span config (probably a reserved ID)

Remaining TODOs for this PR:
* failing: no idea why: storage/gc_queue_test.go TestGCQueueIntentResolution
* need to port: kv/split_test.go TestRangeSplitsWithWritePressure
* failing kv and client tests with --race. no clue why.
* port cli zones commands
* complete tests on config/config and sql/config
@mberhault mberhault mentioned this issue Sep 14, 2015
Merged
mberhault pushed a commit that referenced this issue Sep 14, 2015
Use zone configs from system table.
13644ca 
Fixes #2090 (make configs aware of structured layout)
Fixes #2179 (sync system DB across nodes)
Fixes #2272 (split at table boundaries)

This switches all zone configs from the old-school prefix-based one to
the system.zones table.

Basic summary:
* the systemdb span (all system tables) gets gossiped as
  config.SystemConfig
* config.SystemConfig has two basic methods: ComputeSplitKeys(start,
  end) and GetZoneConfigForKey(range start key)
* we split at every user object ID (database or table) below the
  currently allocated one
* if a table does not have a zone config, we use its databases's config.
* if a database does not have a zone config, we use the hard-coded
  default.

Some likely changes in the very near future:
* port cli zones commands
* pre-build list of actual zones from existing tables (not databases)
  and only split on those.
* allow setting the system span config
* allow setting the non-structured span config (probably a reserved ID)
mberhault pushed a commit that referenced this issue Sep 15, 2015
Use zone configs from system table.
b253376 
Fixes #2090 (make configs aware of structured layout)
Fixes #2179 (sync system DB across nodes)
Fixes #2272 (split at table boundaries)

This switches all zone configs from the old-school prefix-based one to
the system.zones table.

Basic summary:
* the systemdb span (all system tables) gets gossiped as
  config.SystemConfig
* config.SystemConfig has two basic methods: ComputeSplitKeys(start,
  end) and GetZoneConfigForKey(range start key)
* we split at every user object ID (database or table) below the
  currently allocated one
* if a table does not have a zone config, we use its databases's config.
* if a database does not have a zone config, we use the hard-coded
  default.

Some likely changes in the very near future:
* port cli zones commands
* pre-build list of actual zones from existing tables (not databases)
  and only split on those.
* allow setting the system span config
* allow setting the non-structured span config (probably a reserved ID)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mberhault mberhault

Labels
None yet
Projects
None yet
Milestone
Beta
Development

Successfully merging a pull request may close this issue.

Use zone configs from system table. cockroachdb/cockroach
5 participants
@thschroeter @mberhault @jess-edwards @spencerkimball @petermattis