Add causality region column and implement isolation rules #4162

leoyvens · 2022-11-11T12:48:19Z

For tracking the necessary tables, a migration adds a has_causality_region column to subgraph_deployment of type text[], listing all entity types that need a causality region column for that deployment. This also fiddled a bit with DDL formatting, and changed a test so that it tests the exact DDL formatting.

This now also fully implements #3770.

lutter · 2022-11-23T19:38:06Z

One comment: since the set of entity types that need to store causality regions is static, this new column should go into subgraph_manifest. I'd like to change things so that only data that changes frequently (like the subgraph head pointer) are stored in subgraph_deployment and that slow moving data goes into subgraph_manifest.

And to bikeshed: has_causality_region makes me think this is a boolean column. Maybe entities_with_causality_region would be better name (though I'll live with either)

leoyvens · 2022-11-24T12:30:58Z

@mangas thanks taking a stab at this, I've moved your WIP commit to a separate branch. I'll take some things from it but I suspect some of the changes, such as adding a CR to the mapping trigger, are not necessary.

@lutter thanks for the review, I'll make those changes.

mangas · 2022-12-09T16:18:23Z

core/src/subgraph/registrar.rs

@@ -620,11 +620,30 @@ async fn create_subgraph_version<C: Blockchain, S: SubgraphStore>(
        "block" => format!("{:?}", base_block.as_ref().map(|(_,ptr)| ptr.number))
    );

+    // Entity types that may be touched by offchain data sources need a causality region column.
+    let needs_causality_region = manifest


I think this could benefit from some test coverage, just making sure that given some manifests, the DeploymentCreate ends up with the right entities

mangas · 2022-12-09T16:19:16Z

core/src/subgraph/runner.rs

@@ -998,7 +999,14 @@ async fn update_proof_of_indexing(
        // Create the special POI entity key specific to this causality_region
        let entity_key = EntityKey {
            entity_type: POI_OBJECT.to_owned(),
+
+            // There are two things called causality regions here, one is the causality region for


comment is really helpful 👍

graph/src/components/store/mod.rs

mangas · 2022-12-09T16:32:13Z

graph/src/data/subgraph/schema.rs

@@ -136,6 +137,15 @@ impl DeploymentCreate {
        self.debug_fork = fork;
        self
    }
+
+    pub fn entities_with_causality_region(


nitpick: with_offchain_entities?

An entity table might have a causality region column and stores some entities from onchain data sources. So calling this offchain_entities might be more confusing.

mangas · 2022-12-09T16:33:44Z

graph/src/data/subgraph/schema.rs

+    pub fn new(
+        raw_yaml: String,
+        manifest: &super::SubgraphManifest<impl Blockchain>,
+        entities_with_causality_region: Vec<EntityType>,


Suggested change

entities_with_causality_region: Vec<EntityType>,

offchan_entities: Vec<EntityType>,

with or without CR I think should be a detail of the store

mangas · 2022-12-09T16:36:07Z

graph/src/data_source/causality_region.rs

+            entity
+                .get("causality_region")
+                .and_then(|v| v.as_int())
+                .unwrap_or(0),


Suggested change

.unwrap_or(0),

.unwrap_or(CausalityRegion::ONCHAIN),

covering this with a basic unit test would also be good. If we don't replace the constant it could happen that someone changes the value and then this would be broken.

I added this to the relational_bytes::find test.

mangas · 2022-12-09T16:38:20Z

graph/src/data_source/mod.rs

@@ -206,6 +206,13 @@ impl<C: Blockchain> DataSource<C> {
            Self::Offchain(_) => vec![],
        }
    }
+
+    pub fn causality_region(&self) -> CausalityRegion {


If not covered already then adding a simple test would be good

I've added a data_source_helpers test.

mangas · 2022-12-09T16:38:34Z

graph/src/data_source/mod.rs

@@ -250,6 +257,13 @@ impl<C: Blockchain> DataSourceTemplate<C> {
        }
    }

+    pub fn as_offchain(&self) -> Option<&offchain::DataSourceTemplate> {


mangas · 2022-12-09T16:44:51Z

store/postgres/src/deployment_store.rs

@@ -1043,17 +1050,17 @@ impl DeploymentStore {
    ) -> Result<Option<Entity>, StoreError> {
        let conn = self.get_conn()?;
        let layout = self.layout(&conn, site)?;
-        layout.find(&conn, &key.entity_type, &key.entity_id, block)
+        layout.find(&conn, &key, block)
    }

    /// Retrieve all the entities matching `ids_for_type` from the


Seems to have some more details we could add here now

Improved the comment a bit.

mangas · 2022-12-09T16:46:40Z

store/postgres/src/relational.rs

@@ -500,34 +505,39 @@ impl Layout {
    pub fn find_many(


maybe comment on what is going on in this function after the changes could be helpful

mangas · 2022-12-09T16:48:58Z

store/postgres/src/relational.rs

@@ -1213,6 +1221,10 @@ pub struct Table {
    /// Entities in this table are immutable, i.e., will never be updated or
    /// deleted
    pub(crate) immutable: bool,
+
+    /// Whether this table has an explicit `causality_region` column. If `false`, then the column is
+    /// not present and the causality region for all rows is implicitly `0`.


Suggested change

/// not present and the causality region for all rows is implicitly `0`.

/// not present and the causality region for all rows is implicitly `CausalityRegion::ONCHAIN` .

mangas · 2022-12-09T16:56:06Z

store/postgres/src/relational_queries.rs

@@ -3511,6 +3556,24 @@ impl<'a> QueryFragment<Pg> for CopyEntityBatchQuery<'a> {
            }
            (false, false) => out.push_sql(BLOCK_RANGE_COLUMN),
        }
+
+        match (self.src.has_causality_region, self.dst.has_causality_region) {


Not sure if I missed it but I don't think these are now covered in the database tests other than the integration test that uses the ipfs SG right? Can you please ensure this is covered on the store tests? Apologies if it is and I missed it

Yes, all these graft combinations should be tested. But those tests are probably not trivial to write and at this point this has been open for too long. I'm thinking of merging it without these tests, with the risk that there may be a bug in grafting a subgraph with file data sources.

mangas · 2022-12-09T17:08:13Z

store/postgres/tests/relational_bytes.rs

+        let id2_key = EntityKey {
+            entity_id: ID2.into(),
+            entity_type: THING.clone(),
+            causality_region: CausalityRegion::ONCHAIN,


I guess these are the ones I missed earlier. maybe add one that's not ONCHAIN?

It would be a bit annoying to test this here since it would be necessary to add to the test layout a new entity type that somehow has a causality region column. I added coverage for looking up offchain entities to the integration tests, and added a debug_assert! to the entity cache that ensures cache consistency (so the test is not looking up only on the cache).

I think this is the price we have to pay for the design decision and current test strategy. IMO it's worth adding this coverage even if it takes longer to write. I would, however, add it in a separate PR.

mangas · 2022-12-09T17:09:19Z

approving as in general looks good to me. Would be good to fix the comments you think are worth fixing.

leoyvens · 2023-01-05T00:58:46Z

This now fully implements the isolation rules. The review comments have been addressed and this is ready for a final review.

mangas · 2023-01-20T15:36:27Z

store/postgres/src/relational.rs

            let entity_id = Word::from(data.id().expect("Invalid ID for entity."));
            processed_entities.insert((entity_type.clone(), entity_id.clone()));

-            // `__typename` is not a real field.
-            data.remove("__typename")


is this ok to remove now? I seem to remember this being added (by Eva?) because this was causing some issues?

This was just a refactor, a couple lines above the last parameter to deserialize_with_layout was changed to true so that the typename is already removed.

For now this just tracks the tables that need the column and adds the column to the DDL, but still unconditionally inserts 0. Inserting the correct causality region is follow up work.

…ith_causality_region`

The tricky part was changing `get_many` to return the entity key.

It was just necessary to make sure that `find` and `find_many` use the causality region in their where clause.

Callers wanted that anyways, and it helps tests.

* build(deps): bump blake3 from 1.3.1 to 1.3.2 (graphprotocol#4194) Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.3.1 to 1.3.2. - [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases) - [Commits](BLAKE3-team/BLAKE3@1.3.1...1.3.2) --- updated-dependencies: - dependency-name: blake3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump cid from 0.8.6 to 0.9.0 (graphprotocol#4193) Bumps [cid](https://github.com/multiformats/rust-cid) from 0.8.6 to 0.9.0. - [Release notes](https://github.com/multiformats/rust-cid/releases) - [Commits](multiformats/rust-cid@v0.8.6...v0.9.0) --- updated-dependencies: - dependency-name: cid dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump indexmap from 1.9.1 to 1.9.2 (graphprotocol#4192) Bumps [indexmap](https://github.com/bluss/indexmap) from 1.9.1 to 1.9.2. - [Release notes](https://github.com/bluss/indexmap/releases) - [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md) - [Commits](indexmap-rs/indexmap@1.9.1...1.9.2) --- updated-dependencies: - dependency-name: indexmap dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * all: fix some Clippy warnings (graphprotocol#4178) * node, store: Add `graphman stats target` * store: Make resolve_column_names return SqlName * node, store: Allow analyzing all tables with `graphman analyze` * node, store: Add `graphman stats set-target` * store/postgres: insist on a database with C locale * store: remove unused lazy_static! (graphprotocol#4200) * Update docker-compose.yml * Update docker-compose.yml * Fix: run locale check after migrations (graphprotocol#4201) * store/postgres: run locale check after migrations * store/postgres: place succes log message after locale check * Update docker-compose.yml * Update docker-compose.yml * ci: fix Postgres' locale in CI * integration-tests: createdb with C locale Co-authored-by: tilacog <tilacog@gmail.com> Co-authored-by: Adam Fuller <azf20@users.noreply.github.com> Co-authored-by: Filippo Costa <filippo@neysofu.me> Co-authored-by: Filippo Costa <filippo@edgeandnode.com> * update locale for parallel tests (graphprotocol#4205) * store: Unfail subgraph status properly after rewind This is being done by changing `revert_subgraph_errors` to change both the `failed` and `status` columns in the case of both being `failed`. Without this change, rewinding failed subgraphs past the failed block would result in the `failed` status staying the same. * tests: Test that rewind resets subgraph health Ports the `fatal-error` test from the legacy test suite to the new test suite, and extends it to check that the subgraph health is reset to `Healthy` after a rewind. * enable firehose eth block ingestor (graphprotocol#4204) * firehose ingestor transforms (graphprotocol#4216) * Updated `Substreams` to latest version of Protobuf definition and activated `ProductionMode` by default (graphprotocol#4211) The production mode is required to benefits from automatic backprocessing and downloading of block scoped data message as they are produced. This will drastically improve the ingestion speed of a substreams (time to gather some metrics!). Updated also the instructions to re-generate the `substreams.proto` file with more instructions of how we do it. Co-authored-by: Matthieu Vachon <matt@streamingfast.io> * Remove unused deps from `graph-server-metrics` (graphprotocol#4224) * server-metrics: remove unused deps * metrics-server: remove MetricsServer trait * build(deps): bump openssl from 0.10.42 to 0.10.43 (graphprotocol#4208) Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.42 to 0.10.43. - [Release notes](https://github.com/sfackler/rust-openssl/releases) - [Commits](sfackler/rust-openssl@openssl-v0.10.42...openssl-v0.10.43) --- updated-dependencies: - dependency-name: openssl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * The `SubstreamsBlockStream` was using `request.clone()` side stepping latest cursor value (graphprotocol#4228) The `request.clone()` does not correctly use the `latest_cursor` value which is the valid up to date in memory cursor to use on re-connection. This led to poisining error in `graph-node` where the same block was processed multiple time because the cursor was not correctly used. Fixed by moving the request creation directly where it's needed which will use the correct up to date `latest_cursor` value now. Co-authored-by: Matthieu Vachon <matt@streamingfast.io> * fail when ens rainbow not present (graphprotocol#4219) * docker: use exec form for CMD directive (graphprotocol#4217) * graph, graphql: expose metrics for GraphQL validations failures (graphprotocol#4230) * `EmptyNodeCapabilities` for chains that don't discriminate nodes (graphprotocol#3985) * chain(all), graph: EmptyNodeCapabilities * chain-ethereum: fix PartialOrd for NodeCapabilities The Ord and PartialOrd implementations of NodeCapabilities currently break trait rules: `NodeCapabilities {true, false}` is both greater than and less than (at the same time) `NodeCapabilities {false, true}`. It'd be nice to get this fixed before we get some obscure bug during comparisons, or a panic due to std assuming transitivity during sorts. * build(deps): bump express in /tests/integration-tests (graphprotocol#4235) Bumps [express](https://github.com/expressjs/express) from 4.17.1 to 4.18.2. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](expressjs/express@4.17.1...4.18.2) --- updated-dependencies: - dependency-name: express dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build: remove cargo-chef, revert to stable rust (graphprotocol#4187) * build(deps): bump base64 from 0.13.1 to 0.20.0 (graphprotocol#4241) Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.13.1 to 0.20.0. - [Release notes](https://github.com/marshallpierce/rust-base64/releases) - [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md) - [Commits](marshallpierce/rust-base64@v0.13.1...v0.20.0) --- updated-dependencies: - dependency-name: base64 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * graph, node: Move regex into the graph crate * node, store: Parse index definitions * node, store: Allow suppressing standard indexes in 'graphman index list' * graph, node: Colorize the output of 'graphman index list' * node, store: Use index::Method when creating manual indexes * node, store: Add SQL printing to `graphman index list` * node: Fix output from `graphman index list` * node, store: Parse the index method using FromStr * graphql: Simplify the arguments for prefetch::fetch * store: Simplify arguments for Layout.query * graph, graphql, store: Pass trace flag for query * all: Allow tracing GraphQL queries via HTTP * graph, graphql: Include query_id and variables in trace * graph, node: Print execution errors from `graphman query` * graph, graphql: Include block number in query trace * build(deps): bump openssl from 0.10.43 to 0.10.45 (graphprotocol#4267) Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.43 to 0.10.45. - [Release notes](https://github.com/sfackler/rust-openssl/releases) - [Commits](sfackler/rust-openssl@openssl-v0.10.43...openssl-v0.10.45) --- updated-dependencies: - dependency-name: openssl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump proc-macro2 from 1.0.47 to 1.0.49 (graphprotocol#4259) Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.47 to 1.0.49. - [Release notes](https://github.com/dtolnay/proc-macro2/releases) - [Commits](dtolnay/proc-macro2@1.0.47...1.0.49) --- updated-dependencies: - dependency-name: proc-macro2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump quote from 1.0.20 to 1.0.23 (graphprotocol#4258) Bumps [quote](https://github.com/dtolnay/quote) from 1.0.20 to 1.0.23. - [Release notes](https://github.com/dtolnay/quote/releases) - [Commits](dtolnay/quote@1.0.20...1.0.23) --- updated-dependencies: - dependency-name: quote dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump anyhow from 1.0.66 to 1.0.68 (graphprotocol#4257) Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.66 to 1.0.68. - [Release notes](https://github.com/dtolnay/anyhow/releases) - [Commits](dtolnay/anyhow@1.0.66...1.0.68) --- updated-dependencies: - dependency-name: anyhow dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump semver from 1.0.14 to 1.0.16 (graphprotocol#4256) Bumps [semver](https://github.com/dtolnay/semver) from 1.0.14 to 1.0.16. - [Release notes](https://github.com/dtolnay/semver/releases) - [Commits](dtolnay/semver@1.0.14...1.0.16) --- updated-dependencies: - dependency-name: semver dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump num_cpus from 1.14.0 to 1.15.0 (graphprotocol#4265) Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.14.0 to 1.15.0. - [Release notes](https://github.com/seanmonstar/num_cpus/releases) - [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md) - [Commits](seanmonstar/num_cpus@v1.14.0...v1.15.0) --- updated-dependencies: - dependency-name: num_cpus dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump termcolor from 1.1.2 to 1.1.3 (graphprotocol#4264) Bumps [termcolor](https://github.com/BurntSushi/termcolor) from 1.1.2 to 1.1.3. - [Release notes](https://github.com/BurntSushi/termcolor/releases) - [Commits](BurntSushi/termcolor@1.1.2...1.1.3) --- updated-dependencies: - dependency-name: termcolor dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Updated link to graph-cli networks list link (graphprotocol#4246) * Update protobuf-related dependencies (graphprotocol#4272) * ci: run code coverage stats every 3 days (graphprotocol#4188) * fix cloud build (graphprotocol#4279) * Fix attestability of ValueParseError, EnumCoercionError, ScalarCoercionError (graphprotocol#4278) * Fix attestability of ValueParseError * Fix attestability of EnumCoercionError, ScalarCoercionError * build(deps): bump git-testament from 0.2.0 to 0.2.2 (graphprotocol#4275) Bumps [git-testament](https://github.com/kinnison/git-testament) from 0.2.0 to 0.2.2. - [Release notes](https://github.com/kinnison/git-testament/releases) - [Commits](kinnison/git-testament@0.2.0...0.2.2) --- updated-dependencies: - dependency-name: git-testament dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Update ipfs api (graphprotocol#4283) * build(deps): bump blake3 from 1.3.2 to 1.3.3 (graphprotocol#4209) Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.3.2 to 1.3.3. - [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases) - [Commits](BLAKE3-team/BLAKE3@1.3.2...1.3.3) --- updated-dependencies: - dependency-name: blake3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: Include graft ipfs lookup for raw_yaml in infinite retries (graphprotocol#4284) * substreams: Fix spurious builds (graphprotocol#4286) * graphql: Allow gt/lt comparisons with `Bytes` fields Fixes graphprotocol#4282 * graph, store: Avoid a bind variable for fulltext queries We control the string we pass as the language, and therefore do not need to use a bind variable. We just need to make sure we are passing a valid SQL string constant * store: Properly account for fulltext columns in inserts Fixes graphprotocol#2330 * docker: adds support for ipfs auth urls (graphprotocol#4252) * `v0.29.0` release notes, `NEWS.md` and Cargo package version (graphprotocol#4298) * Release v0.29.0 * news: release v0.29.0 * build(deps): bump proc-macro2 from 1.0.49 to 1.0.50 (graphprotocol#4295) Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.49 to 1.0.50. - [Release notes](https://github.com/dtolnay/proc-macro2/releases) - [Commits](dtolnay/proc-macro2@1.0.49...1.0.50) --- updated-dependencies: - dependency-name: proc-macro2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump cid from 0.9.0 to 0.10.1 (graphprotocol#4289) Bumps [cid](https://github.com/multiformats/rust-cid) from 0.9.0 to 0.10.1. - [Release notes](https://github.com/multiformats/rust-cid/releases) - [Changelog](https://github.com/multiformats/rust-cid/blob/master/CHANGELOG.md) - [Commits](multiformats/rust-cid@v0.9.0...v0.10.1) --- updated-dependencies: - dependency-name: cid dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Cache recent blocks in memory to reduce load on the db (graphprotocol#4215) * store: cache recent blocks in memory * store: refactor RecentBlocksCache's Inner methods * store: fix ancestor/child naming in RecentBlocksCache * store: use a BTreeMap for RecentCachedBlocks This commit significatly alters the design of RecentCachedBlocks. The most prominent changes are: 1. We don't require a `.set_chain_head` call anymore. Block insertion and chain head update attempt are now the same thing. 2. We don't evict all items in the cache anymore every time the chain head advances. 3. Unlike the previous data structure, we are now limited to storing a contiguous range of blocks in the cache. This is not really a drawback (as the cache contents will usually be identical, i.e. the last N blocks before the chain head), but it's worth pointing out. * rust: upgrade to 1.66 * store: readability improvements to RecentBlocksCache * store: fix RecentBlocksCache for NEAR This commit removes the requirement of continuous block number ranges for RecentBlocksCache. Two reasons for that: - NEAR doesn't have continuous block number ranges, so the previous cache design would have limited effectiveness for NEAR chains. - Parent hash comparisons are actually enough to uphold all invariants, so block number checks were unnecessary in the first place. The actual code changes are small and mostly limited to `insert_block`. * build(deps): bump tower-test from `c9d84cd` to `b01bb12` (graphprotocol#4296) Bumps [tower-test](https://github.com/tower-rs/tower) from `c9d84cd` to `b01bb12`. - [Release notes](https://github.com/tower-rs/tower/releases) - [Commits](tower-rs/tower@c9d84cd...b01bb12) --- updated-dependencies: - dependency-name: tower-test dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump prost from 0.11.5 to 0.11.6 (graphprotocol#4297) Bumps [prost](https://github.com/tokio-rs/prost) from 0.11.5 to 0.11.6. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Commits](tokio-rs/prost@v0.11.5...v0.11.6) --- updated-dependencies: - dependency-name: prost dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump termcolor from 1.1.3 to 1.2.0 (graphprotocol#4294) Bumps [termcolor](https://github.com/BurntSushi/termcolor) from 1.1.3 to 1.2.0. - [Release notes](https://github.com/BurntSushi/termcolor/releases) - [Commits](BurntSushi/termcolor@1.1.3...1.2.0) --- updated-dependencies: - dependency-name: termcolor dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump atomic_refcell from 0.1.8 to 0.1.9 (graphprotocol#4290) Bumps [atomic_refcell](https://github.com/bholley/atomic_refcell) from 0.1.8 to 0.1.9. - [Release notes](https://github.com/bholley/atomic_refcell/releases) - [Commits](https://github.com/bholley/atomic_refcell/commits) --- updated-dependencies: - dependency-name: atomic_refcell dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump bumpalo from 3.7.0 to 3.12.0 (graphprotocol#4306) * Env. vars. for forking (graphprotocol#4308) * adding env params for forking opts * adding documentation Co-authored-by: Jeff Wu <jeffywu@pm.me> * build(deps): bump prost-types from 0.11.5 to 0.11.6 (graphprotocol#4312) * build(deps): bump git-testament from 0.2.2 to 0.2.4 (graphprotocol#4313) * Add causality region column and implement isolation rules (graphprotocol#4162) * feature(offchain): Add `causality_region` column to entity tables For now this just tracks the tables that need the column and adds the column to the DDL, but still unconditionally inserts 0. Inserting the correct causality region is follow up work. * store: Move `has_causality_region` to manifest, rename to `entities_with_causality_region` * *: Add `causality_region` to EntityKey The tricky part was changing `get_many` to return the entity key. * store: Insert the causality region * store: Read isolation between causality regions It was just necessary to make sure that `find` and `find_many` use the causality region in their where clause. * fix: Fix release build * provider: Make stop idempotent Callers wanted that anyways, and it helps tests. * tests: Refactor file ds test to use events * tests: Test conflict between onchain and offchain * tests: Test conflict between offchain and offchain * test: Fix unit test * tests: Improve tests and comments to address review * fix: Change migration to add column 'if not exists' * store: Add a materialized view 'info.chain_sizes' * config: Bump default ipfs timeout to 60 seconds (graphprotocol#4324) It used to be that 30 seconds was sufficient, but now we see ipfs requests taking minutes to find a file in the DHT. So 60 seconds seems like a reasonable step. * strip all null bytes from utf8 strings that come from substreams, like it's done on other sources (graphprotocol#4328) * fix test * add protobuf * fix database setup for unit test on kyber-ci --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Filippo Costa <filippocosta.italy@gmail.com> Co-authored-by: David Lutterkort <lutter@watzmann.net> Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com> Co-authored-by: Adam Fuller <azf20@users.noreply.github.com> Co-authored-by: Tiago Guimarães <tilacog@gmail.com> Co-authored-by: Filippo Costa <filippo@neysofu.me> Co-authored-by: Filippo Costa <filippo@edgeandnode.com> Co-authored-by: Filipe Azevedo <filipe@azevedo.io> Co-authored-by: Eva Pace <eba.pachi@gmail.com> Co-authored-by: Leonardo Yvens <leoyvens@gmail.com> Co-authored-by: Matthieu Vachon <matthieu.o.vachon@gmail.com> Co-authored-by: Matthieu Vachon <matt@streamingfast.io> Co-authored-by: Matthieu Vachon <matthieu.vachon@dfuse.io> Co-authored-by: Shude Li <islishude@gmail.com> Co-authored-by: Dotan Simha <dotansimha@gmail.com> Co-authored-by: mudabbirk <59587832+mudabbirk@users.noreply.github.com> Co-authored-by: Theo Butler <theodusbutler@gmail.com> Co-authored-by: Leo <leonard.mocanu@ymail.com> Co-authored-by: Jeff Wu <jeffywu@pm.me> Co-authored-by: Stéphane Duchesneau <stephane.duchesneau@gmail.com> Co-authored-by: Vu Tran <vutran@Vus-MacBook-Pro.lan> Co-authored-by: Vu Tran <vutran@Vus-MacBook-Pro.local>

leoyvens requested a review from mangas November 11, 2022 12:48

leoyvens mentioned this pull request Nov 11, 2022

Isolate onchain data sources from accessing and modifying offchain entities #3770

Closed

Base automatically changed from leo/enforce-enitites-on-offchain to master November 11, 2022 15:09

leoyvens force-pushed the leo/add-causality-region-column branch from 33e7ee7 to 5c8c6e0 Compare November 11, 2022 15:11

mangas force-pushed the leo/add-causality-region-column branch 4 times, most recently from e7e4149 to 72b9e38 Compare November 18, 2022 16:15

mangas force-pushed the leo/add-causality-region-column branch from 72b9e38 to 857dea5 Compare November 23, 2022 13:51

leoyvens force-pushed the leo/add-causality-region-column branch from 857dea5 to 453a386 Compare November 24, 2022 12:30

leoyvens force-pushed the leo/add-causality-region-column branch 2 times, most recently from 6ee05e2 to 02d0d12 Compare November 28, 2022 15:48

leoyvens force-pushed the leo/add-causality-region-column branch 2 times, most recently from a0b47b2 to 5ccf68f Compare December 9, 2022 14:46

mangas reviewed Dec 9, 2022

View reviewed changes

graph/src/components/store/mod.rs Show resolved Hide resolved

mangas reviewed Dec 9, 2022

View reviewed changes

mangas approved these changes Dec 9, 2022

View reviewed changes

leoyvens force-pushed the leo/add-causality-region-column branch 2 times, most recently from 2986760 to 07184bb Compare January 4, 2023 20:55

leoyvens changed the title ~~Add causality region column~~ Add causality region column and implement isolation rules Jan 4, 2023

leoyvens force-pushed the leo/add-causality-region-column branch 2 times, most recently from 594f919 to 65860c6 Compare January 5, 2023 00:47

mangas reviewed Jan 20, 2023

View reviewed changes

leoyvens added 13 commits January 23, 2023 20:38

feature(offchain): Add causality_region column to entity tables

074e99f

For now this just tracks the tables that need the column and adds the column to the DDL, but still unconditionally inserts 0. Inserting the correct causality region is follow up work.

store: Move has_causality_region to manifest, rename to `entities_w…

535f912

…ith_causality_region`

*: Add causality_region to EntityKey

648a6b7

The tricky part was changing `get_many` to return the entity key.

store: Insert the causality region

091155c

store: Read isolation between causality regions

1399f74

It was just necessary to make sure that `find` and `find_many` use the causality region in their where clause.

fix: Fix release build

c8f1698

provider: Make stop idempotent

bb6d9a5

Callers wanted that anyways, and it helps tests.

tests: Refactor file ds test to use events

9c7de71

tests: Test conflict between onchain and offchain

ea6ec6f

tests: Test conflict between offchain and offchain

5c27c39

test: Fix unit test

a06bc92

tests: Improve tests and comments to address review

0d0463a

fix: Change migration to add column 'if not exists'

7e6bc96

leoyvens force-pushed the leo/add-causality-region-column branch from a4f9c3a to 7e6bc96 Compare January 23, 2023 19:38

leoyvens mentioned this pull request Jan 25, 2023

store: Add a materialized view 'info.chain_sizes' #4318

Merged

leoyvens merged commit 04fbd9d into master Jan 25, 2023

leoyvens deleted the leo/add-causality-region-column branch January 25, 2023 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add causality region column and implement isolation rules #4162

Add causality region column and implement isolation rules #4162

leoyvens commented Nov 11, 2022 •

edited

Loading

lutter commented Nov 23, 2022

leoyvens commented Nov 24, 2022

mangas Dec 9, 2022

mangas Dec 9, 2022

mangas Dec 9, 2022 •

edited

Loading

leoyvens Jan 4, 2023

mangas Dec 9, 2022

mangas Dec 9, 2022

mangas Dec 9, 2022

mangas Dec 9, 2022

leoyvens Jan 4, 2023

mangas Dec 9, 2022

leoyvens Jan 4, 2023

mangas Dec 9, 2022

mangas Dec 9, 2022

leoyvens Jan 4, 2023

mangas Dec 9, 2022

mangas Dec 9, 2022

leoyvens Jan 4, 2023

mangas Dec 9, 2022

leoyvens Jan 5, 2023 •

edited

Loading

mangas Dec 9, 2022

leoyvens Jan 5, 2023

mangas Jan 20, 2023

mangas commented Dec 9, 2022

leoyvens commented Jan 5, 2023

mangas Jan 20, 2023

leoyvens Jan 23, 2023

	entities_with_causality_region: Vec<EntityType>,
	offchan_entities: Vec<EntityType>,

	/// not present and the causality region for all rows is implicitly `0`.
	/// not present and the causality region for all rows is implicitly `CausalityRegion::ONCHAIN` .

Add causality region column and implement isolation rules #4162

Add causality region column and implement isolation rules #4162

Conversation

leoyvens commented Nov 11, 2022 • edited Loading

lutter commented Nov 23, 2022

leoyvens commented Nov 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mangas Dec 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leoyvens Jan 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mangas commented Dec 9, 2022

leoyvens commented Jan 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leoyvens commented Nov 11, 2022 •

edited

Loading

mangas Dec 9, 2022 •

edited

Loading

leoyvens Jan 5, 2023 •

edited

Loading