Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-723] add block number + stage to grpc updates #1252

Merged
merged 3 commits into from
Mar 27, 2024
Merged

[CT-723] add block number + stage to grpc updates #1252

merged 3 commits into from
Mar 27, 2024

Conversation

jayy04
Copy link
Contributor

@jayy04 jayy04 commented Mar 26, 2024

Changelist

[Describe or list the changes made in this PR]

Test Plan

[Describe how this PR was tested (if applicable)]

Author/Reviewer Checklist

  • If this PR has changes that result in a different app state given the same prior state and transaction list, manually add the state-breaking label.
  • If the PR has breaking postgres changes to the indexer add the indexer-postgres-breaking label.
  • If this PR isn't state-breaking but has changes that modify behavior in PrepareProposal or ProcessProposal, manually add the label proposal-breaking.
  • If this PR is one of many that implement a specific feature, manually label them all feature:[feature-name].
  • If you wish to for mergify-bot to automatically create a PR to backport your change to a release branch, manually add the label backport/[branch-name].
  • Manually add any of the following labels: refactor, chore, bug.

Copy link

linear bot commented Mar 26, 2024

Copy link
Contributor

coderabbitai bot commented Mar 26, 2024

Walkthrough

The recent updates aim to enhance the protocol's flexibility in handling order book updates by introducing additional context parameters and distinct execution modes. These changes provide more control over how order book updates are processed and dispatched, incorporating block height information and specific execution modes for different stages of protocol operations.

Changes

Files Change Summary
proto/dydxprotocol/clob/query.proto Added block_height and exec_mode fields to StreamOrderbookUpdatesResponse.
protocol/app/app.go Updated context with execution modes (ExecModeBeginBlock, ExecModeEndBlock, ExecModePrepareCheckState) for block handling and state checks.
protocol/lib/context.go Introduced custom execution modes ExecModeBeginBlock, ExecModeEndBlock, ExecModePrepareCheckState with values 100, 101, and 102.
protocol/mocks/ClobKeeper.go, protocol/mocks/MemClobKeeper.go Updated SendOrderbookUpdates function signature to include a ctx parameter.
protocol/streaming/grpc/..., protocol/streaming/grpc/types/manager.go, protocol/testutil/memclob/keeper.go Added blockHeight and execMode parameters in SendOrderbookUpdates function affecting order book update logic.
protocol/x/clob/... (multiple files) Updated SendOrderbookUpdates function across various components to include a ctx parameter, enhancing the handling of order book updates with the inclusion of execution context.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

@@ -1683,6 +1683,8 @@ func (app *App) PreBlocker(ctx sdk.Context, _ *abci.RequestFinalizeBlock) (*sdk.

// BeginBlocker application updates every begin block
func (app *App) BeginBlocker(ctx sdk.Context) (sdk.BeginBlock, error) {
ctx = ctx.WithExecMode(lib.ExecModeBeginBlock)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wanna do

	ctx = log.AddPersistentTagsToLogger(ctx,
		log.Handler, XXXXXX,
		log.BlockHeight, ctx.BlockHeight(),
	)

while you are at it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the strings not the ints though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean to move this from x/clob's begin blocker to here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can just add some tags to the app-level app.go file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

going to punt on this for now so that I can merge. let's do this in a separate PR

@@ -11,6 +11,13 @@ import (
"github.com/dydxprotocol/v4-chain/protocol/lib/log"
)

// Custom exec modes
const (
ExecModeBeginBlock = 100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using ints for these values?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are meant to augment these

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 5722446 and cccc397.
Files ignored due to path filters (1)
  • protocol/x/clob/types/query.pb.go is excluded by: !**/*.pb.go
Files selected for processing (15)
  • proto/dydxprotocol/clob/query.proto (1 hunks)
  • protocol/app/app.go (3 hunks)
  • protocol/lib/context.go (1 hunks)
  • protocol/mocks/ClobKeeper.go (1 hunks)
  • protocol/mocks/MemClobKeeper.go (1 hunks)
  • protocol/streaming/grpc/grpc_streaming_manager.go (3 hunks)
  • protocol/streaming/grpc/noop_streaming_manager.go (2 hunks)
  • protocol/streaming/grpc/types/manager.go (2 hunks)
  • protocol/testutil/memclob/keeper.go (1 hunks)
  • protocol/x/clob/abci.go (1 hunks)
  • protocol/x/clob/keeper/keeper.go (1 hunks)
  • protocol/x/clob/keeper/order_state.go (1 hunks)
  • protocol/x/clob/memclob/memclob.go (3 hunks)
  • protocol/x/clob/types/clob_keeper.go (1 hunks)
  • protocol/x/clob/types/mem_clob_keeper.go (1 hunks)
Additional comments: 34
protocol/streaming/grpc/types/manager.go (1)
  • 22-23: The addition of blockHeight and execMode parameters to the SendOrderbookUpdates method in the GrpcStreamingManager interface aligns with the PR's objectives to enhance gRPC updates with additional data fields. This change will require updates in all implementations of this interface to accommodate the new method signature. The choice of types for these parameters seems appropriate and well-justified.
protocol/streaming/grpc/noop_streaming_manager.go (1)
  • 33-34: The update to the SendOrderbookUpdates method in the NoopGrpcStreamingManager struct, adding blockHeight and execMode parameters, correctly reflects the changes made to the GrpcStreamingManager interface. This no-op implementation is consistent with the expected behavior of not performing any operation while fulfilling the interface contract.
protocol/lib/context.go (1)
  • 16-18: The introduction of custom execution modes (ExecModeBeginBlock, ExecModeEndBlock, and ExecModePrepareCheckState) with distinct values is a logical enhancement to the system's capabilities, likely supporting the broader goal of providing more context during block handling and state checks. This addition appears well-justified and consistent with the PR's objectives.
protocol/x/clob/types/mem_clob_keeper.go (1)
  • 119-119: The addition of the ctx sdk.Context parameter to the SendOrderbookUpdates method in the MemClobKeeper interface aligns with the PR's objectives to enhance context handling within the system. This change will require updates in all implementations of this interface to accommodate the new method signature. The inclusion of the context parameter is a standard and beneficial practice for passing request-scoped information.
protocol/x/clob/types/clob_keeper.go (1)
  • 150-150: The addition of the ctx sdk.Context parameter to the SendOrderbookUpdates method in the ClobKeeper interface aligns with the PR's objectives to enhance context handling within the system. This change will require updates in all implementations of this interface to accommodate the new method signature. The inclusion of the context parameter is a standard and beneficial practice for passing request-scoped information.
protocol/streaming/grpc/grpc_streaming_manager.go (1)
  • 116-125: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [80-122]

The update to the SendOrderbookUpdates method in the GrpcStreamingManagerImpl struct, adding blockHeight and execMode parameters, correctly reflects the changes made to the GrpcStreamingManager interface. The method's updated logic to incorporate these parameters into the StreamOrderbookUpdatesResponse aligns with the PR's objectives to enhance the granularity and context of order book updates streamed via gRPC. This implementation is well-executed and enhances the system's capabilities.

proto/dydxprotocol/clob/query.proto (1)
  • 159-162: The addition of block_height and exec_mode fields to the StreamOrderbookUpdatesResponse message significantly enhances the protocol's gRPC streaming capabilities by providing additional context regarding the state of the order book at the time of the update. These changes are well-aligned with the PR's objectives and are correctly implemented within the protobuf definition.
protocol/x/clob/keeper/keeper.go (1)
  • 236-254: The update to the SendOrderbookUpdates method in the Keeper struct, adding a ctx sdk.Context parameter, and its updated invocation in InitializeNewGrpcStreams correctly reflect the changes made to enhance context handling within the system. This change enhances the method's capability to provide more detailed context in orderbook updates, aligning with the PR's objectives. The implementation is well-executed and enhances the system's capabilities.
protocol/x/clob/abci.go (1)
  • 178-178: The addition of the context parameter ctx to the SendOrderbookUpdates function call aligns with the PR objectives of enhancing the gRPC updates mechanism. This change is crucial for incorporating the additional context into the streaming updates mechanism.
protocol/x/clob/keeper/order_state.go (1)
  • 303-303: The addition of the context parameter ctx to the SendOrderbookUpdates function call within the PruneStateFillAmountsForShortTermOrders function is a necessary update to enhance the gRPC updates mechanism. This change ensures that the additional context is seamlessly integrated into the protocol's operational flow.
protocol/testutil/memclob/keeper.go (1)
  • 516-516: The addition of the context parameter ctx to the SendOrderbookUpdates function in the FakeMemClobKeeper struct is consistent with the overarching goal of enhancing the gRPC updates mechanism. This change ensures that the test utilities are aligned with the modifications made to the main implementation.
protocol/mocks/MemClobKeeper.go (1)
  • 437-439: The addition of the ctx parameter to the SendOrderbookUpdates function aligns with the PR objectives of enhancing the gRPC updates mechanism by introducing additional data fields. This change ensures that the mock implementation remains consistent with the actual interface, which is crucial for accurate testing and simulation. It's important to ensure that all usages of this mock function in tests are updated accordingly to include the new ctx parameter.
protocol/mocks/ClobKeeper.go (1)
  • 1138-1140: The addition of the ctx parameter to the SendOrderbookUpdates function aligns well with Go's conventions for context passing and supports the PR's objectives for enhanced context handling. This change is correctly implemented.
protocol/x/clob/memclob/memclob.go (21)
  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [72-85]

The constructor NewMemClobPriceTimePriority initializes generateOrderbookUpdates to false directly. If this behavior is intentional and critical to the system's operation, consider documenting this choice for clarity.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [87-91]

The SetClobKeeper method correctly sets the clobKeeper field. This is a straightforward setter method with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [93-97]

The SetGenerateOrderbookUpdates method correctly sets the generateOrderbookUpdates field. This is a straightforward setter method with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [99-169]

The CancelOrder method correctly handles the cancellation of short-term orders, including state updates and off-chain message generation. However, its complexity could impact maintainability. Consider refactoring it into smaller functions for better readability and maintainability. Additionally, ensure consistent use of telemetry counters throughout the codebase.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [171-185]

The CreateOrderbook method correctly initializes an orderbook for a given clobPairId. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [187-207]

The CountSubaccountShortTermOrders method correctly counts the number of open short-term orders for a given subaccount. However, the method name could be more descriptive of its specific functionality. Consider renaming it to CountSubaccountOpenShortTermOrders for clarity.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [209-215]

The GetOrder method correctly retrieves an order by its ID and returns it along with a boolean indicating whether the order was found. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [217-223]

The GetCancelOrder method correctly retrieves the tilBlock expiry of an order cancellation and returns a boolean indicating whether the expiry exists. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [225-237]

The GetOrderFilledAmount method correctly retrieves the total filled amount of an order from state. It appropriately handles the case where the order does not exist in state by returning a filled amount of 0. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [239-257]

The GetSubaccountOrders method correctly retrieves all of a subaccount's orders on a specific CLOB and side. However, it could potentially throw a panic if the side is invalid or the orderbook does not exist. Consider improving error handling by returning an error in these cases for better robustness and clarity.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [259-269]

The InsertZeroFillDeleveragingIntoOperationsQueue method correctly inserts a zero-fill deleveraging operation into the operations queue. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [271-429]

The mustUpdateMemclobStateWithMatches method correctly updates the memclob state by applying matches to all bookkeeping data structures. However, its complexity could impact maintainability. Consider refactoring it into smaller functions for better readability and maintainability. Additionally, consider improving error handling by using error returns instead of panics where possible.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [431-477]

The SetMemclobGauges method correctly sets gauges for each orderbook and the operations queue based on current memclob state. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [479-507]

The mustRemoveOrder method correctly removes an order from all data structures for tracking open orders in the memclob. However, it uses panic for error handling if the order does not exist. Consider improving error handling by returning an error in this case for better robustness and clarity.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [509-539]

The mustUpdateOrderbookStateWithMatchedMakerOrder method correctly updates the orderbook with a matched maker order, including removing the order if it's fully filled. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [541-548]

The updateResultToOrderStatus helper function correctly translates the result of a collateralization check into a resulting order status. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [550-564]

The GetOrderRemainingAmount method correctly calculates the remaining amount of an order by subtracting its filled amount from its size. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [566-588]

The RemoveOrderIfFilled method correctly removes an order from the orderbook if it's fully filled in state. However, it uses panic for error handling if the order has no remaining amount. Consider improving error handling by returning an error in this case for better robustness and clarity.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [590-628]

The maybeCancelReduceOnlyOrders method correctly cancels all open reduce-only orders on the CLOB pair if the new fill would change the position side of the subaccount. This is a straightforward implementation with no issues.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [630-650]

The GetMidPrice method correctly calculates the mid price of the orderbook for a given clob pair. However, consider reviewing the use of telemetry counters in this context to ensure it aligns with the overall telemetry strategy.

  • 1523-1529: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [652-688]

The getImpactPriceSubticks method correctly calculates the impact ask or bid price given the clob pair and orderbook. However, its complexity could impact maintainability. Consider refactoring it into smaller functions for better readability and maintainability.

Comment on lines 1683 to 1690

// BeginBlocker application updates every begin block
func (app *App) BeginBlocker(ctx sdk.Context) (sdk.BeginBlock, error) {
ctx = ctx.WithExecMode(lib.ExecModeBeginBlock)

// Update the proposer address in the logger for the panic logging middleware.
proposerAddr := sdk.ConsAddress(ctx.BlockHeader().ProposerAddress)
middleware.Logger = ctx.Logger().With("proposer_cons_addr", proposerAddr.String())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 NOTE
This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [738-738]

The gRPC server is initialized without specifying credentials, which could potentially expose the server to unencrypted connections. It's recommended to secure the gRPC server by using SSL/TLS credentials.

- app.Server = daemonserver.NewServer(logger, grpc.NewServer(), &daemontypes.FileHandlerImpl{}, daemonFlags.Shared.SocketAddress)
+ creds, err := credentials.NewServerTLSFromFile("cert.pem", "cert.key")
+ if err != nil {
+     log.Fatalf("Failed to generate credentials %v", err)
+ }
+ app.Server = daemonserver.NewServer(logger, grpc.NewServer(grpc.Creds(creds)), &daemontypes.FileHandlerImpl{}, daemonFlags.Shared.SocketAddress)

offchainUpdates *types.OffchainUpdates,
snapshot bool,
) {
if len(offchainUpdates.Messages) == 0 {
return
}

k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates, snapshot)
k.GetGrpcStreamingManager().SendOrderbookUpdates(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't really feel strongly at all but why don't we pass in the ctx instead of the blockheight and exec mode

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can abstract away the context from the streaming manager. logic to get block height/exec mode doesn't really belong there.

let me know if you have a preference

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between cccc397 and e80d76a.
Files selected for processing (1)
  • indexer/packages/v4-protos/src/codegen/dydxprotocol/clob/query.ts (6 hunks)
Additional comments: 3
indexer/packages/v4-protos/src/codegen/dydxprotocol/clob/query.ts (3)
  • 236-244: The addition of blockHeight and execMode fields to StreamOrderbookUpdatesResponse aligns with the PR objectives to enhance gRPC updates with additional data fields. Ensure that the data types of these fields are consistent with their expected values and usage throughout the application.
  • 262-270: Similarly, the addition of block_height and exec_mode to StreamOrderbookUpdatesResponseSDKType is consistent with the changes made to the non-SDK type version. It's good practice to maintain parallel structure between these types for clarity and maintainability.
  • 938-951: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [925-991]

Initialization of blockHeight and execMode to 0 in createBaseStreamOrderbookUpdatesResponse is appropriate for numeric fields. However, consider documenting the significance of the initial value 0 for these fields, especially if they have specific meanings or implications in the context of order book updates.

Consider adding comments to explain the choice of initial values for blockHeight and execMode, especially if 0 has specific implications in the application's context.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between e80d76a and c5be93e.
Files selected for processing (1)
  • protocol/x/clob/abci.go (1 hunks)
Files skipped from review as they are similar to previous changes (1)
  • protocol/x/clob/abci.go

@jayy04 jayy04 merged commit 9d1076f into main Mar 27, 2024
32 of 33 checks passed
@jayy04 jayy04 deleted the jy/ct-723 branch March 27, 2024 12:46
jayy04 added a commit that referenced this pull request Mar 28, 2024
* [CT-723] add block number + stage to grpc updates

* add indexer changes
jayy04 added a commit that referenced this pull request Mar 28, 2024
* [CT-723] add block number + stage to grpc updates

* add indexer changes
jayy04 added a commit that referenced this pull request Mar 28, 2024
* [CT-645] Move off chain updates and v1 to a different package (#1131)

* [CT-645] Add protos for orderbook stream query service

* move removal reasons to a separate package

* [CT-645] Add protos for orderbook stream query service (#1133)

* [CT-645] Add protos for orderbook stream query service

* make update not nullable

* fix build

* [CT-644] instantiate grpc stream manager (#1134)

* [CT-644] instantiate grpc stream manager

* update type

* update channel type

* [CT-646] stream offchain updates through stream manager (#1138)

* [CT-646] stream offchain updates through stream manager

* comments

* fix lint

* get rid of finished

* comments

* comments

* [CT-652] add command line flag for full node streaming (#1145)

* [CT-647] construct the initial orderbook snapshot (#1147)

* [CT-647] construct the initial orderbook snapshot

* [CT-647] initialize new streams and send orderbook snapshot (#1152)

* [CT-647] initialize new streams and send orderbook snapshot

* use sync once

* comments

* [CT-700] separate indexer and grpc streaming events (#1209)

* [CT-700] separate indexer and grpc streaming events

* fix tests

* comments

* update

* [CT-700] only send response when there is at least one update (#1216)

* [CT-712] send order update when short term order state fill amounts are pruned (#1241)

* [CT-712] send fill amount updates for reverted operations (#1240)

* [CT-723] add block number + stage to grpc updates (#1252)

* [CT-723] add block number + stage to grpc updates

* add indexer changes

* [CT-727] avoid state reads when sending updates (#1261)
jayy04 added a commit that referenced this pull request Apr 1, 2024
…#1269)

* [CT-700] separate indexer and grpc streaming events (#1209)

* [CT-700] separate indexer and grpc streaming events

* fix tests

* comments

* update

* [CT-700] only send response when there is at least one update (#1216)

* [CT-712] send order update when short term order state fill amounts are pruned (#1241)

* [CT-712] send fill amount updates for reverted operations (#1240)

* [CT-723] add block number + stage to grpc updates (#1252)

* [CT-723] add block number + stage to grpc updates

* add indexer changes

* [CT-727] avoid state reads when sending updates (#1261)

* fix lint|

* [CT-712] send updates for both normal order matches and liquidation (#1280)

* fix test

* fix test

* update type
dydxwill pushed a commit that referenced this pull request Apr 8, 2024
* [CT-645] Move off chain updates and v1 to a different package (#1131)

* [CT-645] Add protos for orderbook stream query service

* move removal reasons to a separate package

* [CT-645] Add protos for orderbook stream query service (#1133)

* [CT-645] Add protos for orderbook stream query service

* make update not nullable

* fix build

* [CT-644] instantiate grpc stream manager (#1134)

* [CT-644] instantiate grpc stream manager

* update type

* update channel type

* [CT-646] stream offchain updates through stream manager (#1138)

* [CT-646] stream offchain updates through stream manager

* comments

* fix lint

* get rid of finished

* comments

* comments

* [CT-652] add command line flag for full node streaming (#1145)

* [CT-647] construct the initial orderbook snapshot (#1147)

* [CT-647] construct the initial orderbook snapshot

* [CT-647] initialize new streams and send orderbook snapshot (#1152)

* [CT-647] initialize new streams and send orderbook snapshot

* use sync once

* comments

* [CT-700] separate indexer and grpc streaming events (#1209)

* [CT-700] separate indexer and grpc streaming events

* fix tests

* comments

* update

* [CT-700] only send response when there is at least one update (#1216)

* [CT-712] send order update when short term order state fill amounts are pruned (#1241)

* [CT-712] send fill amount updates for reverted operations (#1240)

* [CT-723] add block number + stage to grpc updates (#1252)

* [CT-723] add block number + stage to grpc updates

* add indexer changes

* [CT-727] avoid state reads when sending updates (#1261)
dydxwill added a commit that referenced this pull request Apr 8, 2024
* [OTE-221] Add query for PendingSendPacket (backport #1176) (#1221)


---------

Co-authored-by: Teddy Ding <teddy@dydx.exchange>
(cherry picked from commit e545bbf)

# Conflicts:
#	indexer/packages/v4-protos/src/codegen/dydxprotocol/bundle.ts
#	indexer/packages/v4-protos/src/codegen/gogoproto/bundle.ts
#	indexer/packages/v4-protos/src/codegen/google/bundle.ts
#	protocol/go.mod

* fix protos

* update go.mod

---------

Co-authored-by: Mohammed Affan <affanmd@nyu.edu>
Co-authored-by: affan <affan@dydx.exchange>

* [Backport v4.x] backport full node streaming to v4.x branch (#1270)

* [CT-645] Move off chain updates and v1 to a different package (#1131)

* [CT-645] Add protos for orderbook stream query service

* move removal reasons to a separate package

* [CT-645] Add protos for orderbook stream query service (#1133)

* [CT-645] Add protos for orderbook stream query service

* make update not nullable

* fix build

* [CT-644] instantiate grpc stream manager (#1134)

* [CT-644] instantiate grpc stream manager

* update type

* update channel type

* [CT-646] stream offchain updates through stream manager (#1138)

* [CT-646] stream offchain updates through stream manager

* comments

* fix lint

* get rid of finished

* comments

* comments

* [CT-652] add command line flag for full node streaming (#1145)

* [CT-647] construct the initial orderbook snapshot (#1147)

* [CT-647] construct the initial orderbook snapshot

* [CT-647] initialize new streams and send orderbook snapshot (#1152)

* [CT-647] initialize new streams and send orderbook snapshot

* use sync once

* comments

* [CT-700] separate indexer and grpc streaming events (#1209)

* [CT-700] separate indexer and grpc streaming events

* fix tests

* comments

* update

* [CT-700] only send response when there is at least one update (#1216)

* [CT-712] send order update when short term order state fill amounts are pruned (#1241)

* [CT-712] send fill amount updates for reverted operations (#1240)

* [CT-723] add block number + stage to grpc updates (#1252)

* [CT-723] add block number + stage to grpc updates

* add indexer changes

* [CT-727] avoid state reads when sending updates (#1261)

* [CT-712] send updates for both normal order matches and liquidation (#1280) (#1281)

* Fix lib.ErrorLogWithError: properly pass in args (#1306) (#1310)

(cherry picked from commit a91c1ca)

Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com>

* fix broken tests (#1312) (#1316)

(cherry picked from commit 5ec37d2)

Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com>

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Mohammed Affan <affanmd@nyu.edu>
Co-authored-by: affan <affan@dydx.exchange>
Co-authored-by: jayy04 <103467857+jayy04@users.noreply.github.com>
Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

2 participants