Add erasure rust logic under feature flag #131

sakridge · 2018-04-18T22:16:28Z

No description provided.

codecov-io · 2018-04-18T22:27:04Z

Codecov Report

Merging #131 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #131   +/-   ##
=======================================
  Coverage   96.39%   96.39%           
=======================================
  Files          18       18           
  Lines        1886     1886           
=======================================
  Hits         1818     1818           
  Misses         68       68

Impacted Files	Coverage Δ
src/lib.rs	`100% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e11078...dea5ab2. Read the comment docs.

garious

The core code looks nice. The tests are quite verbose. Test code clarity is more valuable to me than either docs or core code clarity. It's our mechanically-verified specification and the thing I'll read closely any time refactoring causes test failures.

garious · 2018-04-18T22:51:04Z

src/erasure.rs

+//   coding blocks are present to restore
+pub fn recover(
+    re: &BlobRecycler,
+    window: &mut Vec<Option<SharedBlob>>,


The function allocates a new blob and sticks it into the window, so the window should be mut.

garious · 2018-04-18T22:53:25Z

src/erasure.rs

+    if data_missing > 0 {
+        if (data_missing + coded_missing) <= MAX_MISSING {
+            let mut blobs: Vec<SharedBlob> = Vec::new();
+            let mut locks = Vec::new();


This is a vector of guards, not locks.

Isn't it holding write locks on blobs?

They're guards, which are like auto_ptr in c++. I'd make more since to call the list blobs and the previous list blob_locks. In the following loop, you use the guard as though it is a blob, so blobs is a reasonable name. blob_guards would be good too.

garious · 2018-04-18T23:03:41Z

src/erasure.rs

+            for mut x in v.iter_mut() {
+                *x += i;
+            }
+            vs.push(v);


How about something like?

let mut vs: Vec<Vec<u8>> = (0..4).map(|i| (i .. (16 + i).collect()).collect();

garious · 2018-04-18T23:06:31Z

src/erasure.rs

+        for _i in 0..m {
+            let new_block = vec![0u8; v_orig.len()];
+            coding_blocks.push(new_block);
+        }


let coding_blocks = (0..2).map(|_| vec![0u8; 16]).collect();

garious · 2018-04-18T23:06:55Z

src/erasure.rs

+            let mut coding_blocks_slices = Vec::new();
+            for mut coding_block in coding_blocks.iter_mut() {
+                coding_blocks_slices.push(coding_block.as_mut_slice());
+            }


Also a map()

garious · 2018-04-18T23:07:05Z

src/erasure.rs

+            let mut v_slices = Vec::new();
+            for mut v in vs.iter() {
+                v_slices.push(v.as_slice());
+            }


This is a map() too

garious · 2018-04-18T23:08:10Z

src/erasure.rs

+            );
+        }
+        trace!("coding blocks:");
+        for b in coding_blocks.iter() {


In Rust, it's more idiomatic to write &coding_blocks instead of coding_blocks.iter() in for loops.

garious · 2018-04-18T23:08:34Z

src/erasure.rs

+            }
+            let mut v_slices = Vec::new();
+            for mut v in vs.iter_mut() {
+                v_slices.push(v.as_mut_slice());


This is a map()

garious · 2018-04-18T23:08:42Z

src/erasure.rs

+        {
+            let mut coding_blocks_slices = Vec::new();
+            for mut coding_block in coding_blocks.iter() {
+                coding_blocks_slices.push(coding_block.as_slice());


This is a map()

garious · 2018-04-18T23:11:18Z

Cargo.toml

@@ -8,6 +8,7 @@ repository = "https://github.com/solana-labs/solana"
 authors = [
    "Anatoly Yakovenko <anatoly@solana.co>",
    "Greg Fitzgerald <greg@solana.co>",


Can you switch Anatoly and me to .io while you're in the neighborhood?

garious · 2018-04-18T23:20:06Z

Also, it looks like the whole erasure.rs doesn't need to be under the feature flag. Looks like there's a good chunk of pure Rust in there with no FFI calls. That (and its tests) can be compiled in.

sakridge · 2018-04-19T00:13:12Z

@garious All the functions eventually call the FFI ones to do work. I'm not really sure what part could be run/tested without the flag.

only reroute if relayer connected (solana-labs#123) feat: add client tls config (solana-labs#121) remove extra val (solana-labs#129) fix clippy (solana-labs#130) copy all binaries to docker-output (solana-labs#131) Ledger tool halts at slot passed to create-snapshot (solana-labs#118) update program submodule (solana-labs#133) quick fix for tips and clearing old bundles (solana-labs#135) update submodule to new program (solana-labs#136) Improve stake-meta-generator usability (solana-labs#134) pinning submodule head (solana-labs#140) Use BundleAccountLocker when handling tip txs (solana-labs#147) Add metrics for relayer + block engine proxy (solana-labs#149) Build claim-mev in docker (solana-labs#141) Rework bundle receiving and add metrics (solana-labs#152) (solana-labs#154) update submodule + dev files (solana-labs#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (solana-labs#159) update jito-programs submodule (solana-labs#160) Separate MEV tip related workflow (solana-labs#161) Add block builder fee protos (solana-labs#162) fix jito programs (solana-labs#163) update submodule so autosnapshot exits out of ledger tool early (solana-labs#164) Pipe through block builder fee (solana-labs#167) pull in new snapshot code (solana-labs#171) block builder bug (solana-labs#172) Pull in new slack autosnapshot submodule (solana-labs#174) sort stake meta json and use int math (solana-labs#176) add accountsdb conn submod (solana-labs#169) Update tip distribution parameters (solana-labs#177) new submodules (solana-labs#180) Add buildkite link for jito CI (solana-labs#183) Fixed broken links to repositories (solana-labs#184) Changed from ssh to https transfer for clone Seg/update submods (solana-labs#187) fix tests (solana-labs#190) rm geyser submod (solana-labs#192) rm dangling geyser references (solana-labs#193) fix syntax err (solana-labs#195) use deterministic req ids in batch calls (solana-labs#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (solana-labs#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (solana-labs#203) Add logging for sol balance desired (solana-labs#205) * add logging * add logging * update msg * tweak vars update submodule (solana-labs#204) use efficient data structures when calling batch_simulate_bundles (solana-labs#206) [JIT-504] Add low balance check in uploading merkle roots (solana-labs#209) add config to simulate on top of working bank (solana-labs#211) rm frozen bank check simulate_bundle rpc bugfixes (solana-labs#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (solana-labs#210) * add files * switch to include bump update submodule (solana-labs#217) add amount filter (solana-labs#218) update autosnapshot (solana-labs#222) Print TX error in Bundles (solana-labs#223) add new args to support single relayer and block-engine endpoints (solana-labs#224) point to new jito-programs submod and invoke updated init tda instruction (solana-labs#228) fix clippy errors (solana-labs#230) fix validator start scripts (solana-labs#232) Point README to gitbook (solana-labs#237) use packaged cargo bin to build (solana-labs#239) Add validator identity pubkey to StakeMeta (solana-labs#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (solana-labs#240) Seg/tip distributor master (solana-labs#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (solana-labs#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (solana-labs#247) Improve docker caching delay constructing claim mev txs (solana-labs#253) fix stake meta tests from bb fee (solana-labs#254) fix tests Buffer bundles that exceed cost model (solana-labs#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (solana-labs#260) update Cargo.lock to correct solana versions in jito-programs submodule (solana-labs#265) fix simulate_bundle client and better error handling (solana-labs#267) update submod (solana-labs#272) Preallocate Bundle Cost (solana-labs#238) fix Dockerfile (solana-labs#278) Fix Tests (solana-labs#279) Fix Tests (solana-labs#281) * fix tests update jito-programs submod (solana-labs#282) add reclaim rent workflow (solana-labs#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (solana-labs#292) Remove security.md (solana-labs#293) demote frequent relayer_stage-stream_error to warn (solana-labs#275) account for case where TDA exists but not allocated (solana-labs#295) implement better retries for tip-distributor workflows (solana-labs#297) limit number of concurrent rpc calls (solana-labs#298) Discard Empty Packet Batches (solana-labs#299) Identity Hotswap (solana-labs#290) small fixes (solana-labs#305) Set backend config from admin rpc (solana-labs#304) Admin Shred Receiver Change (solana-labs#306) Seg/rm bundle UUID (solana-labs#309) Fix github workflow to recursively clone (solana-labs#327) Add recursive checkout for downstream-project-spl.yaml (solana-labs#341) Use cluster info functions for tpu (solana-labs#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (solana-labs#374) Updates bootstrap and start scripts needed for local dev. (solana-labs#384) Remove Deprecated Cli Args (solana-labs#387) Master Rebase improve simulate_bundle errors and response (solana-labs#404) derive Clone on accountoverrides (solana-labs#416) Add upsert to AccountOverrides (solana-labs#419) update jito-programs (solana-labs#430) [JIT-1661] Faster Autosnapshot (solana-labs#436) Reverts simulate_transaction result calls to upstream (solana-labs#446) Don't unlock accounts in TransactionBatches used during simulation (solana-labs#449) first pass at wiring up jito-plugin (solana-labs#428) [JIT-1713] Fix bundle's blockspace preallocation (solana-labs#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (solana-labs#491) [JIT-1710] - Optimize Bundle Consumer Checks (solana-labs#490) Add Blockhash Metrics to Bundle Committer (solana-labs#500) add priority fee ix to mev-claim (solana-labs#520) Update Autosnapshot (solana-labs#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (solana-labs#582) Update CI (solana-labs#584) - Add recursive submodule checkouts. - Re-add solana-secondary step - More release fixes Fix release process Backports solana-labs#595: correctly initialize account overrides (solana-labs#596) Fix: Ensure set contact info to UDP port instead of QUIC (Backport solana-labs#601 to v1.18) (solana-labs#602) Buffer bundles that exceed processing time and make the allowed processing time longer (solana-labs#610)

sakridge requested review from garious and aeyakovenko April 18, 2018 22:16

garious reviewed Apr 18, 2018

View reviewed changes

sakridge force-pushed the erasure branch from 05371dc to f7f5aec Compare April 19, 2018 00:05

garious approved these changes Apr 19, 2018

View reviewed changes

Add erasure rust logic under feature flag

dea5ab2

sakridge force-pushed the erasure branch from f7f5aec to dea5ab2 Compare April 19, 2018 02:50

garious merged commit e949211 into solana-labs:master Apr 19, 2018

This was referenced May 17, 2022

[Snyk] Upgrade rpc-websockets from 7.4.16 to 7.4.18 Balantion2020/Balantion#21

Open

[Snyk] Upgrade rpc-websockets from 7.4.16 to 7.4.18 Balantion2020/Balantion#30

Open

Balantion2020 mentioned this pull request May 26, 2022

[Snyk] Upgrade rpc-websockets from 7.4.16 to 7.4.18 Balantion2020/Balantion#42

Open

snyk-bot mentioned this pull request May 28, 2022

[Snyk] Upgrade rpc-websockets from 7.4.16 to 7.4.18 Balantion2020/Balantion#44

Open

snyk-bot mentioned this pull request Jun 10, 2022

[Snyk] Upgrade rpc-websockets from 7.4.16 to 7.4.18 Balantion2020/Balantion#88

Open

Balantion2020 mentioned this pull request Jun 24, 2022

[Snyk] Upgrade rpc-websockets from 7.4.16 to 7.5.0 Balantion2020/Balantion#127

Open

adamlaska mentioned this pull request Jul 10, 2022

[Snyk] Upgrade rpc-websockets from 7.4.14 to 7.5.0 adamlaska/solana#23

Closed

Digiminey mentioned this pull request Feb 2, 2024

[Snyk] Security upgrade @metaplex/js from 4.9.0 to 4.12.0 Digiminey/solana#91

Open

Digiminey mentioned this pull request Mar 15, 2024

[Snyk] Security upgrade @metaplex/js from 4.9.0 to 4.12.0 Digiminey/solana#94

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add erasure rust logic under feature flag #131

Add erasure rust logic under feature flag #131

sakridge commented Apr 18, 2018

codecov-io commented Apr 18, 2018 •

edited

Loading

garious left a comment

garious Apr 18, 2018

sakridge Apr 19, 2018

garious Apr 18, 2018

sakridge Apr 19, 2018

garious Apr 19, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious Apr 18, 2018

garious commented Apr 18, 2018

sakridge commented Apr 19, 2018

Add erasure rust logic under feature flag #131

Add erasure rust logic under feature flag #131

Conversation

sakridge commented Apr 18, 2018

codecov-io commented Apr 18, 2018 • edited Loading

Codecov Report

garious left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garious commented Apr 18, 2018

sakridge commented Apr 19, 2018

codecov-io commented Apr 18, 2018 •

edited

Loading