Improve carbon ingestion performance #1327

richardartoul · 2019-01-29T23:10:52Z

No description provided.

codecov · 2019-01-29T23:32:04Z

Codecov Report

Merging #1327 into master will decrease coverage by 8.2%.
The diff coverage is 64.2%.

@@           Coverage Diff            @@
##           master   #1327     +/-   ##
========================================
- Coverage    70.7%   62.4%   -8.3%     
========================================
  Files         814     806      -8     
  Lines       69314   68813    -501     
========================================
- Hits        49005   42961   -6044     
- Misses      17181   22818   +5637     
+ Partials     3128    3034     -94

Flag	Coverage Δ
#aggregator	`74.3% <ø> (-7.2%)`	⬇️
#cluster	`70.3% <ø> (-15.3%)`	⬇️
#collector	`78.4% <ø> (ø)`	⬆️
#dbnode	`75.4% <ø> (-5.6%)`	⬇️
#m3em	`68.3% <ø> (-4.9%)`	⬇️
#m3ninx	`72.9% <ø> (-1.7%)`	⬇️
#m3nsch	`28.4% <ø> (-22.8%)`	⬇️
#metrics	`17.8% <0%> (-0.1%)`	⬇️
#msg	`75% <ø> (ø)`	⬆️
#query	`45.7% <69.2%> (-18.6%)`	⬇️
#x	`72.7% <ø> (-3.2%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c85f85...99e7b41. Read the comment docs.

codecov · 2019-01-29T23:32:36Z

Codecov Report

Merging #1327 into master will increase coverage by <.1%.
The diff coverage is 64.2%.

@@           Coverage Diff            @@
##           master   #1327     +/-   ##
========================================
+ Coverage    70.5%   70.6%   +<.1%     
========================================
  Files         816     814      -2     
  Lines       69452   69316    -136     
========================================
- Hits        49033   48999     -34     
+ Misses      17291   17180    -111     
- Partials     3128    3137      +9

Flag	Coverage Δ
#aggregator	`81.4% <ø> (-0.1%)`	⬇️
#cluster	`85.6% <ø> (ø)`	⬆️
#collector	`78.4% <ø> (ø)`	⬆️
#dbnode	`80.9% <ø> (-0.1%)`	⬇️
#m3em	`73.2% <ø> (ø)`	⬆️
#m3ninx	`74.6% <ø> (ø)`	⬆️
#m3nsch	`51.1% <ø> (ø)`	⬆️
#metrics	`17.8% <0%> (-0.1%)`	⬇️
#msg	`74.9% <ø> (-0.2%)`	⬇️
#query	`64.4% <69.2%> (+0.4%)`	⬆️
#x	`75.9% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c85f85...8b9a7db. Read the comment docs.

scripts/development/m3_stack/docker-compose.yml

robskillington · 2019-01-30T14:38:25Z

src/cmd/services/m3coordinator/ingest/carbon/ingest.go

 	m3xserver "github.com/m3db/m3x/server"
 	xsync "github.com/m3db/m3x/sync"
 	xtime "github.com/m3db/m3x/time"

 	"github.com/uber-go/tally"
 )

+const (
+	maxResourcePoolNameSize = 200


Maybe bump this to 256 for "power of 2-ness". Also IDs can get long in graphite land, heh.

should I go higher than 256?

src/cmd/services/m3coordinator/ingest/carbon/ingest.go

src/cmd/tools/carbon_load/main/main.go

robskillington

LGTM other than nits

Richard Artoul added 8 commits January 29, 2019 17:45

squash

ba67135

fix brkoen pooling

de7c220

wip pooling

a8e7cdf

pool tags

596e417

add comment

bc9f053

delete binary

c7128ea

fix server.go

ba9843a

Dont reallocate tags

99e7b41

Richard Artoul added 2 commits January 30, 2019 09:30

add comments

cffc4bc

revert DOCKERFILE changes

4dc020d

robskillington reviewed Jan 30, 2019

View reviewed changes

scripts/development/m3_stack/docker-compose.yml Show resolved Hide resolved

robskillington reviewed Jan 30, 2019

View reviewed changes

change defaults and add comment

bbc3984

robskillington reviewed Jan 30, 2019

View reviewed changes

src/cmd/services/m3coordinator/ingest/carbon/ingest.go Outdated Show resolved Hide resolved

richardartoul changed the title ~~[WIP] - Improve carbon ingestion performance~~ Improve carbon ingestion performance Jan 30, 2019

robskillington reviewed Jan 30, 2019

View reviewed changes

src/cmd/services/m3coordinator/ingest/carbon/ingest.go Outdated Show resolved Hide resolved

Richard Artoul added 3 commits January 30, 2019 09:41

expose more dbnode ports in start_m3

efee18c

bump max id size to 256

59fb141

reset slice in append to be safe

254d9a8

robskillington reviewed Jan 30, 2019

View reviewed changes

src/cmd/services/m3coordinator/ingest/carbon/ingest.go Outdated Show resolved Hide resolved

Richard Artoul added 5 commits January 30, 2019 09:52

default to max pool name size of 1024

a7639cf

context per connection

c4f30ee

convert resource to pointer type

6d37520

remove TODO

59b6cc6

fix broken test

cb8d7a5

robskillington reviewed Jan 30, 2019

View reviewed changes

src/cmd/services/m3coordinator/ingest/carbon/ingest.go Show resolved Hide resolved

robskillington reviewed Jan 30, 2019

View reviewed changes

src/cmd/tools/carbon_load/main/main.go Outdated Show resolved Hide resolved

robskillington reviewed Jan 30, 2019

View reviewed changes

src/cmd/tools/carbon_load/main/main.go Outdated Show resolved Hide resolved

robskillington approved these changes Jan 30, 2019

View reviewed changes

address feedback

8b9a7db

richardartoul merged commit 876ae85 into master Jan 30, 2019

justinjc deleted the ra/carbon-bench branch June 17, 2019 21:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve carbon ingestion performance #1327

Improve carbon ingestion performance #1327

richardartoul commented Jan 29, 2019

codecov bot commented Jan 29, 2019

codecov bot commented Jan 29, 2019 •

edited

Loading

robskillington Jan 30, 2019

richardartoul Jan 30, 2019

robskillington left a comment

Improve carbon ingestion performance #1327

Improve carbon ingestion performance #1327

Conversation

richardartoul commented Jan 29, 2019

codecov bot commented Jan 29, 2019

Codecov Report

codecov bot commented Jan 29, 2019 • edited Loading

Codecov Report

robskillington Jan 30, 2019

Choose a reason for hiding this comment

richardartoul Jan 30, 2019

Choose a reason for hiding this comment

robskillington left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 29, 2019 •

edited

Loading