Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification #17559

jpeletier · 2018-08-31T09:27:31Z

Note: Discussion and review of this PR happened in 3 steps here:

manifestless updates - multihash removal [PREVIEW] MRU code simplification. Predictable update lookup addresses. Multihash and metadata chunk removal. ethersphere/swarm#813
query strings [PREVIEW] MRU query strings and manifest-less queries ethersphere/swarm#839
adaptive frequency - epoch based addressing [PREVIEW] Adaptive frequency MRUs ethersphere/swarm#881

Issue for roadmap tracking: ethersphere/swarm#910

Abstract

The current MRU implementation requires users to agree upon a predefined frequency and start time to start publishing updates about a certain topic. This causes lots of problems if that update frequency is not honored and requires users to know other user's update frequencies / start times in order to look up their updates on common topics

This PR removes this limitation via a novel adaptive frequency resource lookup algorithm. This algorithm automatically adjusts to the publisher's actual update frequency and converges quickly whether an update is found or not.

Users "following" a publisher automatically "tune" to the perceived frequency and can guess easily where the next update ought to be, meaning that subsequent lookups to get a newer update run faster or can be prefetched. This also allows to monitor a resource easily.

The Swarm team is working on a paper to describe this approach. In the meantime, the algorithm is described below.

As a result, interactions with Swarm's MRUs are greatly simplified since the user doesn't have to come up with a start time and frequency upfront, but rather start publishing updates about the topic they want.

In addition, this PR brings to Swarm:

Removed multihash flag in MRUs. bzz: scheme will detect multihashes automatically
Removed the need to publish a "metachunk" in order to create a resource
Increased test coverage
Simplified API
Predictable update lookup keys
Readable resource manifest JSON contains information about the resource lookup information
Code simplification throughout: No multihash and not having to deal with metadata chunks greatly simplified code.

Terminology

Topic: 32-byte arbitrary byte array specifying what the MRU contains information about or acting as a "meeting point"
UserAddress: Public address of any user.
Epoch time span at a specific frequency level. See Adaptive frequency algorithm below.
View: Concatenation of (Topic and UserAddress): This represents a particular user's updates (point of view) about a specific topic.
UpdateAddr: Hash of (View, Epoch): This allows to look up a user's series of updates over time.

What applications does this PR enable?

Data feeds, microblogging applications, metadata feeds about specific content...

API changes

HTTP API

To publish an update:

1.- Get resource metainformation

GET /bzz-resource:/?topic=<TOPIC>&user=<USER>&meta=1
GET /bzz-resource:/<MANIFEST OR ENS NAME>/?meta=1

Where:

user: Ethereum address of the user who publishes the resource
topic: Resource topic, encoded as a hex string.

Note:

If topic is omitted, it is assumed to be zero, 0x000...

if name=<name> is provided, a subtopic is composed with that name

A common use is to omit topic and just use name, allowing for human-readable topics.

You will receive a JSON like the below:

{
  "view": {
    "topic": "0x6a61766900000000000000000000000000000000000000000000000000000000",
    "user": "0xdfa2db618eacbfe84e94a71dda2492240993c45b"
  },
  "epoch": {
    "level": 16,
    "time": 1534237239
  }
}

2.- Post the update

Extract the fields out of the JSON and build a query string as below:

POST /bzz-resource:/?topic=<TOPIC>&user=<USER>&level=<LEVEL>&time=<TIME>&signature=<SIGNATURE>

body: binary stream with the update data.

(more information about what each of these fields are in the adaptive frequency algorithm below)

To get the last update:

GET /bzz-resource:/?topic=<TOPIC>&user=<USER>
GET /bzz-resource:/<MANIFEST OR ENS NAME>

Note:

Again, if topic is omitted, it is assumed to be zero, 0x000...

if name=<name> is provided, a subtopic is composed with that name

A common use is to omit topic and just use name, allowing for human-readable topics.
Thus, this is also valid: GET /bzz-resource:/?name=profile-picture&user=<USER>

To get a previous update:

Add an addtional time parameter. The last update before that time will be looked up.

GET /bzz-resource:/?topic=<TOPIC>&user=<USER>&time=<T>
GET /bzz-resource:/<MANIFEST OR ENS NAME>?time=<T>

Advanced search:

If you have an idea of when the last update happened, you can also hint the lookup algorithm by adding the following extra parameters:

hint.time: Time at when you think the last update happened
hint.level: Integer. Approximate period you think the updates where happening at, expressed as log2(T), rounded up. For example, a resource updating every 300 seconds, level should be set to 9. log2(300) = 8.22. See the Adaptive Frequency algorithm below for details on this.

Note that this would only affect first lookups. Your swarm node will keep track of last updates and automatically use the last seen update as a hint. Using these parameters would override that automatic hint.

To publish a manifest:

POST /bzz-resource:/?topic=<TOPIC>&user=<USER>&manifest=1 with an empty body.

Note: this functionality could be moved to the client and removed from the node, since this just creates a JSON and publishes it to bzz-raw, so the client could actually create this itself and call client.UploadRaw(). Don't expect this call to be available in future releases.

CLI

Creating a resource manifest:

swarm resource create is redefined as a command line to create and publish a MRU manifest only.

swarm resource create [command options]

creates and publishes a new Mutable Resource manifest pointing to a specified user's updates about a particular topic.
          The topic can be specified directly with the --topic flag as an hex string
          If no topic is specified, the default topic (zero) will be used
          The --name flag can be used to specify subtopics with a specific name
          The --user flag allows to have this manifest refer to a user other than yourself. If not specified,
          it will then default to your local account (--bzzaccount)

OPTIONS:
--name value   User-defined name for the new resource, limited to 32 characters. If combined with topic, the resource will be a subtopic with this name
--topic value  User-defined topic this resource is tracking, hex encoded. Limited to 64 hexadecimal characters
--user value   Indicates the user who updates the resource

Update a resource

swarm resource update [command options] <0x Hex data>

creates a new update on the specified topic
          The topic can be specified directly with the --topic flag as an hex string
          If no topic is specified, the default topic (zero) will be used
          The --name flag can be used to specify subtopics with a specific name.
          If you have a manifest, you can specify it with --manifest instead of --topic / --name
          to refer to the resource

OPTIONS:
--manifest value  Refers to the resource through a manifest
--name value      User-defined name for the new resource, limited to 32 characters. If combined with topic, the resource will be a subtopic with this name
--topic value     User-defined topic this resource is tracking, hex encoded. Limited to 64 hexadecimal characters

Quick and dirty test:

In the example, the user wants to publish his/her profile picture so it can be found by anyone who knows his/her Ethereum address.

# (OPTIONAL) Publish a manifest:
swarm --bzzaccount "your public key" resource create --name "profile-picture"

# the above command will output a manifest hash. We will refer to it as $MH later on

# Publish the first update:
$ IMAGE=$(swarm up myprofilepicture.jpg) && swarm --bzzaccount "<YOUR ADDRESS>" resource update --name "profile-picture" "0x1b20$IMAGE"

# To retrieve the latest update:
$ curl 'http://localhost:8500/bzz-resource:/?name=profile-picture&user=<YOUR ADDRESS>'

# Alternatively, if we created a manifest, we can use it:
$ IMAGE=$(swarm up myprofilepicture.jpg) && swarm --bzzaccount "your public key" resource update --manifest "$MH" "0x1b20$IMAGE"

# Your last profile picture can be viewed in:
# http://localhost:8500/bzz:/$MH
# (Only if a manifest was published)

Adaptive frequency lookup algorithm

At the core of this PR is a new lookup algorithm with the following properties:

Does not require the user to commit to an update frequency without affecting lookup time linearly with the time difference since now and when that last update happened.
The algorithm finishes quickly if no updates found
Once the last update is found, subsequent lookups take less time.
If we have a rough idea of where the last update was, we can hint the algorithm to get a faster lookup time.
It allows time-based lookups.
The lookup key of the next update can be predicted and monitored.

Revamping the resource frequency concept

Note: Starting with this PR, in the documentation and this text, we use the strict definition of frequency as f = 1 / T. Thus, higher frequencies mean shorter periods of time.

In this new implementation, period lengths are expressed as powers of 2. The highest frequency (shortest period, update every second) is expressed as 2⁰ = 1 second. The lowest update frequency is currently set to 2²⁵ = 33554432 seconds which equals to roughly one year.

Therefore, the frequency can be encoded as just the exponent. We call this exponent frequency level, or level for short. A higher level means a longer period and thus a smaller frequency.

Introducing Epochs

Now that we have determined a set of finite possible frequencies, we can divide time in a grid of epochs. One epoch is a concrete time range at a specific frequency level, starting at a specific point in time, called the epoch base time. Level 0 epochs have a maximum length of 2⁰ = 1 seconds. Level 3 epochs have a maximum length of 2³ = 8 seconds, etc.

To refer to a specific epoch, or epoch ID we need to know the epoch base time and the epoch level

We will use this epoch addressing scheme to derive a chunk address in which to store a particular update.

Epoch base time

To caclculate the epoch base time of any given instant in time at a particular level, we use the simple formula:

baseTime(t, level) = t & ( 0xFFFFFFFFFFFFFFFF << level )

In other words, we are dropping the level lowest significant bits of t.

Seeding algorithm

The seeding algorithm describes the process followed by the update publisher to determine in what epoch "plant" the content so it can be found (harvested) by users. The algorithm works as follows:

First updates

Any first resource update will have a level of 25.

Note: We have chosen 25 as the highest level. This is a constant in code and can be changed.

Thus, if as of writing this it is August 12th 2018 at 16:51 UTC, Unix Time is 1534092715. Therefore, the epoch base time is 1534092715 & 0xFFFFFFFFFE000000 = 1509949440

The epoch id for a first update now is therefore (1509949440, 25)

Subsequent updates

To determine the epoch in which to store a subsequent update, the publisher needs to know where they stored the previous update. This should be straightforward. However, if the publisher can't or does not want to keep track of this, it can always use the harvesting algorithm (see below) to find their last update.

The selected epoch for a subsequent update must be the epoch with the highest possible level that is not already occupied by a previous update.

Let's say that we want to update our resource 5 minutes later. The Unix Time is now 1534093015.

We calculate getBaseTime(1534093015, 25) = 1509949440.
This results in the same epoch as before (1534093015, 25). Therefore, we decrease the level and calculate again:
getBaseTime(1534093015, 24) = 1526726656

Thus, the next update will be located at (1526726656, 24)

If the publisher keeps updating the resource exactly every 5 minutes, the epoch grid will look like this:

update #1,  t=1534092715, epoch=(1509949440, 25)
update #2,  t=1534093015, epoch=(1526726656, 24)
update #3,  t=1534093315, epoch=(1526726656, 23)
update #4,  t=1534093615, epoch=(1530920960, 22)
update #5,  t=1534093915, epoch=(1533018112, 21)
update #6,  t=1534094215, epoch=(1534066688, 20)
update #7,  t=1534094515, epoch=(1534066688, 19)
update #8,  t=1534094815, epoch=(1534066688, 18)
update #9,  t=1534095115, epoch=(1534066688, 17)
update #10, t=1534095415, epoch=(1534066688, 16)
update #11, t=1534095715, epoch=(1534066688, 15)
update #12, t=1534096015, epoch=(1534083072, 14)
update #13, t=1534096315, epoch=(1534091264, 13)
update #14, t=1534096615, epoch=(1534095360, 12)
update #15, t=1534096915, epoch=(1534095360, 11)
update #16, t=1534097215, epoch=(1534096384, 10)
update #17, t=1534097515, epoch=(1534096896, 9)
update #18, t=1534097815, epoch=(1534097408, 11)
update #19, t=1534098115, epoch=(1534097408, 10)
update #20, t=1534098415, epoch=(1534097920, 9)
update #21, t=1534098715, epoch=(1534098176, 8)
update #22, t=1534099015, epoch=(1534098432, 10)
update #23, t=1534099315, epoch=(1534098944, 9)
update #24, t=1534099615, epoch=(1534099200, 8)
update #25, t=1534099915, epoch=(1534099456, 15)
update #26, t=1534100215, epoch=(1534099456, 14)
update #27, t=1534100515, epoch=(1534099456, 13)
update #28, t=1534100815, epoch=(1534099456, 12)
update #29, t=1534101115, epoch=(1534099456, 11)
update #30, t=1534101415, epoch=(1534100480, 10)

If the publisher keeps updating every 5 minutes (300s), we can expect the updates to stay around level 8-9 (2⁸ = 256 seconds, 2⁹ = 512 seconds). The publisher can however, at any time vary this update frequency or just update randomly. This does not affect the algorithm.

Here is a closer look at the converging levels further down:

Harvesting algorithm

The harvesting algorithm describes how to find the latest update of a resource. This involves looking up an epoch and walking back in time until we find the last update.

Start Epoch

To select the best starting epoch to walk our grid, we have to assume the worst case, which is that the resource was never updated after we last saw it.

If we don't know when the resource was last updated, we asume 0 as the "last time" it was updated.

We can guess a start level as the position of the nonzero bit of XOR(last base time, now) counting from the left. The bigger the difference among the two times (last update time and now), the higher the level will be as the update frequency we are estimating is lower.

If the resulting level is higher than 25, we use 25.

Walking the grid - a simple algorithm

Consider the following grid. In it we have marked in yellow where the updates have happened in the past.

All the above is unknown to the harvester. All we know is the last seen update happened at (20,2), marked in light orange. We call this the "hint". The algorithm will consider this hint but will discard it if it proves it really did not contain an update. An invalid hint can pontentially slow down the algorithm but won't stop it from finding the last update.

Now it is t=26 and we want to look for the last update. Our guess at a start level is:

XOR(20, 26) = 14 = 1110b, in which the first nonzero bit counting from the left is bit #3. Thus, our first lookup will happen at (baseTime(26,3), 3) = (24,3), shown in dark blue below:

If a lookup at (24,3) fails, we consider that there are no updates at lower levels either, since the seeding algorithm would've filled (24,3) before going down. This means there are no updates on or after t=24. Thus, our search area would be reduced to the area directly below (20,2) (green area). We restart the algorithm as if now was 23

If however, a lookup at (24,3) succeeds, then we know the last update could either be the one at (24,3) itself or may be there is a later one in the epochs below (blue area). At this point we consider that the update is in 24 <= t <= 26. We restart the algorithm with the hint set to (24,3) instead of the original one. If the lookup then fails, then the last update was indeed in (24,3)

This is how the algorithm would play out if now t=26 if the last update is in (22,1)

Lookups are, in this order:

#1 (24,3) (fails)
#2 (22,1) (succeeds, but we don't know if this would be the latest update). Go down one level to confirm.
#3 (23,0) (fails)
#4 (22,0) (fails, so we return the last found value at (22,1)

Locking on / following a resource

Once we have found the last update of a resource, we can easily calculate in what epoch will the next update appear, if the publisher actually makes it.

In figure 9 above, if the last update was found at (22,1) and now it is t=26, the next update must happen exactly at (24,3). This holds true until t=32. Beyond that point, the next update can be expected at (32,4) until t=48

Therefore, the node following the resource could sample the expected epoch, keeping in sync with the publisher.

Final notes:

Please let me know your feedback, questions and test issues. I hope you like this feature. I am available on Gitter (@jpeletier) in #orange-lounge channel. Enjoy!!

swarm/api/api.go

zelig · 2018-09-16T23:15:08Z

swarm/storage/mru/lookup/lookup.go

+// but limited to not return a level that is smaller than the last-1
+func GetNextLevel(last Epoch, now uint64) uint8 {
+	// First XOR the last epoch base time with the current clock.
+	// This will set all the common most significant bits will to zero.


bits will to zero -> bits to zero

fixed thanks.

swarm/storage/mru/lookup/lookup.go

zelig · 2018-09-16T23:21:07Z

swarm/storage/mru/lookup/lookup.go

+				return value, nil
+			}
+			hint = epoch
+		} else {


put a continue here rather than an else clause

changed. Thanks

zelig · 2018-09-16T23:26:56Z

swarm/storage/mru/request_test.go

+		t.Fatalf("Expected epoch to be '%s', was '%s'", epoch.String(), checkUpdate.Epoch.String())
+	}
+	if !bytes.Equal(data, checkUpdate.data) {
+		t.Fatalf("Expectedn data '%x', was '%x'", data, checkUpdate.data)


Expectedn -> Expected

fixed, thanks.

zelig · 2018-09-16T23:33:09Z

swarm/storage/mru/request_test.go

+	}
+
+	// mess with the lookup key to make sure Verify fails:
+	recoveredRequest.Time = 77999


not trivial that changing the time here changes the epoch and therefore the address

added an extra comment, though line 185 explained it.

zelig · 2018-09-16T23:34:21Z

swarm/storage/mru/request_test.go

+	r.data = []byte("Al bien hacer jamás le falta premio") // put some arbitrary length data
+	_, err = r.toChunk()
+	if err == nil {
+		t.Fatal("expected request.toChunk to fail when there is no signature", err)


err is nil, not needed as arg

fixed. thanks

zelig · 2018-09-16T23:36:22Z

swarm/storage/mru/cacheentry.go

+const (
+	hasherCount            = 8
+	resourceHashAlgorithm  = storage.SHA3Hash
+	defaultRetrieveTimeout = 100 * time.Millisecond


This is very very low. In network scenario this will be longer, so MRU resolver will be fooled

This number has been as-is since the first version. What would be a good timeout?.
Take into account that the best-case scenario will take 2-3 lookups.

FTR, our convo in orange lounge

lash
@nolash
Sep 17 09:25
maybe the initial lookup should be async and bathced at least, if the timeout increases? And then cancel if the newest comes back?

Javier Peletier
@jpeletier
Sep 17 09:35

Can you elaborate? Each epoch lookup fail/success implies the algorithm takes a diferent path for the next epoch lookup. I thought about this, but would need to run it in parallel, forking on every lookup while we wait for the timeout or success.

So this would allow us to increase the timeout without sacrificing total lookup time, but would produce more lookups.

Viktor Trón
@zelig
Sep 17 09:59
This change is not for this PR for sure. As long as syncing, retrieval works, it's ok. Nonetheless 100millisecs is just too short for network retrieval. I worry that on the main cluster this would result in outdated resolutions cos some chunks just won't arrive that fast.
An alternative to parallelism is to go ahead after a short time but backtrack if a chunk that we thought was not found arrives later. That's probably easier to implement.

Another alternative or additional measure is we have redundancy (probably needed anyway for root chunks) and retrieve alternatives parallelly. That might guarantee some upper limit on successful retrieval latency and therefore makes the 'not found' decision reliable

swarm/storage/mru/handler.go

zelig · 2018-09-16T23:47:05Z

swarm/storage/mru/handler.go

-		log.Warn("Handler.get with invalid rootAddr")
+// Retrieves the resource cache value for the given nameHash
+func (h *Handler) get(view *View) *cacheEntry {
+	if view == nil {


do we need these cjecks on get/set?

Trying to avoid a null-pointer panic in case some code screwed up

unnecessary; do not screw up code 😄

ok, removed.

swarm/storage/mru/lookup: fixed getBaseTime Added NewEpoch constructor swarm/api/client: better error handling in GetResource() swarm/storage/mru: Renamed structures. Renamed ResourceMetadata to ResourceID. Renamed ResourceID.Name to ResourceID.Topic swarm/storage/mru: Added binarySerializer interface and test tools swarm/storage/mru/lookup: Changed base time to time and + marshallers swarm/storage/mru: Added ResourceID (former resourceMetadata) swarm/storage/mru: Added ResourceViewId and serialization tests swarm/storage/mru/lookup: fixed epoch unmarshaller. Added Epoch Equals swarm/storage/mru: Fixes as per review comments cmd/swarm: reworded resource create/update help text regarding topic swarm/storage/mru: Added UpdateLookup and serializer tests swarm/storage/mru: Added UpdateHeader, serializers and tests swarm/storage/mru: changed UpdateAddr / epoch to Base() swarm/storage/mru: Added resourceUpdate serializer and tests swarm/storage/mru: Added SignedResourceUpdate tests and serializers swarm/storage/mru/lookup: fixed GetFirstEpoch bug swarm/storage/mru: refactor, comments, cleanup Also added tests for Topic swarm/storage/mru: handler tests pass swarm/storage/mru: all resource package tests pass swarm/storage/mru: resource test pass after adding timestamp checking support swarm/storage/mru: Added JSON serializers to ResourceIDView structures swarm/storage/mru: Sever, client, API test pass swarm/storage/mru: server test pass swarm/storage/mru: Added topic length check swarm/storage/mru: removed some literals, improved "previous lookup" test case swarm/storage/mru: some fixes and comments as per review swarm/storage/mru: first working version without metadata chunk swarm/storage/mru: Various fixes as per review swarm/storage/mru: client test pass swarm/storage/mru: resource query strings and manifest-less queries swarm/storage/mru: simplify naming swarm/storage/mru: first autofreq working version swarm/storage/mru: renamed ToValues to AppendValues swarm/resource/mru: Added ToValues / FromValues for URL query strings swarm/storage/mru: Changed POST resource to work with query strings. No more JSON. swarm/storage/mru: removed resourceid swarm/storage/mru: Opened up structures swarm/storage/mru: Merged Request and SignedResourceUpdate swarm/storage/mru: removed initial data from CLI resource create swarm/storage/mru: Refactor Topic as a direct fixed-length array swarm/storage/mru/lookup: Comprehensive GetNextLevel tests swarm/storage/mru: Added comments Added length checks in Topic swarm/storage/mru: fixes in tests and some code comments swarm/storage/mru/lookup: new optimized lookup algorithm swarm/api: moved getResourceView to api out of server swarm/storage/mru: Lookup algorithm working swarm/storage/mru: comments and renamed NewLookupParams Deleted commented code swarm/storage/mru/lookup: renamed Epoch.LaterThan to After swarm/storage/mru/lookup: Comments and tidying naming swarm/storage/mru: fix lookup algorithm swarm/storage/mru: exposed lookup hint removed updateheader swarm/storage/mru/lookup: changed GetNextEpoch for initial values swarm/storage/mru: resource tests pass swarm/storage/mru: valueSerializer interface and tests swarm/storage/mru/lookup: Comments, improvements, fixes, more tests swarm/storage/mru: renamed UpdateLookup to ID, LookupParams to Query swarm/storage/mru: renamed query receiver var swarm/cmd: MRU CLI tests

cmd/swarm: remove rogue fmt swarm/storage/mru: Add version / header for future use-

jpeletier requested a review from nolash as a code owner August 31, 2018 09:27

jpeletier mentioned this pull request Sep 3, 2018

add ownerAddr to examples ethersphere/swarm-guide#113

Merged

zelig assigned jpeletier Sep 4, 2018

zelig added type:feature please review labels Sep 4, 2018

jpeletier force-pushed the mru-autofreq branch from 384357d to ec8790d Compare September 4, 2018 15:19

jpeletier mentioned this pull request Sep 7, 2018

[REVIEW] Adaptive-frequency MRU, pre-refactor ethersphere/swarm#914

Closed

jpeletier force-pushed the mru-autofreq branch from 43f4b23 to 7d5f455 Compare September 12, 2018 04:48

zelig approved these changes Sep 12, 2018

View reviewed changes

swarm/api/api.go Show resolved Hide resolved

nolash approved these changes Sep 12, 2018

View reviewed changes

jpeletier force-pushed the mru-autofreq branch 2 times, most recently from 72d6416 to 6a6affd Compare September 13, 2018 11:54

nolash mentioned this pull request Sep 14, 2018

cmd/swarm: MRU smoke tests #17672

Closed

zelig reviewed Sep 16, 2018

View reviewed changes

jpeletier force-pushed the mru-autofreq branch from d990b5f to 8667a51 Compare September 18, 2018 07:02

jpeletier force-pushed the mru-autofreq branch from 8667a51 to 25340fe Compare September 25, 2018 08:09

jpeletier added 3 commits September 27, 2018 09:22

cmd/swarm: remove rogue fmt

c389726

swarm/storage/mru: Add version / header for future use

2d212ed

jpeletier force-pushed the mru-autofreq branch from 81c7367 to 8a85203 Compare September 27, 2018 07:23

nonsense mentioned this pull request Sep 28, 2018

Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification #17778

Closed

swarm/storage/mru: Fixes/comments as per review

07c2aa7

cmd/swarm: remove rogue fmt swarm/storage/mru: Add version / header for future use-

jpeletier force-pushed the mru-autofreq branch from 49b7125 to 07c2aa7 Compare September 28, 2018 08:19

jpeletier added 2 commits September 28, 2018 11:12

swarm/storage/mru: fix linter errors

7f2677f

cmd/swarm: Speeded up TestCLIResourceUpdate

c4a399c

holiman merged commit 2c110c8 into ethereum:master Sep 28, 2018

nonsense added this to the 1.8.17 milestone Oct 8, 2018

jpeletier mentioned this pull request Dec 16, 2018

bzz-script: self-validating chunks ethersphere/swarm#1062

Closed

This was referenced Apr 8, 2019

Swarm/feeds: Parallel feed lookups #19414

Merged

Parallel feed lookups ethersphere/swarm#1332

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification #17559

Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification #17559

jpeletier commented Aug 31, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 18, 2018

zelig Sep 16, 2018

jpeletier Sep 17, 2018

zelig Sep 18, 2018

jpeletier Sep 18, 2018

Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification #17559

Swarm MRUs: Adaptive frequency / Predictable lookups / API simplification #17559

Conversation

jpeletier commented Aug 31, 2018

Abstract

Terminology

What applications does this PR enable?

API changes

HTTP API

To publish an update:

1.- Get resource metainformation

2.- Post the update

To get the last update:

To get a previous update:

Advanced search:

To publish a manifest:

CLI

Creating a resource manifest:

Update a resource

Quick and dirty test:

Adaptive frequency lookup algorithm

Revamping the resource frequency concept

Introducing Epochs

Epoch base time

Seeding algorithm

First updates

Subsequent updates

Harvesting algorithm

Start Epoch

Walking the grid - a simple algorithm

Locking on / following a resource

Final notes:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment