Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPNS pinning: ipfs name follow #4435

Open
ghost opened this issue Nov 29, 2017 · 25 comments
Open

IPNS pinning: ipfs name follow #4435

ghost opened this issue Nov 29, 2017 · 25 comments
Labels
kind/feature A new feature topic/commands Topic commands topic/ipns Topic ipns
Milestone

Comments

@ghost
Copy link

ghost commented Nov 29, 2017

cc @vyzo

USAGE
  ipfs name follow <ipns-path> - Track changes to IPNS names.

SYNOPSIS
  ipfs name follow [--pin=true] [--allow-expired=true] [--] <ipns-path>

ARGUMENTS

  <ipns-path> - IPNS path of the name that should be followed.

OPTIONS

  --pin           bool   - Always pin the name's most recent content. Default: true
  --allow-expired bool   - Allow lookup even if a name's TTL is reached. Default: true

DESCRIPTION

  The 'ipfs name follow' command can be used to track changes to an IPNS name.
  A followed name will be refreshed in the background in intervals, so that
  lookups through an 'ipfs' command or the HTTP API are always instant and up-to-date.

  When called with '--pin' (default), after refreshes the name's new content
  is pinned. This is equivalent to 'ipfs pin replace /ipfs/QmOld /ipfs/QmNew'.

  When called with '--allow-expired' (default), lookups of the name through
  the 'ipfs' command or the HTTP API always return the last known content,
  even if the TTL has been reached and the name has expired. If this is the
  case, a warning is printed along with whatever is the command's respones.

  The combination of '--pin' and '--allow-expired' allows for convenient
  offline use cases. For example, on a long flight I'd like to have IPFS docs
  with me, and I'd like them to be usable at /ipns/docs.ipfs.io as always.

  Open question: pubsub

  Open question: refresh after ttl/2 or whatever
  - /ipns/QmFoo:       ttl = ipns-ttl
  - /ipns/dnslink.net: ttl = min(dns-ttl, ipns-ttl)

  Open question: storage of followings
@ghost ghost added topic/commands Topic commands kind/feature A new feature topic/ipns Topic ipns labels Nov 29, 2017
@Stebalien
Copy link
Member

Stebalien commented Nov 29, 2017

Is this only active when the command is active? I see two different commands here: ipfs pin add --follow and ipfs resolve --follow.

Pinning

ipfs pin add --follow [--allow-expired=true] [--lazy=false] /ipns/QmId

This command will create a new (permanent) pin that will "follow" this IPNS address while the daemon is active.

Note: --lazy exists to mirror --prefetch below (basically: make a non-strict pin that doesn't download everything up-front).

Use-cases: Mirroring, backup, offline-replication.

Follow

ipfs resolve --follow [--prefetch] [--quiet] /ipns/...

Unlike the previous command, this "follow" would only remain active while the command is active. By default, unless --quiet is passed, it would print out each new target ipfs address as it finds newer IPNS records. If --prefetch is specified, this command will prefetch the linked contents. TODO: It should probably pin the contents with some form of temporary (low-priority?) pin but we don't really have those at the moment.

Use-cases: Authenticated data streams, website "feeds".

@vyzo vyzo self-assigned this Nov 29, 2017
@ghost
Copy link
Author

ghost commented Nov 29, 2017

@Stebalien yeah your proposal sounds good too!

@vyzo
Copy link
Contributor

vyzo commented Nov 29, 2017

I am not sure we need a behaviour that is only available while the command is active - do we really care all that much about seeing background resolutions displayed in the terminal?

edit: of course it's not just human eyeballs consuming the output

@vyzo
Copy link
Contributor

vyzo commented Nov 29, 2017

There is also the implicit relationship to ipns pubsub.
We can have the follow command work through pubsub or it can be something that does background resolutions.
The former is more efficient and adds value to ipns pubsub, the latter always works regardless of whether the other end have enabled pubsub.

@Stebalien
Copy link
Member

edit: of course it's not just human eyeballs consuming the output

Exactly. That's more for e.g., webapps.

We can have the follow command work through pubsub or it can be something that does background resolutions.
The former is more efficient and adds value to ipns pubsub, the latter always works regardless of whether the other end have enabled pubsub.

I'd do both. Always use pubsub and occasionally check the DHT for updates (more frequently if we see updates through the DHT that we don't see through pubsub).

@matthewrobertbell
Copy link

An alternative could be allowing "pinning" of IPNS names, so that they are both automatically updated, the records republished by that node and the current IPFS object which is pointed to being automatically pinned as well.

This would mean that there is no requirement to keep a command running.

@rklaehn
Copy link

rklaehn commented Nov 30, 2017

@Stebalien

so the ipfs resolve --follow --prefetch would only print out a new hash for a name once everything is pinned?

the ipfs pin add --follow command is permanently running in the background, but it would be great if there was a way to track what it is currently doing (what is the latest hash for the name, what is the latest hash that is fully pinned, etc.)

@vyzo
Copy link
Contributor

vyzo commented Nov 30, 2017

I think I prefer the ipfs name follow approach, as it's clear that it something that works with namesys (ie it's ipns functionality).

@Stebalien
Copy link
Member

An alternative could be allowing "pinning" of IPNS names, so that they are both automatically updated, the records republished by that node and the current IPFS object which is pointed to being automatically pinned as well.

That's the first command I proposed.

so the ipfs resolve --follow --prefetch would only print out a new hash for a name once everything is pinned?

No. Actually, it wouldn't even pin. That command would just print out the hash and then start fetching in the background. If you wanted a pin, you'd use ipfs pin add --follow ... (you could also do both).

the ipfs pin add --follow command is permanently running in the background, but it would be great if there was a way to track what it is currently doing (what is the latest hash for the name, what is the latest hash that is fully pinned, etc.)

For the latest hash, you'd just resolve the IPNS name (which should use the cache). "Fully pinned" is a bit trickier. The second we get a new hash, we'd unpin the previous version and start pinning a new one.

We could have a --keep-history (to keep past versions). However, we probably wouldn't have a way to query past versions.

I think I prefer the ipfs name follow approach, as it's clear that it something that works with namesys (ie it's ipns functionality).

Good point. I agree.

@mitar
Copy link

mitar commented Feb 4, 2018

I think this is duplicate of #1467?

@Stebalien
Copy link
Member

@mitar this issue exists to discuss specific designs (although it does address the needs of #1467).

@lockedshadow
Copy link

What a wonderful name! This upcoming feature looks very interesting.

And do I understand correctly that "following" nodes will republish the last seen value of given name, even if the creator of that name (with its private key) disappears once? Will the new nodes be able to resolve a value of that name, if the "following" ones will be present in network?

@vyzo
Copy link
Contributor

vyzo commented May 9, 2018

that's an interesting idea.

@sonatagreen
Copy link

We could have a --keep-history (to keep past versions). However, we probably wouldn't have a way to query past versions.

This part feels like the sort of thing that might be best handled with IPRS involved somehow.

@hsanjuan
Copy link
Contributor

Could we have most of this as a library independent from go-ipfs? A library that, given a libp2p host and an IPNS key(s) and some config, puts new value on a channel for consumption.

It would be useful for cluster, and I think for other people.

@magik6k
Copy link
Member

magik6k commented Feb 18, 2019

This would require extracting namesys, but can be done.

@NatoBoram
Copy link
Contributor

Hi! Is there any news related to this issue?

Termux has started to publish their packages via IPFS under /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx and /ipns/k51qzi5uqu5dj05z8mr958kwvrg7a0wqouj5nnoo5uqu1btnsljvpznfaav9nk. I'd like to be able to automatically follow these pins whenever they're updated.

Unfortunately, right now, it requires a fair share of scripting skills to get done.

I do have something in Bash...
#!/bin/bash

function update() {
	local name="$1"
	local ipns="$2"

	printf "Updating \e[33m$name\e[0m...\n"

	local oldIpfs="$(ipfs files stat --hash $name)"
	local newIpfs="$(ipfs name resolve $ipns)"

	ipfs pin update "$oldIpfs" "$newIpfs"
	ipfs files rm -r $name
	ipfs files cp "$newIpfs" $name
	ipfs files stat $name

	printf "\n"
}

update '/termux-stable' 'k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx' &
update '/termux-unstable' 'k51qzi5uqu5dj05z8mr958kwvrg7a0wqouj5nnoo5uqu1btnsljvpznfaav9nk' &
wait

echo 'Done.'
But that's considerably harder to do in PowerShell Core...
#Requires -PSEdition Core

$functions = {
	function update($name, $ipns) {
		Write-Host 'Updating ' -NoNewline
		Write-Host "$name" -ForegroundColor Yellow -NoNewline
		Write-Host '...'

		$oldIpfs = $(ipfs files stat --hash $name)
		$newIpfs = $(ipfs name resolve $ipns)

		ipfs pin update $oldIpfs $newIpfs
		ipfs files rm -r $name
		ipfs files cp $newIpfs $name
		ipfs files stat $name

		Write-Host
	}
}

Start-ThreadJob -InitializationScript $functions -ScriptBlock { update '/termux-stable' 'k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx' }
Start-ThreadJob -InitializationScript $functions -ScriptBlock { update '/termux-unstable' 'k51qzi5uqu5dj05z8mr958kwvrg7a0wqouj5nnoo5uqu1btnsljvpznfaav9nk' }
Get-Job | Wait-Job | Receive-Job

Write-Output 'Done.'

Another problem is that it has to be executed periodically and that also requires a fair share of skills in systemd or Task Scheduler and I think following an IPNS address should be a built-in feature.

More precisely, the workflow I'm looking for is something that can

  • Add an IPNS address to a tracking list
  • Optionally pin all new updates
  • Optionally unpin all stale updates
  • Works in the background as part of a regular IPFS daemon

@RangerMauve
Copy link

This would be very nice to have.

I'm working on an IPFS devgrant to create a protocol test suite for browsers and I need to have some test files as well as the entire suite pinned and available over IPNS.

It's possible to do some hacky stuff to make it work, but first class support for ipns when pinning would make life easier.

I also saw @lidel mention that IPNS is a lot speedier now when pubsub is enabled, which would also make it easier for gateways to be reactive to updates instead of periodically polling.

@Forza-tng
Copy link

Hi! I'd also like to mention the importance of being able to pin a /ipns/some.domain.tld/ and have it automatically download new and updated files.

My use case is being file mirror for a software project and without this functionality it is very difficult to keep the local mirror updated.

@hsanjuan
Copy link
Contributor

My use case is being file mirror for a software project and without this functionality it is very difficult to keep the local mirror updated.

You can use ipfs-cluster to follow one or several pins. I.e run a ipfs-cluster-service on one side and ipfs-cluster-follow on the "clients". https://cluster.ipfs.io/documentation/collaborative/

@NatoBoram
Copy link
Contributor

NatoBoram commented Jul 13, 2021

@hsanjuan You still need some scripting, but replace ipfs by ipfs-cluster-ctl. Plus, if you only have one node to maintain and use to follow the IPNS, then you could just use ipfs directly. Even if you had multiple nodes, ipfs-cluster-service doesn't offer much over ipfs if it can't distribute part of a pin instead of the whole thing; at that point. just run the script on all ipfs clients.

If the maintainer uses IPNS instead of IPFS Cluster, then there's no good reason to use IPFS Cluster yourself, unless you have like 4+ machines.

@lanzafame
Copy link
Contributor

@NatoBoram

ipfs-cluster-service doesn't offer much over ipfs if it can't distribute part of a pin instead of the whole thing

Except it offers the feature that is being requested, which is to follow a pin or pinset. Please note that you don't need to run an actual cluster of ipfs-cluster nodes for this feature to work. It only requires a single ipfs node with a single ipfs-cluster node running beside it. Would ipfs name follow be simpler? yes. Is writing the systemd files to start an ipfs and ipfs-cluster node easier than scheduling a script? yes.

Sharding is implemented in cluster but is patiently waiting on a feature in the IPFS pinning system to have it enabled #5133.

@MayeulC
Copy link

MayeulC commented Feb 8, 2023

This is quite similar to #1467

My use-case is the following: I have a document (website) that is updated through a CI system, running on the "cloud".
I could publish an updated IPNS record in the same CI, and have my local IPFS servers mirror it automatically, while the CI job is ephemeral.

TBH, I'm not sure if that would be the best way to go about it (I need to ensure the content is fully mirrored before shutting down the CI job, so I may need some communication with the IPFS servers anyway), but that seems one of the simplest.

@lordcirth
Copy link

This is quite similar to #1467

My use-case is the following: I have a document (website) that is updated through a CI system, running on the "cloud". I could publish an updated IPNS record in the same CI, and have my local IPFS servers mirror it automatically, while the CI job is ephemeral.

TBH, I'm not sure if that would be the best way to go about it (I need to ensure the content is fully mirrored before shutting down the CI job, so I may need some communication with the IPFS servers anyway), but that seems one of the simplest.

Perhaps you could poll findprovs to test that other nodes are mirroring?

@MayeulC
Copy link

MayeulC commented Feb 26, 2023

True.

I actually wanted to edit this comment, as I think the real benefit here is that you don't have to share the private key with many servers, only with the initial publisher, which can be ephemeral. This could have security benefits.

@hacdias hacdias changed the title ipfs name follow IPNS pinning: ipfs name follow Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature A new feature topic/commands Topic commands topic/ipns Topic ipns
Projects
No open projects
Status: 🥞 Todo
Development

No branches or pull requests