proposal: promote Lead Maintainer Protocol to be Org wide -> Repo Lead Maintainer Protocol

[daviddias]

Here is my first pass and proposal to promote the Lead Maintainer Protocol to be Org wide. I'm confident it can bring more agility and clarity when it comes to maintain code and documentation across the organization.

[nmarley]

It seems like you are removing the overall "Tech Lead" role and only having Lead Maintainers per-project or per-working-group. Is this also part of the intent of this change? Is the Tech-Lead role previously filled by you and @whyrusleeping going to be deprecated in lieu of Working Group Captains?

[daviddias]

Thank you for asking this question! I can understand why this PR would give that idea, however, it's intent is far from changing the definition of Tech Lead and who wears that hat. I believe that will have to be another discussion.

What this PR proposes is that we make the Maintainer role something to be Org wide so that every project benefits from having a DRI and good Repo grooming practices.

[momack2]

I still find all the different titles confusing - is this the appropriate breakdown:

• Project creator (@jbenet) - owns and communicates the vision, has the final say on escalations, the buck stops here
• IPFS Project Captain (@daviddias) - steers the org, master architect/strategist, can settle disagreements between individual WG Leads/Captains
• Working Group Tech Lead (Captain?) - steers the working group effort, manages vision and execution for many repos. Can resolve disputes / guide direction for individual module maintainers.
• Module Lead Maintainer - main shepherd for a particular module including pruning, improvement, and incorporating contributions

aka - the "lead maintainer" title is focused specifically at the repo level, though tech leads / captains are effectively doing the same role for a broader project?

[daviddias]

aka - the "lead maintainer" title is focused specifically at the repo level, though tech leads / captains are effectively doing the same role for a broader project?

That is correct. This question eventually takes us to what is the Governance of the protocol/project. However, I was hoping we can have that discussion separately as I believe it will require its own pace of review.

Meanwhile, I hope to provide more clarity with my last commit -- 5337910 -- in which I added to the title of the protocol "Repo", so that it is more clear that this is a "Repo Lead Maintainer" protocol and not a "Project Maintainer/Leader/Sheppard" protocol.

Sidenote: The reason why to used the word Repo and not Module is just to accommodate that we have repos that are not code and in some languages, a software unit is called a package and not a module. To accommodate everyone, we can use the word Repo which is common to all.

[flyingzumwalt]

Overall
👍 to having a maintainer protocol defined
👍 to empowering more people to feel ownership over repos and get to know them really well
👍 to reducing the bus factor by increasing the number of people with complete knowledge of components
🎉 for testing this within the js org before spreading further

Prior to being involved with IPFS I worked in a context where we prioritized empowering and encouraging new maintainers whenever possible. I think it's an extremely positive practice to implement and dedicate ongoing resources to.

Observation

➡️ This protocol necessitates setting a really high standard of quality for in-depth documentation, learning opportunities, and encouragement/celebration of initiative. It also provides a means for living up to that high standard (empower lead maintainers to make sure that the repos they maintain match expectations). This will benefit the project in many ways.

In addition to implementing the protocol, it's important to allocate time and resources for in-depth learning opportunities at advanced skill levels as well as beginner and intermediat. We do a fair amount of that, but there's always room for more.

❓Question/Concern: Is this doing enough to prevent exploitation by malicious actors

Note: It would be valid to simply respond "Yes. I think we're doing enough without making any other changes. I merely want to make sure this is given adequate consideration before proceeding."

In 2016 when I sought to push for this kind of change -- broadening the set of people who are empowered to act as maintainers of repos -- people cautioned me that the approach of quickly and eagerly embracing new maintainers (which is common in open source) is dangerous for projects like ipfs and lib2p, which are meant to be underlying infrastructure that operates underneath broad swaths of applications. They pointed to systems projects like the linux kernel, language maintainer teams, etc. as examples of open source projects that tightly constrain the path to becoming a maintainer. They specifically flagged the danger of malicious actors exploiting the process in order to achieve maintainership of key repositories and then quietly inject malicious code into the system.

That's not how I usually think. My default inclination is to throw the doors open, embrace anyone who shows interest, and nourish their growth, but in this case I found the argument for caution compelling. I was convinced that we are obliged to be more cautious than other types of open source projects, and I resigned myself to seeing the size and diversity of the maintainers group grow more slowly than I would have usually wanted.

This proposal is one step in the direction of loosening the controls and broadening the exposure area for malicious infiltration, so I want to make sure the implications have been fully considered. Though the lead maintainers protocol in this PR doesn't specify anything about the path to becoming a regular maintainer (someone with merge rights) as opposed to the lead maintainer, this is an important moment to pause and reflect on whether we're striking the right balance between spreading out responsibility and maintaining security.

On one hand, if the number of maintainers is kept too low, the huge number of repos, PRs and contributors will overwhelm them -- slowing things down and exposing you to risk of inadequate controls due to the sheer volume of stuff to review. On the other hand, by increasing the number of maintainers and spreading out these responsibilities, you increase the openings for a malicious actor to quietly infiltrate into a maintainers group and gradually expand their privileges.

The existing practice in this org (I'm not sure if it's written down) is to emphasize "proof of work" as the primary control -- someone must do the work of engaging with the existing maintainers, submitting high-quality PRs, and demonstrating sufficient understanding of the code base in order to become a maintainer. This "proof of work" requires time, technical skill and effort which hopefully makes it prohibitively expensive for a malicious actor whose motivation is merely to inject exploits into the code base.

Possibly this "proof of work" is the only real protection that we can implement and, as with so many security measures, we have to avoid making it so strict that it prevents us from getting work done. In that case,

• ❓Is there anything we should do to ensure that lead maintainers are also upholding these security measures?
• ❓Are there any modifications we should apply to this simple maintainers protocol in order to fulfill our commitment to be diligent about protecting our users?

It would be valid to simply respond "Yes. I think we're doing enough without making any other changes. I merely want to make sure this is given adequate consideration before proceeding."

[daviddias]

➡️ This protocol necessitates setting a really high standard of quality for in-depth documentation, learning opportunities, and encouragement/celebration of initiative. It also provides a means for living up to that high standard (empower lead maintainers to make sure that the repos they maintain match expectations). This will benefit the project in many ways.

Fully agreed. In part, that is why there is a nomination process and it is not "land grabbing" per se. Even amazing maintainers can let a best practice slip from time to time and so, it is important that we have documentation, coaching and periodic review to identify missed opportunities and reset the high bar.

In addition to implementing the protocol, it's important to allocate time and resources for in-depth learning opportunities at advanced skill levels as well as beginner and intermediat. We do a fair amount of that, but there's always room for more.

Would you like to articulate what could some of those opportunities that you have in your mind?

❓Question/Concern: Is this doing enough to prevent exploitation by malicious actors
❓Is there anything we should do to ensure that lead maintainers are also upholding these security measures?

Yes and no. There is never 100% security, our code is not formally verified (yet! :)) and we don't do strict security verifications such as background checks, profiling of humans and their close networks and anything you could expect from a high security operation.

What we do to prevent malicious actors today is:

• Have a nomination process (aka it is not land grabbing) - Before becoming a Maintainer and gain push to master, we need to develop the trust necessary to feel confident that we are putting all our users in good hands. Note: This is already a practice, multiple people have been rejected in the past for lacking this trust for the JS ecosystem experiment.
• We don't give away "Maintainership" the top level modules (aka the modules that users install, e.g. js-ipfs, js-libp2p, js-ipld, go-ipfs) to non Core Contributors/Tech Leads. Releasing a cut of our main software packages requires a Release Dance that even the Core Contributor/Tech Lead can't do in a single step.

What we will be adding in the future:

• A full Security program for the IPFS project (Responsible Disclosure Program, Bug bounty, Periodic Security Audits)

What we should also consider to add:

• Our own auditing. package-table module already lets us to do that to some extent. We can expand the tooling with the habit at looking at the list, the length that someone has been maintaining the project and consider creating a practice of rotating responsibility to keep things fresh, avoiding the scenario where a module grows in complexity and then instead of a bus factor of 1, we have a bus factor of 2, but still a bus factor.

Though the lead maintainers protocol in this PR doesn't specify anything about the path to becoming a regular maintainer (someone with merge rights) as opposed to the lead maintainer, this is an important moment to pause and reflect on whether we're striking the right balance between spreading out responsibility and maintaining security.

It actually does mention :) see "In practice" section.

Tooling has improved significantly over the years. Today we can make the clear distinction by configuration on "Write Access" (create branches, change branches) vs "Push to Master" (merge to master).

And even after getting "Push to Master" access, there is another level which is getting "Publish/Release" which is what puts code directly into users hands and therefore, it is given to a really small group (2 people).

[daviddias]

@ipfs/wg-captains can I have your review here?

[alanshaw]

Please finish this sentence I can't take the suspense!!!

[alanshaw]

Earlier this is referenced as maintainer.json - please can has consistent?

[alanshaw]

If in a position of great responsibility, 2FA should be enabled where applicable?

[daviddias]

We currently demand 2FA for everyone that has write access. It was a big thing, see ipfs/community#263

[alanshaw]

👍 I was thinking for npm (etc.) also

[daviddias]

Ah, good point!

[alanshaw]

If a JSON then this should be like this for validity and easy copy/paste:

{
  "repoLeadMaintainer": {
    "name": "<Repo Lead Maintainer Name>",
    "email": "<Repo Lead Maintainer Email Address>",
    "username": "<Github Username>"
  },
  "workingGroup": {
    "name": "<Name of the Working Group that owns this Repo>",
    "entryPoint": "<Url to the coordination entry point for the Working Group>"
  }
}

[daviddias]

Hi @ipfs/wg-captains o/ ! Can I have your review on this proposal? I would like to move forward with it so that I can start automating things.

[momack2]

Generally LGTM. What is the process by which this goes from "only used by the JS team" to "used by all teams"? Do we need to have every team define the sub areas/repos that their WG can bless lead maintainers for (and assume the default is the WG Tech Lead for those without maintainer.json files)?

[daviddias]

@momack2 great question! My current plan is:

• Have the blessing of the @ipfs/wg-captains on this PR to merge it
• Adapt the tool https://www.npmjs.com/package/package-table to get a snapshot of all our repos and check in which ones don't have Lead Maintainer yet
• PR the badge of the Working Group that maintains the repo
• Ask the Working Group to nominate a Lead Maintainer for each repo they own

[daviddias]

Given that 1 month has passed since this PR creation and many of the captains plus other stakeholders had time to review without any blockers, I’ll merge it tomorrow (Tuesday) this Friday (I learned that I forgot to request @Stebalien and @eingenito for review thanks to @alanshaw!) if there are no blockers.

[alanshaw]

What's the status of this?

[daviddias]

@alanshaw good reminder. Let's put it on a boat!