Update code insights planning process

handbook/engineering/developer-insights/code-insights/processes.md

@@ -16,28 +16,79 @@ Attendees are encouraged to add comments on talking points asynchronously before
 
 ## Planning and prioritization
 
-We plan and track our day-to-day work on our [Kanban board](https://github.com/orgs/sourcegraph/projects/118). Our current process is as follows:
+We plan and track our day-to-day work on our [GitHub board](https://github.com/orgs/sourcegraph/projects/200/views/1).
+The project is separated into multiple views: **Current iteration**, **Next iteration** (both grouped by issue status) and **All iterations** (grouped by milestone).
 
-- Incoming tickets (e.g. from other teams) arrive in the _Inbox_ column.
-- Work is scheduled by adding a card to the _Planned_ column. This happens after talking through the next priorities in our [weekly sync](#weekly-sync) or raising something asynchronously in the #code-insights Slack channel.
-  - The _Planned_ column is an _ordered_ column, by priority. Priority is discussed on the team.
-  - Work should not be moved into the _Planned_ column until it is ready for development _and_ estimated and prioritized.
-  - Work that is ready for development but not yet estimated or prioritized sits in the _Ready for Dev_ column.
-  - Anything that needs design input gets the `needs-design` label and goes in the _Needs design_ column.
-- When starting work, engineers pull cards from the _Planned_ column and move it to the _In Progress_ column.
-- There should never be more than a couple of cards in the _In Progress_ column at the same time. If we paused work on something (e.g. because priorities changed), it should be moved back to _Planned_ or _Icebox_ as appropriate.
-- Things we triaged from the _Inbox_, but don't plan to work on, go into the _Icebox_.
+Our current process is as follows:
 
-Note: we intentionally plan so we can avoid merging significant work less than two days before a release (if a release is on the 20th, our last day to merge is the 18th). Exceptions require explicit approval of both the PM and EM.
+- We work in **2-week iterations**. Every iteration as a GitHub milestone, which is created at least one iteration in advance.
 
-## Tracking Issues
+- Incoming tickets (from team members or from other teams) are added to the [GitHub project](https://github.com/orgs/sourcegraph/projects/200/views/1) and will be visible in the "All iterations" view under "No milestone". This represents our **backlog** of _unscheduled_ work.
 
-The team makes use of [tracking issues](../../tracking_issues.md) for tracking progress on the implementation of longer projects that need to be broken up into multiple issues.
-The teammates should ensure that a tracking issue is created when starting work on features that are expected to take longer than a few days (or require multiple PRs) to deliver.
+- **While an iteration is ongoing, we plan the next iteration**. This is mostly a mostly asynchronous process.
+
+  - Engineers, designer, PM and EM **_propose_ issues to be worked on** the week _before_ the mid-iteration [Monday sync](#weekly-sync) by adding them to the next iteration milestone and setting "Status" to "Proposed". This is done considering our [higher-level goals and roadmap](./goals.md).
+
+    - The issue should also have its **_Estimate_ column filled out**, so that it can be evaluated whether it fits into the iteration. If the proposer lacks the information to estimate the issue, they reply on the issue in GitHub or raise it in our Slack channel to get the missing information or get an estimate from the appropiate person. Teammates may also discuss this in ad-hoc synchronous meetings if beneficial. An assignee may also already volunteer or be proposed, but this may still be changed at the [Monday sync](#weekly-sync) to distribute workload.
+      - If **technical exploration** is needed to get more information, a _spike_ (a time-boxed investigation task meant to facilitate more granular planning) can be proposed for the next iteration instead to get that information.
+
+  - As much as possible, the proposer **involves the necessary stakeholders _asynchronously_ to get agreement** on whether the issue should be worked on in the next iteration before the [Monday sync](#weekly-sync). For example, the PM or EM might ping engineers in GitHub or Slack on whether an issue seems feasible, or engineers might ping their EM and PM to get buy-in whether the issue fits into our goals.<br>
+    If a task needs synchronous discussion with a smaller group than the whole team, teammates are encouraged to meet synchronously in a smaller group to have more clarity before the mid-iteration [Monday sync](#weekly-sync).
+
+  - Teammates can **_reorder_ proposed issues** on the ["Next iteration" board](https://github.com/orgs/sourcegraph/projects/200/views/4) before the mid-iteration sync to their liking. The order at the time of the mid-iteration Monday sync is the _proposed order_.
+
+  - At the mid-iteration [Monday sync](#weekly-sync), we take a look at the proposed issues together on the ["Next iteration" view](https://github.com/orgs/sourcegraph/projects/200/views/4) to **reach agreement** on the **set** of iteration issues, their **assignees**, and **order**, taking into account our [goals and roadmap](./goals.md), technical aspects, estimates, workloads on individuals, and release dates (the 20th of every month).<br>
+    As a rule of thumb, the sum of estimates for each individual should not exceed **8 days** (out of a 10-work-day iteration) so we have enough buffer, time to reply to asks from other teams, to plan the next iteration's projects, and self-organized slack time.
+
+- During an iteration, teammates **work on their assigned issues for the iteration in the order they are listed** in the ["Current iteration" view](https://github.com/orgs/sourcegraph/projects/200/views/1) of the board. When starting work on a task, the teammate **updates its status column to "In Progress"** to communicate it to their team. This gives a good overview in the ["Current iteraton" view](https://github.com/orgs/sourcegraph/projects/200/views/1), which can also be viewed in [Kanban layout](https://github.com/orgs/sourcegraph/projects/200/views/1?layout=board), on how the iteration is tracking.
+
+- If one or more issues that were planned for an iteration are looking to **not get finished** in the [current iteration](https://github.com/orgs/sourcegraph/projects/200/views/1) (while maintaining sustainable work practices) the assignee **raises this as soon as possible asynchronously** to the team (including the PM and EM), e.g. on the GitHub issue or Slack. These issues then **become _proposed_ issues for the next iteration** (meaning nothing carries over automatically, but we also don't just drop and forget missed issues).
+
+### Releases
+
+Despite following two-week iterations, our [releases are monthly on the 20th](../../releases.md#releases) and we may sometimes need to order tasks in a way that we can get important projects into the next release.
+
+We also intentionally plan so we can avoid merging significant work less than two days before a release (if a release is on the 20th, our last day to merge is the 18th). Exceptions require explicit approval of both the PM and EM.
+
+### Planning larger projects
+
+Larger (usually multi-iteration) projects coming from our roadmap go through three phases:
+
+1. **Product context and scope agreement** (usually 1-2 weeks)<br>
+   The PM creates a product-level RFC summarizing user desires from feedback and proposes a reasonable timeline to reach a scope agreement (mentioned in the RFC).
+   Engineers, designer (as applicable), PM, and EM discuss the needs in the RFC and the feasibility/complexity of different options.
+   **The goal of this phase is to agree on a scope for the project**, which will generally be a _subset_ of the desires the PM outlined in the RFC based on the implementation complexity, how long it would roughly take to implement, and how much time we want to dedicate to the project.
+   After discussion, the agreed-upon scope is recorded in a section at the bottom of the RFC.
+
+2. **Design and implementation planning** (usually 1-2 weeks)<br>
+   For a UI project, our designer will create wireframes and designs which are reviewed and discussed by all teammates in Figma.
+   Similarly, engineers will make a proposal for the needed GraphQL APIs (e.g. in an RFC) and agree on the API shape, think the implementation through, and create concrete implementation sub-tasks with estimates, testing plans and a tracking issue collecting them.
+   The goal of this phase is to be "ready to go" for the implementation phase – having a plan (with reasonable confidence) that only needs to be executed.
+
+3. **Implementation and testing** (usually 1-4 weeks)<br>
+   Engineers execute on the implementation plan, putting a set of issues from the tracking issue into each iteration.
+   This also includes that each sub-implementation-task is sufficiently tested, meaning by the end of this phase the project is ready to ship to customers with confidence.
+
+We sequentialize and parallelize projects in a way that we can _plan_ projects (step 1 and 2) while another project is currently being _implemented_, so that we always have the next project ready to be implemented by the time a project has finished implementation.
+We will however make sure to never have multiple projects in planning phase at the same time, as this leads to cognitive overload while also delivering on an implementation for another projects.
+This means a teammate may have to think at most about 2 projects at any given time: one that is being implemented, and one that is being planned, and a teammate will never have to implement more than one larger project at the same time.
+
+<object data="./project_planning.svg" width="1100" height="263" aria-label="Diagram showing three example projects going through all three project phases, with each product and design phase in parallel to the implementation phase of the previous project."></object>
+
+#### Tracking issues
+
+To plan projects that span multiple iterations and need to be broken up into sub-tasks, we make use of [tracking issues](../../tracking_issues.md).
+The tracking issue is created by one of the teammates with a descriptive accompanying label, e.g. <span class="badge bg-info">insights-dashboards-v1</span> (milestones are not used for this, as they are used for iterations).
+
+Suffixes like `v1` can be used to communicate we are aiming for an initial, well-defined scope to avoid scope creep.
+After the agreed upon scope for the project is completed, the tracking issue is closed.
+Further improvements are either tracked as individual tasks in iterations, or if a new, larger, multi-iteration improvement, a new tracking issue is created with a new label.
+
+Individual tasks of the tracking issue are assigned to iterations through their milestone.
 
 ## Product Feedback
 
-Specific product feedback about well-defined, small features can be found directly in the GitHub board.
+Specific product feedback about well-defined, small features can be found directly in the [GitHub board](https://github.com/orgs/sourcegraph/projects/200/views/7).
 More general product feedback that applies to larger features, or that needs more research and planning to be actionable, is kept in [Productboard](https://sourcegraph.productboard.com/feature-board/1793095-code-insights).
 
 ## Retrospectives