6.0 Upgrade Project

[michaelbaamonde]

This issue encapsulates the work we know we need to do for the 6.0 upgrade project. It's a superset of #14, so I'll close that issue. We will undoubtedly discover things we need to do along the way (the Rumsfeldian "unknown unknowns".) Teams and tasks are broken down by product. Each product also includes its X-Pack features. There is also a "stack" team that covers cross-product considerations. Elasticsearch and Kibana are a bit heavy on things we already know we need to do, so any extra help there would be welcomed. We could also use some more eyes on Logstash, as @jsvd is currently a one-man army!

Elasticsearch

Team

• @clintongormley
• @imotov
• @tvernum

Todos

• Verify upgrade procedure
• Verify breaking changes
• Document rolling upgrades from 5.6 to 6.0
• Document implications of Security requiring TLS between nodes as of 6.0
  • i.e. If you don't have TLS between nodes, you can't do a rolling upgrade from 5.6 to 6.0; you have to do a full-cluster restart
• Document manual reindexing of internal indices (e.g. .watcher, .security)
  • This is for X-Pack users who don't want to use the upgrade assistant tool in Kibana. Probably rare, but a valid use case.
• Verify es-hadoop docs
  • Upgrade docs
  • Breaking changes docs

Logstash

Team

• @jsvd
• TBD

Todos

• Verify (write?) upgrade procedure
  • Looks like this hasn't been updated for 6.0 yet
• Verify (write?) breaking changes
  • Looks like these haven't been updated for 6.0 yet
• Any considerations re: removal of multiple types that need to be documented?
  • Have plugins that we maintain accounted for this?
  • Should we provide some guidance for maintainers of community plugins?

Kibana

Team

• @pickypg
• @LeeDr
• @archanid

Todos

• Verify (write?) upgrade procedure
  • Looks like this hasn't been updated for 6.0 yet
• Verify breaking changes
• Document upgrade assistant (@archanid) Document upgrade assistant #18
• Document manual reindexing of .kibana for users without X-Pack (@tylersmalley) Document manual reindexing of .kibana for users without X-Pack #19

Beats

Team

• @tsg
• @dedemorton

Todos

• Verify (write?) upgrade procedure
  • Looks like these haven't been updated for 6.0 yet
• Verify (write?) breaking changes
  • Looks like these haven't been updated for 6.0 yet
• Any considerations re: removal of multiple types that need to be documented?

Cloud

Team

• @stejacks
• @stu-elastic
• @beiske
• @igor-kupczynski

Todos

• Specifics TBD
  • Is there anything Cloud-specific re: upgrading from 5.x to 6.0? Anything that needs to be changed here, other than 5.6->6.0 not requiring a full-cluster restart?
  • Any special considerations for ECE?
  • @nrichers is working on ECE-specific upgrade documentation, so perhaps it will ultimately just be a matter of linking to it from these docs (https://www.elastic.co/guide/en/cloud/current/upgrading.html; WIP 6.0 doc PR)

Stack

Team

• @inqueue
• @cwurm
• @msimos
• Steven G.

Todos

• Verify stack upgrade order from 5.0 still holds
• Update full-stack upgrade doc
• Test procedure including X-Pack
• Test procedure excluding X-Pack

Misc.

• Mine customer feedback sources (casesearch, enhancements, Discuss, etc.) for any common suggestions/complaints/praise re: the 2.x->5.0 procedure.
  • The idea here is to learn from whatever (if any) mistakes we made last time and get us thinking upfront about potential customer pain points.
  • @michaelbaamonde and @zuketo to lead.
• Document migrating to and living in a post-types world.
  • Reindexing internal indices to their new, typeless mappings is covered under individual products, but there's a broader question of guiding users re: their own data models. We should produce some artifact that discusses some approaches, possibly using our internal experiences with things like Monitoring as a jumping-off point.
  • See https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
• Do we need a migration guide for scripting now that we've removed support for all languages but Painless?
• Ensure instructions also work for our Docker images
  • Docker is A Very Special Snowflake and is consistently surprising, so assume nothing in this scorched hellscape of containers and orchestrators that we, the wailing and the doomed, have inherited

[stejacks]

Added the rest of the cloud team that I know of.

[nrichers]

I added some links to the existing Elastic Cloud (aka "SaaS") upgrade docs and a pointer to the WIP Cloud 6.0 upgrade docs PR. Emphasis on WIP ...

[debadair]

Just FYI, I'm working on incorporating the process @clintongormley outlined into the stack upgrade doc.

[kurtado]

Seeing a lot of interest (and excitement) for this in the field; is there any public-facing information, documentation, blog, etc we can point them to? Or will it become available only when we release 6.0?

[michaelbaamonde]

@kurtado Things are very much in-flight, but docs are deployed whenever there's a merge to the master branch. If people are curious, I'd point them to https://www.elastic.co/guide/en/elastic-stack/master/upgrading-elastic-stack.html, but, again, it's a work in progress. Some recent work that @debadair has done should be available there shortly.

[tylersmalley]

I have opened a PR for manually migrating the Kibana index: elastic/kibana#13696

[zuketo]

@debadair I've compiled some of my notes here: https://docs.google.com/a/elastic.co/document/d/1ZlpO_ANn0FF4sFD5tRUAZSYvVWktetIgYqPdtMH7pLQ/edit?usp=sharing

Please let me know if I can follow up on anything, I've tried to sum up the notes with a table at the top.

[rhoboat]

I have a few PRs up for concerns with how things are worded in the docs.
Original page: https://www.elastic.co/guide/en/elastic-stack/6.0/upgrading-elastic-stack.html
PR: #24

This one in particular is for a page that we have forgotten about:
Original page: https://www.elastic.co/guide/en/kibana/6.0/upgrade.html
PR: elastic/kibana#13897