Unable to access repos after going from 1.15.0 => 1.15.2

[echodreamz]

• Gitea version (or commit ref): 1.15.2
• Git version: 2.33.0
• Operating system: Ubuntu 20.04.2 using systemd and Gitea downloaded from the github releases page (amd64 build)
• Database (use [x]):
  • PostgreSQL
  • [ x] MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • [ x] No
• Log gist:
  2021/09/05 09:58:23 ...ules/context/repo.go:343:repoAssignment() [E] GetUserRepoPermission: skipObjectDecoder: expect object or null, error found in #0 byte of ...|&\003d{\003d}\003d|..., bigger context ...|&\003d{\003d}\003d|... 2021/09/05 09:54:32 ...ters/private/serv.go:293:ServCommand() [E] Unable to get permissions for 3:REMOVED with key 8 in 2:ORG/myrepo Error: skipObjectDecoder: expect object or null, error found in #0 byte of ...|&\003d{\003d}\003d|..., bigger context ...|&\003d{\003d}\003d|... 2021/09/05 09:54:33 modules/ssh/ssh.go:140:sessionHandler() [E] SSH: Wait: exit status 1

Description

When logging in, I am able to see all commits on the dashboard to repos I have access to, but the right side repos card is empty and missing all repos that I have access to. When selecting to view a repo, I get a 500 error with the error above logged. Rolling back to 1.15.0 and everything returns to normal.

SSH access also fails with errors as well as shown above.

[NoirPi]

Facing the same Issue... even if i try version 1.15.1 i get the 500 error

needed to roll back to 1.15.0 to get it back working again.

[echodreamz]

Same here, v1.15.1 also has the same issue.

[XPhilipp]

v1.14.7 has the same issue.

[zeripath]

What version of MySQL are you using?

[zeripath]

Also are you restoring from a backup created on v1.14.3-v1.14.6? These look like the bug v1.14.7 and v1.15.1 were expressly created to fix.

[XPhilipp]

Also are you restoring from a backup created on v1.14.3-v1.14.6? These look like the bug v1.14.7 and v1.15.1 were expressly created to fix.

Yes, was restored from 1.14.5 or 1.14.4) And upgraded later to 1.14.7

What version of MySQL are you using?

mariadb-server/stable,now 1:10.5.11-1 all

[zeripath]

Also are you restoring from a backup created on v1.14.3-v1.14.6? These look like the bug v1.14.7 and v1.15.1 were expressly created to fix.

Yes, was restored from 1.14.5 or 1.14.4) And upgraded later to 1.14.7

This is the problem that 1.14.7 and 1.15.1/2 were released for.

The instructions on the 1.15.1/1.14.7 blog post are:

... We have specially released v1.14.7 as it contains an important fix for an issue with gitea dump on versions 1.14.3-1.14.6 and version 1.15.0.

We highly encourage users on versions v1.14.3-v1.14.7 to update to v1.14.7 before creating their backup dump files, and then update to v1.15.1 for some important bug-fixes.

By ignoring this you've restored from broken dump and now we're in trouble because fixing it is not going to be easy - at all.

I specifically made sure we released 1.14.7 to help stop this from happening.

[zeripath]

Please tell me that your 1.14.x db is still in existence outside of the dump?

[zeripath]

v1.14.7 has the same issue.

This implies that your db may already be hosed. ☹️

[XPhilipp]

Please tell me that your 1.14.x db is still in existence outside of the dump?

Yes, but lagging behind

Any way, can i fix db my self?

[zeripath]

It's not easy to fix.

The issue is that there is a bug with the dump command in 1.14.2-1.14.6 that means that instead of writing JSON it's doing the equivalent of fmt.Println(v) . That means instead of a field containing: {"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true} it ends up containing &{true true true}.

Now...

the table and fields affected by this are:

• Table login_source field cfg
• Table repo_unit field config

both of these are relatively rare to be updated so a good guess for the correct values is going to be the values that are in the previous db before the broken dump.

[echodreamz]

So overall, we need to stop using gitea itself to dump the database, but instead use the dedicated mysqldump tool?

Checked our oldest backup, (we keep 30 days). Unfortunately, it still has the issue within it. What should our next steps be?

Our login_source table is empty, appears to always have been. The repo_unit has about ~300 rows in it, it appears to be storing repo settings?

UPDATE:

After reviewing the repo_unit table, it appeared to be mostly empty JSON data with an &{} in front. We set all rows to {} which allowed the upgrade to 1.15.2 to go without issues, then went to each repo and reset/saved all the settings, which appears to repopulate the repo_unit table with the correct data.

Along with the gitea dump, we are also using mysqldump to make sure the data coming out of MySQL is correct (at least from a format standpoint).

[zeripath]

I'm saying there was a serious regression in 1.14.3-1.14.6 that meant the dumps were broken for these two columns but no one noticed. Certainly having spent so damned long making sure that they worked in the first place I didn't expect that they would suffer a regression like that.

I've fixed that regression and added a test to stop it from happening again, however we clearly need to add a stronger testsuite to the dumping behaviour as this test only covers a very small component.

I'm glad that your repo_units config column appeared so easily fixable but I'm surprised it was empty. Have you been consistently restoring from backups? My suspicion is you've lost some configuration data here. We need to check your older backups - please contact me on discord or on matrix.

It may not be coming across but I'm seriously and very deeply sorry and annoyed about this. It's just not good enough that this has happened. If we're gonna supply a dump command it should bloody well work.

[echodreamz]

No, we've never restored from backups. We've been on Gitea since... mid 2019. This was our first restore. We migrated from an old CentOS-based server to a shiny new Ubuntu-based servers. Did the gitea dump figuring it was good enough. We also had mad issues with the action table, apparently the backup dump done by gitea outputted \n instead of \\n so the data when imported back into MySQL had full line-breaks instead of just the entity, which was then invalid JSON. I was able to fix this pretty easily and everything went OK, though I did not realize issues existed elsewhere. Note that we were on 1.15.0 before we migrated.

Before this, we had no issues with Gitea aside from 1 other report I filed.

I am surprised that gitea dumps its own DB. Especially for MySQL, do a check and see if mysqldump is available (or even allow an option) and use it, why re-invent the wheel when a perfectly good dump tool exists and was created by the people who wrote the database server.

[lunny]

MySQL is not the only database Gitea supports.

For a workaround for your situation, regenerated repo_unit table should fix all the problems. Could you show some records of your repo_unit?

[XPhilipp]

MySQL is not the only database Gitea supports.

For a workaround for your situation, regenerated repo_unit table should fix all the problems. Could you show some records of your repo_unit?

MariaDB [gitea]> select * from repo_unit limit 10;
+----+---------+------+----------------------------------------------------------------------------------------------------------------+--------------+
| id | repo_id | type | config                                                                                                         | created_unix |
+----+---------+------+----------------------------------------------------------------------------------------------------------------+--------------+
|  1 |       1 |    1 | &{}                                                                                                            |   1596995039 |
|  2 |       1 |    2 | &{%!s(bool=true) %!s(bool=true) %!s(bool=true)}                                                                |   1596995039 |
|  3 |       1 |    3 | &{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false)} |   1596995039 |
|  4 |       1 |    4 | &{}                                                                                                            |   1596995039 |
|  5 |       1 |    5 | &{}                                                                                                            |   1596995039 |
|  6 |       2 |    1 | &{}                                                                                                            |   1596995842 |
|  7 |       2 |    2 | &{%!s(bool=true) %!s(bool=true) %!s(bool=true)}                                                                |   1596995842 |
|  8 |       2 |    3 | &{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false)} |   1596995842 |
|  9 |       2 |    4 | &{}                                                                                                            |   1596995842 |
| 10 |       2 |    5 | &{}                                                                                                            |   1596995842 |
+----+---------+------+----------------------------------------------------------------------------------------------------------------+--------------+
10 rows in set (0.000 sec)

[echodreamz]

MySQL is not the only database Gitea supports.

For a workaround for your situation, regenerated repo_unit table should fix all the problems. Could you show some records of your repo_unit?

I realize this, however, I would assume that most are running alongside with sqlite or a flavor of MySQL, which during backup could be checked if the database engine is MySQL, then use mysqldump, then you do not have to worry about the dumped database backup file being properly escaped, because the backup utility is specifically built by the people who also built the database server.

Note that our first ever restore was with a backup created in v1.15.0, we never restored our database before v1.15.0 and have been running for over 2 years, so something certainly went wrong somewhere. After speaking with @zeripath I believe we were able to get our table fixed up. This isnt the only table with issues after doing a backup/restore operation either.

[lunny]

MySQL is not the only database Gitea supports.
For a workaround for your situation, regenerated repo_unit table should fix all the problems. Could you show some records of your repo_unit?

MariaDB [gitea]> select * from repo_unit limit 10;
+----+---------+------+----------------------------------------------------------------------------------------------------------------+--------------+
| id | repo_id | type | config                                                                                                         | created_unix |
+----+---------+------+----------------------------------------------------------------------------------------------------------------+--------------+
|  1 |       1 |    1 | &{}                                                                                                            |   1596995039 |
|  2 |       1 |    2 | &{%!s(bool=true) %!s(bool=true) %!s(bool=true)}                                                                |   1596995039 |
|  3 |       1 |    3 | &{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false)} |   1596995039 |
|  4 |       1 |    4 | &{}                                                                                                            |   1596995039 |
|  5 |       1 |    5 | &{}                                                                                                            |   1596995039 |
|  6 |       2 |    1 | &{}                                                                                                            |   1596995842 |
|  7 |       2 |    2 | &{%!s(bool=true) %!s(bool=true) %!s(bool=true)}                                                                |   1596995842 |
|  8 |       2 |    3 | &{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false)} |   1596995842 |
|  9 |       2 |    4 | &{}                                                                                                            |   1596995842 |
| 10 |       2 |    5 | &{}                                                                                                            |   1596995842 |
+----+---------+------+----------------------------------------------------------------------------------------------------------------+--------------+
10 rows in set (0.000 sec)

A simple work around is to run UPDATE repo_unit set config = "{}", but you will lose some special settings of repository, you can change the setting again in repository settings page.
WARN: Please back up the table before you executed above SQL.

[lunny]

MySQL is not the only database Gitea supports.
For a workaround for your situation, regenerated repo_unit table should fix all the problems. Could you show some records of your repo_unit?

I realize this, however, I would assume that most are running alongside with sqlite or a flavor of MySQL, which during backup could be checked if the database engine is MySQL, then use mysqldump, then you do not have to worry about the dumped database backup file being properly escaped, because the backup utility is specifically built by the people who also built the database server.

Note that our first ever restore was with a backup created in v1.15.0, we never restored our database before v1.15.0 and have been running for over 2 years, so something certainly went wrong somewhere. After speaking with @zeripath I believe we were able to get our table fixed up. This isnt the only table with issues after doing a backup/restore operation either.

Could you also show some data of your repo_unit?

[zeripath]

@echodreamz when we looked at this together both with your db and me looking at the code we found the following options for config field in the repo_unit table:

Type 6 - External Wiki Configuration

The broken form looks like:

&{url}

but should be:

{"ExternalWikiURL":"url"}

Type 7 - External Tracker Configuration

The broken form looks like:

&{a b c}

But should be:

{"ExternalTrackerURL":"a","ExternalTrackerFormat":"b","ExternalTrackerStyle":"c"}

The problem will be if there are spaces in this one.

Type 3 - Pull Request Configuration

The broken form has two types 1.14:

&{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false) }

and 1.15:

&{%!s(bool=false) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=true) %!s(bool=false) %!s(bool=false) %!s(bool=false) merge}

But should be:

{"IgnoreWhitespaceConflicts":false,"AllowMerge":false,"AllowRebase":false,"AllowRebaseMerge":false,"AllowSquash":false,"AllowManualMerge":false,"AutodetectManualMerge":false,"DefaultDeleteBranchAfterMerge":false,"DefaultMergeStyle":""}

Type 2 - Issue Configuration

Broken looks like:

&{%!s(bool=true) %!s(bool=true) %!s(bool=true)}

But should be:

{"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true}

Others

&{}

Should just be

{}

---

The related code is:

gitea/models/repo.go

Lines 399 to 431 in 63d7cbc

	// MustGetUnit always returns a RepoUnit object
	func (repo *Repository) MustGetUnit(tp UnitType) *RepoUnit {
	ru, err := repo.GetUnit(tp)
	if err == nil {
	return ru
	}
	if tp == UnitTypeExternalWiki {
	return &RepoUnit{
	Type: tp,
	Config: new(ExternalWikiConfig),
	}
	} else if tp == UnitTypeExternalTracker {
	return &RepoUnit{
	Type: tp,
	Config: new(ExternalTrackerConfig),
	}
	} else if tp == UnitTypePullRequests {
	return &RepoUnit{
	Type: tp,
	Config: new(PullRequestsConfig),
	}
	} else if tp == UnitTypeIssues {
	return &RepoUnit{
	Type: tp,
	Config: new(IssuesConfig),
	}
	}
	return &RepoUnit{
	Type: tp,
	Config: new(UnitConfig),
	}
	}

[zeripath]

This I think should give enough information to build a doctor command to attempt to fix things.

[NoirPi]

So im confused!

Is there any way to solve this without loosing any settings?
Cause its unpracticable to re-enable those settings to all my repositories on my instance.

[zeripath]

So im confused!

Is there any way to solve this without loosing any settings?
Cause its unpracticable to re-enable those settings to all my repositories on my instance.

First of all do you have your old db?

If you do just take create a dump using 1.14.7 or copy over the contents of the repo_unit table from that.

If you don't, well at present my above comment shows how you can manually fix the problem. I've not made a doctor subcommand personally because the issue seemed to affect very few people and therefore the hours (possibly days) it would take in coding up the fixer would not be worth it.

If however there are a lot of people with this problem then the above comment shows how a doctor command could be made and makes the subcommand a more pressing issue.

[ghmer]

So, our installation broke since 1.15.1 and I had to revert to 1.15.0. We are using postgres as database.
The recently released 1.15.3 also does not work properly.

Can we expect a fix for this behavior, or do we have to stick with 1.15.0?
Cannot say I am very happy with this situation.

[NoirPi]

@ghmer

A simple work around is to run UPDATE repo_unit set config = "{}", but you will lose some special settings of repository, you can change the setting again in repository settings page.
WARN: Please back up the table before you executed above SQL.

This has worked for me sure it was a bit painful to reenable the settings that i needed but yeah seems like the only option actually. As far as i saw this only removes the settings for merge, pull and rebases

[zeripath]

@ghmer

So, our installation broke since 1.15.1 and I had to revert to 1.15.0. We are using postgres as database.

No.

Your installation broke before. It broke the moment you restored from a broken dump created by 1.14.3-1.14.6. The thing is you just haven't realised that it's broken. Do you have your old DB?

Can we expect a fix for this behavior, or do we have to stick with 1.15.0?

Read the comments above.

@NoirPi

This has worked for me sure it was a bit painful to reenable the settings that i needed but yeah seems like the only option actually. As far as i saw this only removes the settings for merge, pull and rebases

Did you not read my comments? I told you how to fix this without loss of these settings.

[zeripath]

I've spent this evening attempting to write a doctor command that would be able to fix these issues.

It's a substantial amount of work - and although it should fix the repo_units issue the login_sources also need to be fixed - Those are a substantially more difficult proposition.

I've pushed up a backport of it to 1.15.

---

People suffering this issue - Please could you take a backup of the repo_unit table, build the PR (backport if you're on 1.15) and then run the doctor command gitea doctor --fix --run fix-broken-repo-units and report if this solves your issue.

If you require builds of PRs you should contact me or another developer on discord.

[zeripath]

OK well that extensive work and request for testing has resulted in a resounding silence. My own very basic tests on SQLite appear to show it works.

I am therefore going to remove the WIP from the PRs fixing the repo_units - these appear to work - but I will do no further work on creating doctor command for the login_source.

If users wish help with fixing login_sources they should contact me or another developer on discord.

[echodreamz]

I've spent this evening attempting to write a doctor command that would be able to fix these issues.

It's a substantial amount of work - and although it should fix the repo_units issue the login_sources also need to be fixed - Those are a substantially more difficult proposition.

I've pushed up a backport of it to 1.15.

People suffering this issue - Please could you take a backup of the repo_unit table, build the PR (backport if you're on 1.15) and then run the doctor command gitea doctor --fix --run fix-broken-repo-units and report if this solves your issue.

If you require builds of PRs you should contact me or another developer on discord.

Thanks for the work on that Zeripath! We had already fixed our DB manually, then went back and redid the settings for each repo a few weeks ago. I am sure this will be beneficial though.

[Snck3rs]

eww guys. This error is an aweful pain in the ass. You should really overthink the dump mechanism if there is no rely on it! And please add a warning to your documentation.

@zeripath thank you a lot for your work! pulling and building the pr would have been a mess working with docker i think, so i've used the manual solution you've provided. Worked fine for me.

Thank you all for your great work on this excellent piece of software!

[zeripath]

eww guys. This error is an aweful pain in the ass. You should really overthink the dump mechanism if there is no rely on it! And please add a warning to your documentation.

Honestly having fixed dump in 1.14.2 I didn't expect it to break in 1.14.3. The answer is that we need better tests for the dump.

Regarding this particular issue though there's a specific warning in the blog and we always advise that you upgrade to the latest minor release. The dump command was fixed as soon as I noticed the problem and I'm not sure what else we could do. Many of these issues have appeared long after 1.14.7 was released which has a fixed dump command.

@zeripath thank you a lot for your work! pulling and building the pr would have been a mess working with docker i think, so i've used the manual solution you've provided. Worked fine for me.

Pulling and building the latest head of release/v1.15 is not difficult. However as I wrote:

If you require builds of PRs you should contact me or another developer on discord.

[zeripath]

How to solve?

Upgrade to at least 1.15.4 and run the command:

gitea doctor --fix --run fix-broken-repo-units

which should fix the repo_unit issues.

What do I do if this fails?

If this command does not work or you have problems with the login_source table:

Open a new issue providing detailed logs and mentioning this issue (#16961), then contact me (@zeripath) on discord or matrix.

Please do not comment on this issue as your message will most likely be lost or ignored.

Please do not open your new issue by clicking on "Reference this in a new issue" as you will not be prompted to give us all the information we need.