secrets/database: fixes external plugin reconnect after shutdown for v4 and v5 interface

[austingebauer]

Overview

This PR fixes an issue that prevents external database plugin processes from restarting (via new process) after being killed. The CloseIfShutdown method was not deleting cached connections because v5.ErrPluginShutdown was not in the switch case. This resulted in indefinite plugin shutdown errors when trying to read credentials after a plugin process had been killed. This system behavior existed in the v4 interface and was depended on by customers.

This PR also fixes handling of plugin shutdown for password generation using the v4 interface. If password generation failed with v4.ErrPluginShutdown, the cached connection was never deleted. This resulted in indefinite unable to generate password: plugin shutdown errors when trying to read credentials after a plugin process had been killed.

Testing

v5 Shutdown

I verified that the plugin restarting behavior existed using Vault 1.5.x and vault-plugin-database-oracle 0.2.x (both on v4 interface). I reproduced the issue and tested the fix using Vault 1.6.x+ and vault-plugin-database-oracle 0.3.x+.

Manual testing output:

# Get PID of vault-plugin-database-oracle plugin
$ pgrep oracle
72091

# Kill the plugin
$ kill -9 72091

# Read credentials, which deletes cached connection via CloseIfShutdown() by this PR
$ vault read database/creds/oraclerole
Error reading database/creds/oraclerole: Error making API request.

URL: GET http://127.0.0.1:8200/v1/database/creds/oraclerole
Code: 500. Errors:

* 1 error occurred:
	* plugin shutdown

# Read credentials starts new process and succeeds
$ vault read database/creds/oraclerole
Key                Value
---                -----
lease_id           database/creds/oraclerole/UtYwEjyWq9lOW4T6mcGs16RO
lease_duration     1h
lease_renewable    true
password           a4qcXnsk-redacted-3OkGcJO
username           V_TOKEN_ORACLERO_FSNDBLA36A66E

# See that there is a new PID for vault-plugin-database-oracle
$ pgrep oracle
72590

v4 Shutdown

I reproduced the issue and tested the fix using Vault 1.6.x+ (v5 interface) and vault-plugin-database-oracle 0.2.x (v4 interface).

Manual testing output:

# Get PID of vault-plugin-database-oracle plugin
$ pgrep oracle
14146

# Kill the plugin
$ kill -9 14146

# Read credentials, which deletes cached connection via CloseIfShutdown() by this PR
$ vault read database/creds/oraclerole
Error reading database/creds/oraclerole: Error making API request.

URL: GET http://127.0.0.1:8200/v1/database/creds/oraclerole
Code: 500. Errors:

* 1 error occurred:
	* unable to generate password: plugin shutdown

# Read credentials starts new process and succeeds
$ vault read database/creds/oraclerole
Key                Value
---                -----
lease_id           database/creds/oraclerole/CIBrSIA6LG8T326DN0L97rR4
lease_duration     1h
lease_renewable    true
password           A1a_n0-redacted-JBiuBliwq
username           V_TOKEN_ORACLERO_AEHZFBXBKZFAL

# See that there is a new PID for vault-plugin-database-oracle
$ pgrep oracle
14297

[calvn]

Nice! We should backport this to 1.7x and 1.6.x as well.

[benashz]

LGTM