Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add option to enable TLS for minio #62

Merged
merged 3 commits into from
Jan 19, 2023
Merged

Add option to enable TLS for minio #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7235464
Add option to enable TLS for minio
saswatamcode Jan 19, 2023
eedac0d
Fix readiness probe
saswatamcode Jan 19, 2023
746bb59
Fix lint
saswatamcode Jan 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,21 +48,20 @@ Let's go through an example leveraging the `go test` flow:

4. Use `e2emon.AsInstrumented` if you want to be able to query your service for metrics, which is a great way to assess it's internal state in tests! For example see following Etcd definition:

```go mdox-exec="sed -n '228,243p' db/db.go"
```go mdox-exec="sed -n '351,363p' db/db.go"
return e2emon.AsInstrumented(env.Runnable(name).WithPorts(map[string]int{AccessPortName: 2379, "metrics": 9000}).Init(
e2e.StartOptions{
Image: o.image,
Command: e2e.NewCommand(
"/usr/local/bin/etcd",
"--listen-client-urls=http://0.0.0.0:2379",
"--advertise-client-urls=http://0.0.0.0:2379",
"--listen-metrics-urls=http://0.0.0.0:9000",
"--log-level=error",
),
Readiness: e2e.NewHTTPReadinessProbe("metrics", "/health", 200, 204),
},
e2e.StartOptions{
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor nit here. Why modify the spacing? ;)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mdox, I think we were printing an extra bracket earlier, which caused the syntax to be a bit off. 🙂

Image: o.image,
Command: e2e.NewCommand(
"/usr/local/bin/etcd",
"--listen-client-urls=http://0.0.0.0:2379",
"--advertise-client-urls=http://0.0.0.0:2379",
"--listen-metrics-urls=http://0.0.0.0:9000",
"--log-level=error",
),
Readiness: e2e.NewHTTPReadinessProbe("metrics", "/health", 200, 204),
},
), "metrics")
}
```

5. Program your scenario as you want. You can start, wait for their readiness, stop, check their metrics and use their network endpoints from both unit test (`Endpoint`) as well as within each workload (`InternalEndpoint`). You can also access workload directory. There is a shared directory across all workloads. Check `Dir` and `InternalDir` runnable methods.
Expand Down
151 changes: 137 additions & 14 deletions db/db.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,20 @@
package e2edb

import (
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/pem"
"fmt"
"math/big"
"net"
"os"
"path/filepath"
"strconv"
"strings"
"time"

"github.com/efficientgo/core/errors"
"github.com/efficientgo/e2e"
e2emon "github.com/efficientgo/e2e/monitoring"
e2eprof "github.com/efficientgo/e2e/profiling"
Expand All @@ -32,6 +40,7 @@ type options struct {

type minioOptions struct {
enableSSE bool
enableTLS bool
}

func WithImage(image string) Option {
Expand All @@ -52,6 +61,12 @@ func WithMinioSSE() Option {
}
}

func WithMinioTLS() Option {
return func(o *options) {
o.minioOptions.enableTLS = true
}
}

const AccessPortName = "http"

func NewPrometheus(env e2e.Environment, name string, opts ...Option) *e2emon.Prometheus {
Expand Down Expand Up @@ -85,7 +100,6 @@ func NewMinio(env e2e.Environment, name, bktName string, opts ...Option) *e2emon
"MINIO_ROOT_USER=" + MinioAccessKey,
"MINIO_ROOT_PASSWORD=" + MinioSecretKey,
"MINIO_BROWSER=" + "off",
"ENABLE_HTTPS=" + "0",
}

f := env.Runnable(name).WithPorts(ports).Future()
Expand All @@ -105,31 +119,140 @@ func NewMinio(env e2e.Environment, name, bktName string, opts ...Option) *e2emon
command += "curl -sSL --tlsv1.3 -O 'https://raw.githubusercontent.com/minio/kes/master/root.key' -O 'https://raw.githubusercontent.com/minio/kes/master/root.cert' && cp root.* /home/me/ && "
}

var readiness e2e.ReadinessProbe

if o.minioOptions.enableTLS {
if err := os.MkdirAll(filepath.Join(f.Dir(), "certs", "CAs"), 0750); err != nil {
return &e2emon.InstrumentedRunnable{Runnable: e2e.NewFailedRunnable(name, errors.Wrap(err, "create certs dir"))}
}

if err := genCerts(
filepath.Join(f.Dir(), "certs", "public.crt"),
filepath.Join(f.Dir(), "certs", "private.key"),
filepath.Join(f.Dir(), "certs", "CAs", "ca.crt"),
fmt.Sprintf("%s-%s", env.Name(), name),
); err != nil {
return &e2emon.InstrumentedRunnable{Runnable: e2e.NewFailedRunnable(name, errors.Wrap(err, "fail to generate certs"))}
}

envVars = append(envVars, "ENABLE_HTTPS="+"1")
command = command + fmt.Sprintf(
"su - me -s /bin/sh -c 'mkdir -p %s && %s /opt/bin/minio server --certs-dir %s/certs --address :%v --quiet %v'",
filepath.Join(f.Dir(), bktName),
strings.Join(envVars, " "),
f.Dir(),
ports[AccessPortName],
f.Dir(),
)

readiness = e2e.NewHTTPSReadinessProbe(
AccessPortName,
"/minio/health/cluster",
200,
200,
)
} else {
envVars = append(envVars, "ENABLE_HTTPS="+"0")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the default, do we need to specify this if we don't want to enable HTTPS? 🤔

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'll address in follow-up

command = command + fmt.Sprintf(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: we could have one command var instead of two separate in each branch and just (not) include --certs-dir depending on HTTPS option

"su - me -s /bin/sh -c 'mkdir -p %s && %s /opt/bin/minio server --address :%v --quiet %v'",
filepath.Join(f.Dir(), bktName),
strings.Join(envVars, " "),
ports[AccessPortName],
f.Dir(),
)

readiness = e2e.NewHTTPReadinessProbe(
AccessPortName,
"/minio/health/cluster",
200,
200,
)
}

return e2emon.AsInstrumented(f.Init(
e2e.StartOptions{
Image: o.image,
// Create the required bucket before starting minio.
Command: e2e.NewCommandWithoutEntrypoint(
"sh",
"-c",
command+fmt.Sprintf(
"su - me -s /bin/sh -c 'mkdir -p %s && %s /opt/bin/minio server --address :%v --quiet %v'",
filepath.Join(f.Dir(), bktName),
strings.Join(envVars, " "),
ports[AccessPortName],
f.Dir(),
),
),
Readiness: e2e.NewHTTPReadinessProbe(
AccessPortName,
"/minio/health/cluster",
200,
200,
command,
),
Readiness: readiness,
},
), AccessPortName)
}

// genCerts generates certificates and writes those to the provided paths.
func genCerts(certPath, privkeyPath, caPath, serverName string) error {
var caRoot = &x509.Certificate{
SerialNumber: big.NewInt(2019),
NotAfter: time.Now().AddDate(10, 0, 0),
IsCA: true,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,
}

var cert = &x509.Certificate{
SerialNumber: big.NewInt(1658),
DNSNames: []string{serverName},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
NotAfter: time.Now().AddDate(10, 0, 0),
SubjectKeyId: []byte{1, 2, 3},
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
KeyUsage: x509.KeyUsageDigitalSignature,
}

caPrivKey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return err
}

certPrivKey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return err
}
// Generate CA cert.
caBytes, err := x509.CreateCertificate(rand.Reader, caRoot, caRoot, &caPrivKey.PublicKey, caPrivKey)
if err != nil {
return err
}
caPEM := pem.EncodeToMemory(&pem.Block{
Type: "CERTIFICATE",
Bytes: caBytes,
})
err = os.WriteFile(caPath, caPEM, 0644)
if err != nil {
return err
}

// Sign the cert with the CA private key.
certBytes, err := x509.CreateCertificate(rand.Reader, cert, caRoot, &certPrivKey.PublicKey, caPrivKey)
if err != nil {
return err
}
certPEM := pem.EncodeToMemory(&pem.Block{
Type: "CERTIFICATE",
Bytes: certBytes,
})
err = os.WriteFile(certPath, certPEM, 0644)
if err != nil {
return err
}

certPrivKeyPEM := pem.EncodeToMemory(&pem.Block{
Type: "RSA PRIVATE KEY",
Bytes: x509.MarshalPKCS1PrivateKey(certPrivKey),
})
err = os.WriteFile(privkeyPath, certPrivKeyPEM, 0644)
if err != nil {
return err
}

return nil
}

func NewConsul(env e2e.Environment, name string, opts ...Option) *e2emon.InstrumentedRunnable {
o := options{image: "consul:1.8.4"}
for _, opt := range opts {
Expand Down