Easy coding | 中文

This repo contains an example structure for a monolithic Go Web Application.

Project Architecture

This project loosely follows Uncle Bob's Clean Architecture.

Project principle

Single source of truth (SSOT)

Features

Prerequest

go install \
    github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest \
    github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@latest \
    google.golang.org/protobuf/cmd/protoc-gen-go@latest \
    google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest \
    github.com/envoyproxy/protoc-gen-validate@latest

pip3 install pre-commit
pre-commit install

go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

Getup and running

Modify the GOPROXY env in the dockerfile, when the download speed is slow

make deps
make run

The following files will be generated

There are three exported ports

Check rest api server

curl http://localhost:10000/ping

Check grpc api server

go run cmd/client/main.go

Open the following url in the browser

Topic1 API management

Motivation

Api is an abstract concept, is language independent, in many cases, there are many files to describe an api

That violate the Single source of truth principle, it should define the api in one place, and generate other files.

Get started

In this topic, we will add a new greet service

api/greet_apis/greet/greet.proto

syntax = "proto3";

package greet;

option go_package = 'easycoding/api/greet';

message HelloRequest {
    string req = 1;
}

message HelloResponse {
    string res = 1;
}

api/greet_apis/greet/rpc.proto

syntax = "proto3";

package greet;

option go_package = 'easycoding/api/greet';

import "google/api/annotations.proto";
import "greet/greet.proto";

// The greet service definition.
service GreetSvc {
    rpc Hello(HelloRequest) returns (HelloResponse) {
        option (google.api.http) = {
            get: "/hello",
        };
    }
}

api/greet_apis/buf.yaml

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

api/buf.work.yaml

   - payment_apis
   - pet_apis
   - ping_apis
   # add new line
   - greet_apis

Run make gen-api in the workspace, the following files will be generated

api/greet/greet.pb.go
api/greet/greet.pb.validate.go
api/greet/greet.swagger.json
api/greet/rpc_grpc.pb.go
api/greet/rpc.pb.go
api/greet/rpc.pb.gw.go
api/greet/rpc.pb.validate.go
api/greet/rpc.swagger.json
api/api.swagger.json

Implement the greet service

internal/service/greet/service.go

package greet

import (
	"context"
	greet_pb "easycoding/api/greet"

	"github.com/sirupsen/logrus"
)

type service struct{}

var _ greet_pb.GreetSvcServer = (*service)(nil)

func New(logger *logrus.Logger) *service {
	return &service{}
}

func (s *service) Hello(
	ctx context.Context,
	req *greet_pb.HelloRequest,
) (*greet_pb.HelloResponse, error) {
	return &greet_pb.HelloResponse{Res: req.Req}, nil
}

Update internal/service/register.go

var endpointFuns = []RegisterHandlerFromEndpoint{
	ping_pb.RegisterPingSvcHandlerFromEndpoint,
	pet_pb.RegisterPetStoreSvcHandlerFromEndpoint,
    // add new line
	greet_pb.RegisterGreetSvcHandlerFromEndpoint,
}

func RegisterServers(grpcServer *grpc.Server, logger *logrus.Logger, db *gorm.DB) {
	ping_pb.RegisterPingSvcServer(grpcServer, ping_svc.New(logger))
	pet_pb.RegisterPetStoreSvcServer(grpcServer, pet_svc.New(logger, db))
    // add new line
	greet_pb.RegisterGreetSvcServer(grpcServer, greet_svc.New())
}

Run the server

make run

Check the rest http server

curl localhost:10000/hello?req=hi

The new api document is in http://localhost:10002/swagger/, if you want to custom the output of openapi see protoc-gen-openapi for more information.

message MyMessage {
  // This comment will end up direcly in your Open API definition
  string uuid = 1 [(grpc.gateway.protoc_gen_openapiv2.options.openapiv2_field) = {description: "The UUID field."}];
}

The new metrics is in http://localhost:10002/metrics, you can use prometheus-client to custom metrics, see go-grpc-prometheus for example.

customizedCounterMetric = prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "demo_server_say_hello_method_handle_count",
    Help: "Total number of RPCs handled on the server.",
}, []string{"name"})

You can add some validate to your api like following

syntax = "proto3";

package greet;

option go_package = 'easycoding/api/greet';

// add new line
import "validate/validate.proto";

message HelloRequest {
    // add validate
    string req = 1[(validate.rules).string = {min_len: 0, max_len: 10}];
}

message HelloResponse {
    string res = 1;
}

Stop the server and run make gen-api and make run again.

Send the following request and will return error, see protoc-gen-validate for more validate rule.

curl localhost:10000/hello?req=hiiiiiiiiii

Validator is checked request in the grpc middleware, you can find some common middlewares in grpc-middleware, or you can custom your own middleware.

Breaking change detection

cd api/pet_apis
buf breaking   --against "../../.git#branch=master,subdir=api/pet_apis"

Incompatible changes

// api/pet_apis/pet/pet.proto
message Pet {
    int32 pet_id = 1;
    string name = 2;
    // change the following type
    // PetType pet_type = 3;
    string pet_type = 3;
}

Check the compatibility

cd api/pet_apis
buf breaking   --against "../../.git#branch=master,subdir=api/pet_apis"

pet/pet.proto:22:5:Field "3" on message "Pet" changed type from "enum" to "string".

Compatible changes

// api/pet_apis/pet/pet.proto
message Pet {
    int32 pet_id = 1;
    string name = 2;
    PetType pet_type = 3;
    // add the following field
    string address = 4;
}

cd api/pet_apis
buf breaking   --against "../../.git#branch=master,subdir=api/pet_apis"

Topic2 Database migrate

Motivation

Write raw sql to operate database is not easy to maintain, we use ORM to interact with database, ent for this project. Another situation we encounter is that we often upgrade the structure of the database, firstly, in many compaines, people who write the code and deploy applications are different, so it is necessary to manage upgrade and downgrade properly, secondly we can intergrate sql files into intergration test to ensure the correctness of database structure, thirdly, it is hard to write up and down sql file manully.

Get started

For the current time, the database test is totally empty, use the following command to create auto migration sql files

make migrate-generate

The following files will be generated, see migrate for more information

migrations/{timestamp}_changes.up.sql
migrations/{timestamp}_changes.down.sql
atlas.sum

Migrate sql to database, in the cloud native scenario, you usually need to start a kubernetes job to migrate the database, so the command is not intergrate with Makefile.

go run cmd/migrate/main.go step --latest

INFO[0000] Start buffering 20220902103358/u changes
INFO[0000] Read and execute 20220902103358/u changes
INFO[0000] Finished 20220902103358/u changes (read 2.679112ms, ran 10.382479ms)

Migrate successful, use describe pets to check the schema of table pets

+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| id        | bigint       | NO   | PRI | NULL    | auto_increment |
| name      | varchar(255) | NO   |     | NULL    |                |
| type      | tinyint      | NO   |     | NULL    |                |
| create_at | timestamp    | NO   |     | NULL    |                |
+-----------+--------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

Update pkg/ent/schema/pet.go

--- a/pkg/ent/schema/pet.go
+++ b/pkg/ent/schema/pet.go
@@ -15,8 +15,8 @@ type Pet struct {
 // Fields of the Pet.
 func (Pet) Fields() []ent.Field {
        return []ent.Field{
                field.String("name").NotEmpty(),
+               field.Int32("age").NonNegative(),
                field.Int8("type").NonNegative(),
                field.Time("create_at").Default(time.Now()),
        }

Generate orm files

go generate ./pkg/ent

Create migration files and two more files will be generated, and there are five files in migrations/pet

make migrate-generate

Step up

go run cmd/migrate/main.go step --latest

Check the current version of database

go run cmd/migrate/main.go version

Version: 20220723150428, Dirty: false

+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| id        | bigint       | NO   | PRI | NULL    | auto_increment |
| name      | varchar(255) | NO   |     | NULL    |                |
| type      | tinyint      | NO   |     | NULL    |                |
| create_at | timestamp    | NO   |     | NULL    |                |
| age       | int          | NO   |     | NULL    |                |
+-----------+--------------+------+-----+---------+----------------+
5 rows in set (0.00 sec)

Downgrade the database version

go run cmd/migrate/main.go step 1 --reverse

Version: 20220723144816, Dirty: false

+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| id        | bigint       | NO   | PRI | NULL    | auto_increment |
| name      | varchar(255) | NO   |     | NULL    |                |
| type      | tinyint      | NO   |     | NULL    |                |
| create_at | timestamp    | NO   |     | NULL    |                |
+-----------+--------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

Topic3 Linter

Motivation

Bad Good 
func (d *Driver) SetTrips(trips []Trip) {
  d.trips = trips
}

trips := ...
d1.SetTrips(trips)

// Did you mean to modify d1.trips?
trips[0] = ...

func (d *Driver) SetTrips(trips []Trip) {
  d.trips = make([]Trip, len(trips))
  copy(d.trips, trips)
}

trips := ...
d1.SetTrips(trips)

// We can now modify trips[0] without affecting d1.trips.
trips[0] = ...

Alignment

Bad Good 
// 12 bytes
type Foo struct {
	aaa bool
	bbb int32
	ссс bool
}

// 8 bytes
type Foo struct {
	aaa bool
	ссс bool
	bbb int32
}

For more information see Uber golang style guide

Get started

make lint

Topic4 Error handling

Concept

Error vs Exception

Errors are the possible problems in program, is part of bussiness, like database connection failed.

Exception is unexpectable problems in program, is not part of bussiness, like nil pointer exception, array outof range.

Guideline

            ┌───────────────┐
            │               │
            │  handle error │
            │               │
            └───────────────┘
                    ▲
                    │
            ┌───────┴───────┐
            │               │
            │  with message │
            │               │
            └───────────────┘
                    ▲
                    │
            ┌───────┴───────┐
            │               │
            │   wrap error  │
            │               │
            └───────────────┘
                    ▲
                    │
            ┌───────┴───────┐
            │               │
            │   raw  error  │
            │               │
            └───────────────┘

Example

// pkg/orm/pet.go
func (pet *Pet) GetPet(db *gorm.DB, id int32) error {
    // err is a raw err from gorm
	if err := db.Take(pet, "id = ?", id).Error; err != nil {
        // check the type of raw error
		if errors.ErrorIs(err, gorm.ErrRecordNotFound) {
            // wrap error
			return errors.ErrNotFound(err)
		}
        // wrap error
		return errors.ErrInternal(err)
	}
	return nil
}

// internal/service/pet/get_pet.go
func (s *service) getPet(
	ctx context.Context,
	req *pet_pb.GetPetRequest,
) (*pet_pb.GetPetResponse, error) {
	pet := &orm.Pet{}
	if err := pet.GetPet(s.DB, req.PetId); err != nil {
        // with message in service
		return nil, errors.WithMessage(err, "get pet failed")
	}
}

// internal/middleware/log/interceptor.go

// Describe how to log error
func Interceptor(logger *logrus.Logger) func(
	ctx context.Context,
	req interface{},
	_ *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler) (interface{}, error) {
	ops := []grpc_logrus.Option{
        // map to log level
		grpc_logrus.WithLevels(levelFunc),
        // add decider
		grpc_logrus.WithDecider(decider),
	}
	entry := logrus.NewEntry(logger)

	logInterceptorBefore := createBeforeInterceptor(entry)
	logInterceptorAfter := createAfterInterceptor(entry)

	return grpc_middleware.ChainUnaryServer(
		logInterceptorBefore,
		grpc_logrus.UnaryServerInterceptor(entry, ops...),
		logInterceptorAfter,
	)
}

// internal/middleware/error/interceptor.go

// error classification
func Interceptor(logger *logrus.Logger) grpc.UnaryServerInterceptor {
	return func(
		ctx context.Context,
		req interface{},
		reqinfo *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler,
	) (interface{}, error) {
		res, err := handler(ctx, req)
		if err == nil {
			return res, err
		}
		var code codes.Code

		switch {
		case errors.ErrorIs(err, errors.InternalError):
			code = codes.Internal
		case errors.ErrorIs(err, errors.InvalidError):
			code = codes.InvalidArgument
		case errors.ErrorIs(err, errors.NotFoundError):
			code = codes.NotFound
		case errors.ErrorIs(err, errors.PermissionError):
			code = codes.PermissionDenied
		case errors.ErrorIs(err, errors.UnauthorizedError):
			code = codes.Unauthenticated
		default:
			logger.WithError(err).WithField("method", reqinfo.FullMethod).
				Warn("invalid err, without using easycoding/pkg/errors")
			return res, err
		}
		s := status.New(code, err.Error())
		return res, s.Err()
	}
}

Topic5 Configuration

Motivation

Configuration is an abstract concept, language independent, in many cases, we describe configuration is many files

That violate the Single source of truth principle, it should define the api in one place, and generate other files.

Configuration should combine the following sources and bind to struct

Topic6 Unit test and coverage

Get started

Run all tests

make test

Run all tests with coverage

make coverage

Run all tests with coverage and open the report in browser

make coverage-html

Topic7 Release and deploy

Topic8 Monitor, logging and trace

TODO

Inspirations