bug: graceful shutdown is not so graceful #164
Description
Is your feature request related to a problem? Please describe.
Compass panics on graceful shutdown with below stacktrace
panic: Drain() is not implemented
goroutine 56 [running]:
google.golang.org/grpc/internal/transport.(*serverHandlerTransport).Drain(0xc000e340a0?)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/handler_server.go:439 +0x27
google.golang.org/grpc.(*Server).GracefulStop(0xc000821880)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1765 +0x3f4
github.com/odpf/compass/internal/server.Serve.func1()
/home/runner/work/compass/compass/internal/server/server.go:157 +0x1bb
created by github.com/odpf/compass/internal/server.Serve
/home/runner/work/compass/compass/internal/server/server.go:143 +0xaae
Checking from this issue, this looks like the expected behaviour when calling grpcServer.GracefulStop().
Compass calls grpcServer.GracefulStop() as part of graceful shutdown process as seen here.
Compass should not panic on graceful shutdown.
Describe the solution you'd like
1. Remove calling GracefulStop
We can remove calling grpcServer.GracefulStop() here as we are actually already handling shutdown gracefully.
2. Catch panic
Another approach if we just catch that panic here and recover properly.
example
defer func() {
if r := recover(); r != nil {
// handle panic
}
}()
if grpcServer != nil {
logger.Warn("stopping grpc server...")
grpcServer.GracefulStop()
}