Zap

Zap is a fast, structured, leveled logging component built with Go.

Official Repository: uber-go/zap: Blazing fast, structured, leveled logging in Go. (github.com)

Official Documentation: zap package - go.uber.org/zap - Go Packages

Installation

go get -u go.uber.org/zap

Quick Start

The official documentation provides two quick start examples. Both are production-level loggers. The first is a Sugar that supports printf style but with relatively lower performance.

logger, _ := zap.NewProduction()
defer logger.Sync() // Flush buffered log writes to file when program exits
sugar := logger.Sugar()
sugar.Infow("failed to fetch URL",
  "url", url,
  "attempt", 3,
  "backoff", time.Second,
)
sugar.Infof("Failed to fetch URL: %s", url)

The second is logger which has better performance but only supports strongly-typed output.

logger, _ := zap.NewProduction()
defer logger.Sync()
logger.Info("failed to fetch URL",
  // Structured context as strongly typed Field values.
  zap.String("url", url),
  zap.Int("attempt", 3),
  zap.Duration("backoff", time.Second),
)

TIP

Zap is very simple to use. The tricky part is configuring a logger suitable for your project. Official examples are few, so read the source code comments more.

Configuration

Generally, logger configuration is written in configuration files. Zap's configuration also supports deserialization from configuration files, but only supports basic configuration. Even for advanced configuration, the official examples are very concise and not sufficient for production use, so we need to discuss the detailed configuration.

First, let's look at the overall configuration struct. We need to understand the meaning of each field.

type Config struct {
    // Minimum log level
   Level AtomicLevel `json:"level" yaml:"level"`
    // Development mode, mainly affects stack traces
   Development bool `json:"development" yaml:"development"`
    // Caller tracking
   DisableCaller bool `json:"disableCaller" yaml:"disableCaller"`
    // Stack traces
   DisableStacktrace bool `json:"disableStacktrace" yaml:"disableStacktrace"`
    // Sampling, recording only representative logs while limiting log performance impact
   Sampling *SamplingConfig `json:"sampling" yaml:"sampling"`
    // Encoding, either json or console mode
   Encoding string `json:"encoding" yaml:"encoding"`
    // Encoder configuration, mainly output formatting configuration
   EncoderConfig zapcore.EncoderConfig `json:"encoderConfig" yaml:"encoderConfig"`
    // Log file output paths
   OutputPaths []string `json:"outputPaths" yaml:"outputPaths"`
    // Error file output paths
   ErrorOutputPaths []string `json:"errorOutputPaths" yaml:"errorOutputPaths"`
    // Add some default output content to logs
   InitialFields map[string]interface{} `json:"initialFields" yaml:"initialFields"`
}

Below are details about encoder configuration

type EncoderConfig struct {
   // Key value, if key is empty, the corresponding property will not be output
   MessageKey     string `json:"messageKey" yaml:"messageKey"`
   LevelKey       string `json:"levelKey" yaml:"levelKey"`
   TimeKey        string `json:"timeKey" yaml:"timeKey"`
   NameKey        string `json:"nameKey" yaml:"nameKey"`
   CallerKey      string `json:"callerKey" yaml:"callerKey"`
   FunctionKey    string `json:"functionKey" yaml:"functionKey"`
   StacktraceKey  string `json:"stacktraceKey" yaml:"stacktraceKey"`
   SkipLineEnding bool   `json:"skipLineEnding" yaml:"skipLineEnding"`
   LineEnding     string `json:"lineEnding" yaml:"lineEnding"`
   // Some custom encoders
   EncodeLevel    LevelEncoder    `json:"levelEncoder" yaml:"levelEncoder"`
   EncodeTime     TimeEncoder     `json:"timeEncoder" yaml:"timeEncoder"`
   EncodeDuration DurationEncoder `json:"durationEncoder" yaml:"durationEncoder"`
   EncodeCaller   CallerEncoder   `json:"callerEncoder" yaml:"callerEncoder"`
   // Logger name encoder
   EncodeName NameEncoder `json:"nameEncoder" yaml:"nameEncoder"`
   // Reflection encoder, mainly for interface{} types, if no default jsonencoder
   NewReflectedEncoder func(io.Writer) ReflectedEncoder `json:"-" yaml:"-"`
   // Console output separator string
   ConsoleSeparator string `json:"consoleSeparator" yaml:"consoleSeparator"`
}

Option is about configuration switches and applications, with many implementations.

type Option interface {
   apply(*Logger)
}

// Option implementation
type optionFunc func(*Logger)

func (f optionFunc) apply(log *Logger) {
  f(log)
}

// Application
func Development() Option {
  return optionFunc(func(log *Logger) {
    log.development = true
  })
}

This is the most commonly used logger core. Its internal fields basically represent our configuration steps. You can also refer to the official steps for deserializing configuration, which are roughly the same.

type ioCore struct {
   // Log level
   LevelEnabler
   // Log encoding
   enc Encoder
   // Log writing
   out WriteSyncer
}

zap.Encoder is responsible for log formatting and encoding

zap.WriteSyncer is responsible for log output, mainly to files and console

zap.LevelEnabler is the minimum log level; logs below this level are no longer output through syncer.

Log Encoding

Log encoding mainly involves formatting details of logs. First, let's look at the output using the original logger directly.

func TestQuickStart(t *testing.T) {
   rawJSON := []byte(`{
     "level": "debug",
     "encoding": "json",
     "outputPaths": ["stdout"],
     "errorOutputPaths": ["stderr"],
     "initialFields": {"foo": "bar"},
     "encoderConfig": {
       "messageKey": "message",
       "levelKey": "level",
       "levelEncoder": "lowercase"
     }
   }`)

   var cfg zap.Config
   if err := json.Unmarshal(rawJSON, &cfg); err != nil {
      panic(err)
   }
   logger := zap.Must(cfg.Build())
   defer logger.Sync()

   logger.Info("logger construction succeeded")
}

{"level":"info","message":"logger construction succeeded","foo":"bar"}

You'll find this log has several issues:

• No timestamp
• No caller information, don't know where this log was output from, making it hard to troubleshoot errors
• No stack trace

Next, let's solve these problems step by step, mainly by modifying zapcore.EncoderConfig. First, we need to write our own configuration file instead of using the official direct deserialization. Create a configuration file config.yml

# Zap Logger Configuration
zap:
  prefix: ZapLogTest
  timeFormat: 2006/01/02 - 15:04:05.00000
  level: debug
  caller: true
  stackTrace: false
  encode: console
  # Where to output logs: file | console | both
  writer: both
  logFile:
    maxSize: 20
    backups: 5
    compress: true
    output:
      - "./log/output.log"

Mapped struct

// ZapConfig
// @Date: 2023-01-09 16:37:05
// @Description: zap logger configuration struct
type ZapConfig struct {
  Prefix     string         `yaml:"prefix" mapstructure:"prefix"`
  TimeFormat string         `yaml:"timeFormat" mapstructure:"timeFormat"`
  Level      string         `yaml:"level" mapstructure:"level"`
  Caller     bool           `yaml:"caller" mapstructure:"caller"`
  StackTrace bool           `yaml:"stackTrace" mapstructure:"stackTrace"`
  Writer     string         `yaml:"writer" mapstructure:"writer"`
  Encode     string         `yaml:"encode" mapstructure:"encode"`
  LogFile    *LogFileConfig `yaml:"logFile" mapstructure:"logFile"`
}

// LogFileConfig
// @Date: 2023-01-09 16:38:45
// @Description: log file configuration struct
type LogFileConfig struct {
  MaxSize  int      `yaml:"maxSize" mapstructure:"maxSize"`
  BackUps  int      `yaml:"backups" mapstructure:"backups"`
  Compress bool     `yaml:"compress" mapstructure:"compress"`
  Output   []string `yaml:"output" mapstructure:"output"`
  Errput   []string `yaml:"errput" mapstructure:"errput"`
}

TIP

Use Viper to read configuration; code omitted here.

type TimeEncoder func(time.Time, PrimitiveArrayEncoder)

TimeEncoder is actually a function. We can use other time encoders provided by the official or write our own.

func CustomTimeFormatEncoder(t time.Time, encoder zapcore.PrimitiveArrayEncoder) {
   encoder.AppendString(global.Config.ZapConfig.Prefix + "\t" + t.Format(global.Config.ZapConfig.TimeFormat))
}

The overall part is as follows

func zapEncoder(config *ZapConfig) zapcore.Encoder {
   // Create a new configuration
   encoderConfig := zapcore.EncoderConfig{
      TimeKey:       "Time",
      LevelKey:      "Level",
      NameKey:       "Logger",
      CallerKey:     "Caller",
      MessageKey:    "Message",
      StacktraceKey: "StackTrace",
      LineEnding:    zapcore.DefaultLineEnding,
      FunctionKey:   zapcore.OmitKey,
   }
   // Custom time format
   encoderConfig.EncodeTime = CustomTimeFormatEncoder
   // Log level in uppercase
   encoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
   // Second-level time interval
   encoderConfig.EncodeDuration = zapcore.SecondsDurationEncoder
   // Short caller output
   encoderConfig.EncodeCaller = zapcore.ShortCallerEncoder
   // Full serialization of logger name
   encoderConfig.EncodeName = zapcore.FullNameEncoder
   // Final log encoding: json or console
   switch config.Encode {
   case "json":
      {
         return zapcore.NewJSONEncoder(encoderConfig)
      }
   case "console":
      {
         return zapcore.NewConsoleEncoder(encoderConfig)
      }
   }
   // Default console
   return zapcore.NewConsoleEncoder(encoderConfig)
}

Log Output

Log output is divided into console output and file output. We can configure dynamically based on the configuration file. If we want log file rotation, we need to use another third-party dependency.

go get -u github.com/natefinch/lumberjack

The final code is as follows

func zapWriteSyncer(cfg *ZapConfig) zapcore.WriteSyncer {
   syncers := make([]zapcore.WriteSyncer, 0, 2)
   // If console output is enabled, add console writer
   if cfg.Writer == config.WriteBoth || cfg.Writer == config.WriteConsole {
      syncers = append(syncers, zapcore.AddSync(os.Stdout))
   }

   // If file logging is enabled, add writers based on file paths
   if cfg.Writer == config.WriteBoth || cfg.Writer == config.WriteFile {
      // Add log output
      for _, path := range cfg.LogFile.Output {
         logger := &lumberjack.Logger{
            Filename:   path, // File path
            MaxSize:    cfg.LogFile.MaxSize, // Size of split files
            MaxBackups: cfg.LogFile.BackUps, // Number of backups
            Compress:   cfg.LogFile.Compress, // Whether to compress
            LocalTime:  true, // Use local time
         }
         syncers = append(syncers, zapcore.Lock(zapcore.AddSync(logger)))
      }
   }
   return zap.CombineWriteSyncers(syncers...)
}

Log Level

The official documentation has enum items for log levels; just use them directly.

func zapLevelEnabler(cfg *ZapConfig) zapcore.LevelEnabler {
   switch cfg.Level {
   case config.DebugLevel:
      return zap.DebugLevel
   case config.InfoLevel:
      return zap.InfoLevel
   case config.ErrorLevel:
      return zap.ErrorLevel
   case config.PanicLevel:
      return zap.PanicLevel
   case config.FatalLevel:
      return zap.FatalLevel
   }
   // Default Debug level
   return zap.DebugLevel
}

Final Build

func InitZap(config *ZapConfig) *zap.Logger {
   // Build encoder
   encoder := zapEncoder(config)
   // Build log level
   levelEnabler := zapLevelEnabler(config)
   // Finally get Core and Options
   subCore, options := tee(config, encoder, levelEnabler)
   // Create Logger
   return zap.New(subCore, options...)
}

// Merge everything
func tee(cfg *ZapConfig, encoder zapcore.Encoder, levelEnabler zapcore.LevelEnabler) (core zapcore.Core, options []zap.Option) {
   sink := zapWriteSyncer(cfg)
   return zapcore.NewCore(encoder, sink, levelEnabler), buildOptions(cfg, levelEnabler)
}

// Build Options
func buildOptions(cfg *ZapConfig, levelEnabler zapcore.LevelEnabler) (options []zap.Option) {
   if cfg.Caller {
      options = append(options, zap.AddCaller())
   }

   if cfg.StackTrace {
      options = append(options, zap.AddStacktrace(levelEnabler))
   }
   return
}

Final result

ZapLogTest      2023/01/09 - 19:44:00.91076     INFO    demo/zap.go:49     Logger initialization complete