Testing

For developers, good testing can detect errors in the program early, avoiding the mental burden caused by bugs due to untimely maintenance. Therefore, it is very necessary to write good tests. Go provides a very convenient and practical command-line tool go test for testing. You can see testing in the standard library and many open-source frameworks. This tool is very easy to use and currently supports the following types of testing:

• Example testing
• Unit testing
• Benchmark testing
• Fuzz testing

Most of the APIs in Go are provided by the standard library testing.

TIP

Execute the go help testfunc command in the command line to see Go's official explanation of the four testing types above.

Writing Conventions

Before starting to write tests, there are a few conventions to note that will make subsequent learning more convenient.

• Test package: Test files are best kept in a separate package, usually named test.
• Test file: Test files usually end with _test.go. For example, to test a certain function, name it function_test.go. If you want to divide more finely based on test type, you can use the test type as a file prefix, such as benchmark_marshaling_test.go or example_marshaling_test.go.
• Test function: Each test file contains several test functions for different tests. For different test types, the naming style of test functions also differs. For example, example tests are ExampleXXXX, unit tests are TestXXXX, benchmark tests are BenchmarkXXXX, and fuzz tests are FuzzXXXX. This way, even without comments, you can know what type of test it is.

TIP

When the package name is testdata, the package is usually intended to store auxiliary data for testing. When executing tests, Go will ignore packages named testdata.

Following the above conventions and developing good testing styles can save a lot of trouble for future maintenance.

Executing Tests

Executing tests mainly uses the go test command. Let's look at actual code examples. Suppose there is a file to be tested /say/hello.go with the following code:

package say

import "fmt"

func Hello() {
  fmt.Println("hello")
}

func GoodBye() {
  fmt.Println("bye")
}

And a test file /test/example_test.go with the following code:

package test

import (
  "golearn/say"
)

func ExampleHello() {
  say.Hello()
  // Output:
  // hello
}

func ExampleGoodBye() {
  say.GoodBye()
  // Output:
  // bye
}

func ExampleSay() {
  say.Hello()
  say.GoodBye()
  // Output:
  // hello
  // bye
}

There are multiple ways to execute these tests. For example, if you want to execute all test cases under the test package, you can directly execute the following command in the test directory:

$ go test ./
PASS
ok      golearn/test    0.422s

./ represents the current directory. Go will recompile all test files under the test directory and then execute all test cases. From the results, you can see that all test cases passed. You can also follow multiple directories as parameters, such as the command below. Obviously, the main project directory does not have test files to execute.

$ go test ./ ../
ok      golearn/test
?       golearn [no test files]

TIP

When there are multiple packages as execution parameters, Go will not re-execute test cases that have already passed successfully. During execution, (cached) will be appended to the end of the line to indicate that the output result is from the previous cache. Go will cache test results when the test flag parameters are in the following set; otherwise, it will not.

-benchtime, -cpu, -list, -parallel, -run, -short, -timeout, -failfast, -v

If you want to disable caching, you can add the parameter -count=1.

Of course, you can also specify a single test file to execute.

$ go test example_test.go
ok      command-line-arguments  0.457s

Or you can specify a single test case in a test file, for example:

$ go test -run ExampleSay
PASS
ok      golearn/test    0.038s

The above three situations all completed the tests, but the output results are too concise. At this time, you can add the -v parameter to make the output more detailed, for example:

$ go test ./ -v
=== RUN   ExampleHello
--- PASS: ExampleHello (0.00s)
=== RUN   ExampleGoodBye
--- PASS: ExampleGoodBye (0.00s)
=== RUN   ExampleSay
--- PASS: ExampleSay (0.00s)
PASS
ok      golearn/test    0.040s

Now you can clearly see the execution order, time consumption, execution status of each test case, and the overall time consumption.

TIP

The go test command runs all unit tests, example tests, and fuzz tests by default. If you add the -bench parameter, it will run all types of tests, such as the command below:

$ go test -bench .

Therefore, you need to use the -run parameter to specify, for example, the command to run only all benchmark tests is as follows:

$ go test -bench . -run ^$

Common Parameters

Go testing has many flag parameters. Below only common parameters will be introduced. For more details, it is recommended to use the go help testflag command to look up yourself.

Parameter	Description
-o file	Specify the name of the compiled binary file
-c	Only compile test files, but do not run
-json	Output test logs in json format
-exec xprog	Run tests using xprog, equivalent to go run
-bench regexp	Select benchmark tests matching regexp
-fuzz regexp	Select fuzz tests matching regexp
-fuzztime t	Time for fuzz test to end automatically, t is time interval, when unit is x, it means count, e.g., 200x
-fuzzminimizetime t	Minimum time for fuzz test to run, same rules as above
-count n	Run tests n times, default 1 time
-cover	Enable test coverage analysis
-covermode set,count,atomic	Set the mode of coverage analysis
-cpu	Execute GOMAXPROCS for test execution
-failfast	After the first test failure, no new tests will be started
-list regexp	List test cases matching regexp
-parallel n	Allow test cases that call t.Parallel to run in parallel, n is the maximum number of parallel
-run regexp	Only run test cases matching regexp
-skip regexp	Skip test cases matching regexp
-timeout d	If a single test execution time exceeds time interval d, it will panic. d is time interval, e.g., 1s, 1ms, 1ns, etc.
-shuffle off,on,N	Shuffle test execution order, N is random seed, default seed is system time
-v	Output more detailed test logs
-benchmem	Statistics memory allocation for benchmark tests
-blockprofile block.out	Statistics goroutine blocking situation during tests and write to file
-blockprofilerate n	Control goroutine blocking statistics frequency, see more details via command go doc runtime.SetBlockProfileRate
-coverprofile cover.out	Statistics coverage test situation and write to file
-cpuprofile cpu.out	Statistics CPU situation and write to file
-memprofile mem.out	Statistics memory allocation situation and write to file
-memprofilerate n	Control memory allocation statistics frequency, see more details via command go doc runtime.MemProfileRate
-mutexprofile mutex.out	Statistics lock contention situation and write to file
-mutexprofilefraction n	Set statistics for n goroutines competing for one mutex lock
-trace trace.out	Write execution trace situation to file
-outputdir directory	Specify output directory for the above statistics files, default is go test execution directory

Example Testing

Example testing is not like the other three types of testing to discover problems in the program. It is more for demonstrating the usage of a certain function, serving as documentation. Example testing is not an officially defined concept, nor is it a mandatory specification. It is more like an engineering convention, and whether to follow it depends on the developer. Example testing appears very frequently in the standard library, usually code examples written by officials for the standard library, such as the ExampleWithDeadline test function in the standard library context/example_test.go. This function demonstrates the basic usage of DeadlineContext:

// This example passes a context with an arbitrary deadline to tell a blocking
// function that it should abandon its work as soon as it gets to it.
func ExampleWithDeadline() {
   d := time.Now().Add(shortDuration)
   ctx, cancel := context.WithDeadline(context.Background(), d)

   // Even though ctx will be expired, it is good practice to call its
   // cancellation function in any case. Failure to do so may keep the
   // context and its parent alive longer than necessary.
   defer cancel()

   select {
   case <-time.After(1 * time.Second):
      fmt.Println("overslept")
   case <-ctx.Done():
      fmt.Println(ctx.Err())
   }

   // Output:
   // context deadline exceeded
}

On the surface, this test function is just an ordinary function, but example testing is mainly reflected by the Output comment. When the function to be tested has only one line of output, use the Output comment to check the output. First, create a file named hello.go and write the following code:

package say

import "fmt"

func Hello() {
  fmt.Println("hello")
}

func GoodBye() {
  fmt.Println("bye")
}

The SayHello function is the function to be tested. Then create a test file example_test.go and write the following code:

package test

import (
  "golearn/say"
)

func ExampleHello() {
  say.Hello()
  // Output:
  // hello
}

func ExampleGoodBye() {
  say.GoodBye()
  // Output:
  // bye
}

func ExampleSay() {
  say.Hello()
  say.GoodBye()
  // Output:
  // hello
  // bye
}

The Output comment in the function indicates checking whether the function output is hello. Next, execute the test command to see the results.

$ go test -v
=== RUN   ExampleHello
--- PASS: ExampleHello (0.00s)
=== RUN   ExampleGoodBye
--- PASS: ExampleGoodBye (0.00s)
=== RUN   ExampleSay
--- PASS: ExampleSay (0.00s)
PASS
ok      golearn/test    0.448s

From the results, you can see that all tests have passed. Regarding Output, there are the following writing methods. The first is a single line of output, meaning to check whether the output of the function is hello:

// Output:
// hello

The second is multiple lines of output, that is, check whether the output matches in order:

// Output:
// hello
// bye

The third is unordered output, that is, match multiple lines of output without following the order:

// Unordered output:
// bye
// hello

It should be noted that for test functions, only when the last few lines are Output comments will they be considered example tests; otherwise, they are just ordinary functions and will not be executed by Go.

Unit Testing

Unit testing is testing the smallest testable unit in the software. The size definition of a unit depends on the developer. It could be a struct, a package, a function, or a type. Below is still a demonstration through examples. First, create the /tool/math.go file and write the following code:

package tool

type Number interface {
  ~int8 | ~int16 | ~int32 | ~int64 | ~int
}

func SumInt[T Number](a, b T) T {
  return a + b
}

func Equal[T Number](a, b T) bool {
  return a == b
}

Then create a test file /tool_test/unit_test.go. For unit testing, the naming can be unit_test or use the package or function to be tested as the file prefix.

package test_test

import (
  "golearn/tool"
  "testing"
)

func TestSum(t *testing.T) {
  a, b := 10, 101
  expected := 111

  actual := tool.SumInt(a, b)
  if actual != expected {
    t.Errorf("Sum(%d,%d) expected %d,actual is %d", a, b, expected, actual)
  }
}

func TestEqual(t *testing.T) {
  a, b := 10, 101
  expected := false

  actual := tool.Equal(a, b)
  if actual != expected {
    t.Errorf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
  }
}

For unit testing, the naming style of each test case is TestXXXX, and the function parameter must be t *testing.T. testing.T is a struct provided by the testing package for convenient testing, providing many available methods. The t.Errorf in the example is equivalent to t.Logf, used for formatting output of test failure log information. Other commonly used methods include t.Fail to mark the current case as test failure. Similar functionality includes t.FailNow, which also marks as test failure, but the former continues to execute after failure, while the latter stops execution directly. As in the example below, modify the expected result to an incorrect result:

package tool_test

import (
  "golearn/tool"
  "testing"
)

func TestSum(t *testing.T) {
  a, b := 10, 101
  expected := 110

  actual := tool.SumInt(a, b)
  if actual != expected {
        // Errorf uses t.Fail() internally
    t.Errorf("Sum(%d,%d) expected %d,actual is %d", a, b, expected, actual)
  }
  t.Log("test finished")
}

func TestEqual(t *testing.T) {
  a, b := 10, 101
  expected := true

  actual := tool.Equal(a, b)
  if actual != expected {
        // Fatalf uses t.FailNow() internally
    t.Fatalf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
  }
  t.Log("test finished")
}

Execute the above test output as follows:

$ go test tool_test.go -v
=== RUN   TestSum
    tool_test.go:14: Sum(10,101) expected 110,actual is 111
    tool_test.go:16: test finished
--- FAIL: TestSum (0.00s)
=== RUN   TestEqual
    tool_test.go:25: Sum(10,101) expected true,actual is false
--- FAIL: TestEqual (0.00s)
FAIL    command-line-arguments  0.037s

From the test logs, you can see that although the TestSum case failed, it still output "test finished", while TestEqual did not. Similarly, t.SkipNow marks the current case as SKIP and stops execution, but will continue to execute in the next round of testing.

package tool_test

import (
   "golearn/tool"
   "testing"
)

func TestSum(t *testing.T) {
   a, b := 10, 101
   expected := 110

   actual := tool.SumInt(a, b)
   if actual != expected {
      t.Skipf("Sum(%d,%d) expected %d,actual is %d", a, b, expected, actual)
   }
   t.Log("test finished")
}

func TestEqual(t *testing.T) {
   a, b := 10, 101
   expected := true

   actual := tool.Equal(a, b)
   if actual != expected {
      t.Fatalf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
   }
   t.Log("test finished")
}

When executing the test, modify the test count to 2:

$ go test tool_test.go -v -count=2
=== RUN   TestSum
    tool_test.go:14: Sum(10,101) expected 110,actual is 111
--- SKIP: TestSum (0.00s)
=== RUN   TestEqual
    tool_test.go:25: Sum(10,101) expected true,actual is false
--- FAIL: TestEqual (0.00s)
=== RUN   TestSum
    tool_test.go:14: Sum(10,101) expected 110,actual is 111
--- SKIP: TestSum (0.00s)
=== RUN   TestEqual
    tool_test.go:25: Sum(10,101) expected true,actual is false
--- FAIL: TestEqual (0.00s)
FAIL    command-line-arguments  0.468s

In the above example, "test finished" is output on the last line to indicate the test is complete. Actually, you can use t.Cleanup to register a cleanup function specifically for this purpose. This function will be executed when the test case ends, as follows:

package tool_test

import (
  "golearn/tool"
  "testing"
)

func finished(t *testing.T) {
  t.Log("test finished")
}

func TestSum(t *testing.T) {
  t.Cleanup(func() {
    finished(t)
  })

  a, b := 10, 101
  expected := 111

  actual := tool.SumInt(a, b)
  if actual != expected {
    t.Skipf("Sum(%d,%d) expected %d,actual is %d", a, b, expected, actual)
  }

}

func TestEqual(t *testing.T) {
  t.Cleanup(func() {
    finished(t)
  })

  a, b := 10, 101
  expected := false

  actual := tool.Equal(a, b)
  if actual != expected {
    t.Fatalf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
  }
}

After executing the test, the output is as follows:

$ go test tool_test.go -v
=== RUN   TestSum
    tool_test.go:9: test finished
--- PASS: TestSum (0.00s)
=== RUN   TestEqual
    tool_test.go:9: test finished
--- PASS: TestEqual (0.00s)
PASS
ok      command-line-arguments  0.462s

Helper

Using t.Helper() can mark the current function as a helper function. Helper functions will not be executed as a separate test case. When logging, the line number output is also the caller's line number of the helper function. This can make log analysis more accurate and avoid redundant information. For example, the above t.Cleanup example can be modified to a helper function, as follows:

package tool_test

import (
   "golearn/tool"
   "testing"
)

func CleanupHelper(t *testing.T) {
   t.Helper()
   t.Log("test finished")
}

func TestSum(t *testing.T) {
   t.Cleanup(func() {
      CleanupHelper(t)
   })

   a, b := 10, 101
   expected := 111

   actual := tool.SumInt(a, b)
   if actual != expected {
      t.Skipf("Sum(%d,%d) expected %d,actual is %d", a, b, expected, actual)
   }

}

func TestEqual(t *testing.T) {
   t.Cleanup(func() {
      CleanupHelper(t)
   })

   a, b := 10, 101
   expected := false

   t.Helper()
   actual := tool.Equal(a, b)
   if actual != expected {
      t.Fatalf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
   }
}

After executing the test, the output information is as follows. The difference from before is that the line number of "test finished" becomes the caller's line number:

$ go test tool_test.go -v
=== RUN   TestSum
    tool_test.go:15: test finished
--- PASS: TestSum (0.00s)
=== RUN   TestEqual
    tool_test.go:30: test finished
--- PASS: TestEqual (0.00s)
PASS
ok      command-line-arguments  0.464s

TIP

The above operations can only be performed in the main test, that is, directly executed test cases. If used in sub-tests, it will panic.

Sub-tests

In some cases, you may need to test additional test cases within a test case. This nested test case is generally called a sub-test. Through the t.Run() method, the method signature is as follows:

// Run method opens a new goroutine to run the sub-test, blocks and waits for function f to complete before returning
// Return value is whether the test passed
func (t *T) Run(name string, f func(t *T)) bool

Below is an example:

func TestTool(t *testing.T) {
  t.Run("tool.Sum(10,101)", TestSum)
  t.Run("tool.Equal(10,101)", TestEqual)
}

After execution, the result is as follows:

$ go test -run TestTool -v
=== RUN   TestTool
=== RUN   TestTool/tool.Sum(10,101)
    tool_test.go:15: test finished
=== RUN   TestTool/tool.Equal(10,101)
    tool_test.go:30: test finished
--- PASS: TestTool (0.00s)
    --- PASS: TestTool/tool.Sum(10,101) (0.00s)
    --- PASS: TestTool/tool.Equal(10,101) (0.00s)
PASS
ok      golearn/tool_test       0.449s

From the output, you can clearly see the parent-child hierarchy. In the above example, the second sub-test will not be executed until the first sub-test is completed. You can use t.Parallel() to mark test cases as runnable in parallel. In this case, the output order will be uncertain.

package tool_test

import (
  "golearn/tool"
  "testing"
)

func CleanupHelper(t *testing.T) {
  t.Helper()
  t.Log("test finished")
}

func TestSum(t *testing.T) {
  t.Parallel()
  t.Cleanup(func() {
    CleanupHelper(t)
  })

  a, b := 10, 101
  expected := 111

  actual := tool.SumInt(a, b)
  if actual != expected {
    t.Skipf("Sum(%d,%d) expected %d,actual is %d", a, b, expected, actual)
  }

}

func TestEqual(t *testing.T) {
  t.Parallel()
  t.Cleanup(func() {
    CleanupHelper(t)
  })

  a, b := 10, 101
  expected := false

  actual := tool.Equal(a, b)
  if actual != expected {
    t.Fatalf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
  }
}

func TestToolParallel(t *testing.T) {
  t.Log("setup")
  t.Run("tool.Sum(10,101)", TestSum)
  t.Run("tool.Equal(10,101)", TestEqual)
  t.Log("teardown")
}

After executing the test, the output is as follows:

$ go test -run TestTool -v
=== RUN   TestToolParallel
    tool_test.go:46: setup
=== RUN   TestToolParallel/tool.Sum(10,101)
=== PAUSE TestToolParallel/tool.Sum(10,101)
=== RUN   TestToolParallel/tool.Equal(10,101)
=== PAUSE TestToolParallel/tool.Equal(10,101)
=== NAME  TestToolParallel
    tool_test.go:49: teardown
=== CONT  TestToolParallel/tool.Sum(10,101)
=== CONT  TestToolParallel/tool.Equal(10,101)
=== NAME  TestToolParallel/tool.Sum(10,101)
    tool_test.go:16: test finished
=== NAME  TestToolParallel/tool.Equal(10,101)
    tool_test.go:32: test finished
--- PASS: TestToolParallel (0.00s)
    --- PASS: TestToolParallel/tool.Sum(10,101) (0.00s)
    --- PASS: TestToolParallel/tool.Equal(10,101) (0.00s)
PASS
ok      golearn/tool_test       0.444s

From the test results, you can clearly see a blocking wait process. When executing test cases concurrently, like the above example, it certainly cannot proceed normally because subsequent code cannot guarantee synchronous execution. At this time, you can choose to nest another layer of t.Run(), as follows:

func TestToolParallel(t *testing.T) {
  t.Log("setup")
  t.Run("process", func(t *testing.T) {
    t.Run("tool.Sum(10,101)", TestSum)
    t.Run("tool.Equal(10,101)", TestEqual)
  })
  t.Log("teardown")
}

Execute again, and you can see the normal execution results:

$ go test -run TestTool -v
=== RUN   TestToolParallel
    tool_test.go:46: setup
=== RUN   TestToolParallel/process
=== RUN   TestToolParallel/process/tool.Sum(10,101)
=== PAUSE TestToolParallel/process/tool.Sum(10,101)
=== RUN   TestToolParallel/process/tool.Equal(10,101)
=== PAUSE TestToolParallel/process/tool.Equal(10,101)
=== CONT  TestToolParallel/process/tool.Sum(10,101)
=== CONT  TestToolParallel/process/tool.Equal(10,101)
=== NAME  TestToolParallel/process/tool.Sum(10,101)
    tool_test.go:16: test finished
=== NAME  TestToolParallel/process/tool.Equal(10,101)
    tool_test.go:32: test finished
=== NAME  TestToolParallel
    tool_test.go:51: teardown
--- PASS: TestToolParallel (0.00s)
    --- PASS: TestToolParallel/process (0.00s)
        --- PASS: TestToolParallel/process/tool.Sum(10,101) (0.00s)
        --- PASS: TestToolParallel/process/tool.Equal(10,101) (0.00s)
PASS
ok      golearn/tool_test       0.450s

Table-Driven Style

In the above unit tests, the test input data are manually declared variables. When the data volume is small, it doesn't matter, but if you want to test multiple sets of data, it is not practical to declare variables to create test data. Therefore, in general, try to use the form of struct slices. The struct is a temporarily declared anonymous struct. Because this coding style looks like a table, it is called table-driven. Below is an example. This is an example of manually declaring multiple variables to create test data. If there are multiple sets of data, it is not very intuitive, so modify it to table-driven style:

func TestEqual(t *testing.T) {
  t.Cleanup(func() {
    CleanupHelper(t)
  })

  a, b := 10, 101
  expected := false
  actual := tool.Equal(a, b)
  if actual != expected {
    t.Fatalf("Sum(%d,%d) expected %t,actual is %t", a, b, expected, actual)
  }
}

The modified code is as follows:

func TestEqual(t *testing.T) {
  t.Cleanup(func() {
    CleanupHelper(t)
  })

  // table driven style
  testData := []struct {
    a, b int
    exp  bool
  }{
    {10, 101, false},
    {5, 5, true},
    {30, 32, false},
    {100, 101, false},
    {2, 3, false},
    {4, 4, true},
  }

  for _, data := range testData {
    if actual := tool.Equal(data.a, data.b); actual != data.exp {
      t.Fatalf("Sum(%d,%d) expected %t,actual is %t", data.a, data.b, data.exp, actual)
    }
  }
}

This test data looks much more intuitive.

Benchmark Testing

Benchmark testing, also known as performance testing, is usually used to test program performance indicators such as memory usage, CPU usage, execution time, etc. For benchmark testing, test files usually end with bench_test.go, and test case functions must be in BenchmarkXXXX format.

Below is an example of benchmark testing using a string concatenation performance comparison. First, create the file /tool/strConcat.go. As everyone knows, directly using strings for + concatenation has very low performance, while using strings.Builder is much better. In the /tool/strings.go file, create two functions for string concatenation in two ways:

package tool

import "strings"


func ConcatStringDirect(longString string) {
   res := ""
   for i := 0; i < 100_000.; i++ {
      res += longString
   }
}

func ConcatStringWithBuilder(longString string) {
   var res strings.Builder
   for i := 0; i < 100_000.; i++ {
      res.WriteString(longString)
   }
}

Then create a test file /tool_test/bench_tool_test.go with the following code:

package tool_test

import (
  "golearn/tool"
  "testing"
)

var longString = "longStringlongStringlongStringlongStringlongStringlongStringlongStringlongString"

func BenchmarkConcatDirect(b *testing.B) {
  for i := 0; i < b.N; i++ {
    tool.ConcatStringDirect(longString)
  }
}

func BenchmarkConcatBuilder(b *testing.B) {
  for i := 0; i < b.N; i++ {
    tool.ConcatStringWithBuilder(longString)
  }
}

Execute the test command. The command enables detailed logs and memory analysis, specifies the list of CPU cores to use, and executes each test case twice. The output is as follows:

$ go test -v -benchmem -bench . -run bench_tool_test.go -cpu=2,4,8 -count=2
goos: windows
goarch: amd64
pkg: golearn/tool_test
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkConcatDirect
BenchmarkConcatDirect-2                4         277771375 ns/op        4040056736 B/op    10000 allocs/op
BenchmarkConcatDirect-2                4         278500125 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-4                1        1153796000 ns/op        4040068784 B/op    10126 allocs/op
BenchmarkConcatDirect-4                1        1211017600 ns/op        4040073104 B/op    10171 allocs/op
BenchmarkConcatDirect-8                2         665460800 ns/op        4040077760 B/op    10219 allocs/op
BenchmarkConcatDirect-8                2         679774450 ns/op        4040080064 B/op    10243 allocs/op
BenchmarkConcatBuilder
BenchmarkConcatBuilder-2            3428            344530 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            3579            351858 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-4            2448            736177 ns/op         4128185 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1688            662993 ns/op         4128185 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1958            550333 ns/op         4128199 B/op         29 allocs/op
BenchmarkConcatBuilder-8            2174            552113 ns/op         4128196 B/op         29 allocs/op
PASS
ok      golearn/tool_test       21.381s

Below is an explanation of the benchmark test output results. goos represents the operating system running, goarch represents the CPU architecture, pkg is the package where the test is located, and cpu is some information about the CPU. The results of each test case below are separated by each benchmark test name delimiter. The first column BenchmarkConcatDirect-2, the 2 represents the number of CPU cores used. The second column 4 represents the size of b.N in the code, which is the number of loop iterations in the benchmark test. The third column 277771375 ns/op represents the time consumed per loop iteration, ns is nanoseconds. The fourth column 4040056736 B/op represents the byte size of memory allocated per loop iteration. The fifth column 10000 allocs/op represents the number of memory allocations per loop iteration.

Obviously, according to the test results, the performance of using strings.Builder is far higher than using + to concatenate strings. Intuitive data comparison of performance is the purpose of benchmark testing.

benchstat

benchstat is an open-source performance test analysis tool. The sample size of the above performance test is only two groups. Once the number of samples increases, manual analysis will be very time-consuming and laborious. This tool was born to solve performance analysis problems.

First, you need to download the tool:

$ go install golang.org/x/perf/benchstat

Execute the benchmark test twice. This time, modify the sample size to 5 and output to old.txt and new.txt files respectively for comparison. First execution result:

$ go test -v -benchmem -bench . -run bench_tool_test.go -cpu=2,4,8 -count=5 | tee -a old.txt
goos: windows
goarch: amd64
pkg: golearn/tool_test
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkConcatDirect
BenchmarkConcatDirect-2                4         290535650 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         298974625 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         299637800 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         276487000 ns/op        4040056784 B/op    10001 allocs/op
BenchmarkConcatDirect-2                4         356465275 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-4                2         894723200 ns/op        4040077424 B/op    10216 allocs/op
BenchmarkConcatDirect-4                2         785830400 ns/op        4040078288 B/op    10225 allocs/op
BenchmarkConcatDirect-4                2         743634000 ns/op        4040077568 B/op    10217 allocs/op
BenchmarkConcatDirect-4                2         953802700 ns/op        4040075408 B/op    10195 allocs/op
BenchmarkConcatDirect-4                2         953028750 ns/op        4040077520 B/op    10217 allocs/op
BenchmarkConcatDirect-8                2         684023150 ns/op        4040086784 B/op    10313 allocs/op
BenchmarkConcatDirect-8                2         634380250 ns/op        4040090528 B/op    10352 allocs/op
BenchmarkConcatDirect-8                2         685030600 ns/op        4040090768 B/op    10355 allocs/op
BenchmarkConcatDirect-8                2         817909650 ns/op        4040089808 B/op    10345 allocs/op
BenchmarkConcatDirect-8                2         600078100 ns/op        4040095664 B/op    10406 allocs/op
BenchmarkConcatBuilder
BenchmarkConcatBuilder-2            2925            419651 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            2961            423899 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            2714            422275 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            2848            452255 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            2612            454452 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-4             974           1158000 ns/op         4128189 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1098           1068682 ns/op         4128192 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1042           1056570 ns/op         4128194 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1280            978213 ns/op         4128191 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1538           1162108 ns/op         4128190 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1744            700824 ns/op         4128203 B/op         29 allocs/op
BenchmarkConcatBuilder-8            2235            759537 ns/op         4128201 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1556            736455 ns/op         4128204 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1592            825794 ns/op         4128201 B/op         29 allocs/op
BenchmarkConcatBuilder-8            2263            717285 ns/op         4128203 B/op         29 allocs/op
PASS
ok      golearn/tool_test       56.742s

Second execution result:

$ go test -v -benchmem -bench . -run bench_tool_test.go -cpu=2,4,8 -count=5 | tee -a new.txt
goos: windows
goarch: amd64
pkg: golearn/tool_test
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkConcatDirect
BenchmarkConcatDirect-2                4         285074900 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         291517150 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         281901975 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         292320625 ns/op        4040056592 B/op     9999 allocs/op
BenchmarkConcatDirect-2                4         286723000 ns/op        4040056952 B/op    10002 allocs/op
BenchmarkConcatDirect-4                1        1188983000 ns/op        4040071856 B/op    10158 allocs/op
BenchmarkConcatDirect-4                1        1080713900 ns/op        4040070800 B/op    10147 allocs/op
BenchmarkConcatDirect-4                1        1203622300 ns/op        4040067344 B/op    10111 allocs/op
BenchmarkConcatDirect-4                1        1045291300 ns/op        4040070224 B/op    10141 allocs/op
BenchmarkConcatDirect-4                1        1123163300 ns/op        4040070032 B/op    10139 allocs/op
BenchmarkConcatDirect-8                2         790421300 ns/op        4040076656 B/op    10208 allocs/op
BenchmarkConcatDirect-8                2         659047300 ns/op        4040079488 B/op    10237 allocs/op
BenchmarkConcatDirect-8                2         712991800 ns/op        4040077184 B/op    10213 allocs/op
BenchmarkConcatDirect-8                2         706605350 ns/op        4040078000 B/op    10222 allocs/op
BenchmarkConcatDirect-8                2         656195700 ns/op        4040085248 B/op    10297 allocs/op
BenchmarkConcatBuilder
BenchmarkConcatBuilder-2            2726            386412 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            3439            335358 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            3376            338957 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            3870            326301 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-2            4285            339596 ns/op         4128176 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1663            671535 ns/op         4128187 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1507            744885 ns/op         4128191 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1353           1097800 ns/op         4128187 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1388           1006019 ns/op         4128189 B/op         29 allocs/op
BenchmarkConcatBuilder-4            1635            993764 ns/op         4128189 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1332            783599 ns/op         4128198 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1818            729821 ns/op         4128202 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1398            780614 ns/op         4128202 B/op         29 allocs/op
BenchmarkConcatBuilder-8            1526            750513 ns/op         4128204 B/op         29 allocs/op
BenchmarkConcatBuilder-8            2164            704798 ns/op         4128204 B/op         29 allocs/op
PASS
ok      golearn/tool_test       50.387s

Then use benchstat for comparison:

$ benchstat old.txt new.txt
goos: windows
goarch: amd64
pkg: golearn/tool_test
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
                │    old.txt    │               new.txt                │
                │    sec/op     │    sec/op      vs base               │
ConcatDirect-2     299.0m ± ∞ ¹    286.7m ± ∞ ¹        ~ (p=0.310 n=5)
ConcatDirect-4     894.7m ± ∞ ¹   1123.2m ± ∞ ¹  +25.53% (p=0.008 n=5)
ConcatDirect-8     684.0m ± ∞ ¹    706.6m ± ∞ ¹        ~ (p=0.548 n=5)
ConcatBuilder-2    423.9µ ± ∞ ¹    339.0µ ± ∞ ¹  -20.04% (p=0.008 n=5)
ConcatBuilder-4   1068.7µ ± ∞ ¹    993.8µ ± ∞ ¹        ~ (p=0.151 n=5)
ConcatBuilder-8    736.5µ ± ∞ ¹    750.5µ ± ∞ ¹        ~ (p=0.841 n=5)
geomean            19.84m          19.65m         -0.98%
¹ need >= 6 samples for confidence interval at level 0.95

                │    old.txt    │                new.txt                │
                │     B/op      │     B/op       vs base                │
ConcatDirect-2    3.763Gi ± ∞ ¹   3.763Gi ± ∞ ¹       ~ (p=1.000 n=5)
ConcatDirect-4    3.763Gi ± ∞ ¹   3.763Gi ± ∞ ¹  -0.00% (p=0.008 n=5)
ConcatDirect-8    3.763Gi ± ∞ ¹   3.763Gi ± ∞ ¹  -0.00% (p=0.008 n=5)
ConcatBuilder-2   3.937Mi ± ∞ ¹   3.937Mi ± ∞ ¹       ~ (p=1.000 n=5) ²
ConcatBuilder-4   3.937Mi ± ∞ ¹   3.937Mi ± ∞ ¹       ~ (p=0.079 n=5)
ConcatBuilder-8   3.937Mi ± ∞ ¹   3.937Mi ± ∞ ¹       ~ (p=0.952 n=5)
geomean           123.2Mi         123.2Mi        -0.00%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

                │   old.txt    │               new.txt                │
                │  allocs/op   │  allocs/op    vs base                │
ConcatDirect-2    9.999k ± ∞ ¹   9.999k ± ∞ ¹       ~ (p=1.000 n=5)
ConcatDirect-4    10.22k ± ∞ ¹   10.14k ± ∞ ¹  -0.74% (p=0.008 n=5)
ConcatDirect-8    10.35k ± ∞ ¹   10.22k ± ∞ ¹  -1.26% (p=0.008 n=5)
ConcatBuilder-2    29.00 ± ∞ ¹    29.00 ± ∞ ¹       ~ (p=1.000 n=5) ²
ConcatBuilder-4    29.00 ± ∞ ¹    29.00 ± ∞ ¹       ~ (p=1.000 n=5) ²
ConcatBuilder-8    29.00 ± ∞ ¹    29.00 ± ∞ ¹       ~ (p=1.000 n=5) ²
geomean            543.6          541.7        -0.33%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

From the results, you can see that benchstat divides it into three groups: time consumption, memory usage, and memory allocation count. Among them, geomean is the average value, p is the significance level of the sample, the critical interval is usually 0.05, and above 0.05 is not very credible. Take one of the data as follows:

          │    sec/op     │    sec/op      vs base               │
ConcatDirect-4     894.7m ± ∞ ¹   1123.2m ± ∞ ¹  +25.53% (p=0.008 n=5)

You can see that old execution time is 894.7ms, new execution time is 1123.2ms, which is an increase of 25.53% in time consumption.

Fuzz Testing

Fuzz testing is a new feature launched in GO1.18, belonging to an enhancement of unit testing and benchmark testing. The difference is that the test data of the former two need to be manually written by developers, while fuzz testing can generate random test data through a corpus. For more concepts about fuzz testing in Go, you can go to Go Fuzzing. The benefit of fuzz testing is that, compared to fixed test data, random data can better test the boundary conditions of the program. Below is an example from the official tutorial. This time, the function to be tested is a string reversal function. First, create the file /tool/strings.go and write the following code:

package tool

func Reverse(s string) string {
  b := []byte(s)
  for i, j := 0, len(b)-1; i < len(b)/2; i, j = i+1, j-1 {
    b[i], b[j] = b[j], b[i]
  }
  return string(b)
}

Create a fuzz test file /tool_test/fuzz_tool_test.go and write the following code:

package tool

import (
  "golearn/tool"
  "testing"
  "unicode/utf8"
)

func FuzzReverse(f *testing.F) {
  testdata := []string{"hello world!", "nice to meet you", "good bye!"}
  for _, data := range testdata {
    f.Add(data)
  }

  f.Fuzz(func(t *testing.T, str string) {
    first := tool.Reverse(str)
    second := tool.Reverse(first)
    t.Logf("str:%q,first:%q,second:%q", str, first, second)
    if str != second {
      t.Errorf("before: %q, after: %q", str, second)
    }
    if utf8.ValidString(str) && !utf8.ValidString(first) {
      t.Errorf("Reverse produced invalid UTF-8 string %q %q", str, first)
    }
  })
}

In fuzz testing, you first need to add data to the corpus seed bank. In the example, f.Add() is used to add, which helps generate random test data subsequently. Then use f.Fuzz(fn) to test. The function signature is as follows:

func (f *F) Fuzz(ff any)

func (f *F) Add(args ...any)

fn is similar to the logic of a unit test function. The first parameter of the function must be t *testing.T, followed by the parameters you want to generate. Since the input string is unpredictable, here we use the method of reversing twice to verify. Execute the following command:

$ go test -run Fuzz -v
=== RUN   FuzzReverse
=== RUN   FuzzReverse/seed#0
    fuzz_tool_test.go:18: str:"hello world!",first:"!dlrow olleh",second:"hello world!"
=== RUN   FuzzReverse/seed#1
    fuzz_tool_test.go:18: str:"nice to meet you",first:"uoy teem ot ecin",second:"nice to meet you"
=== RUN   FuzzReverse/seed#2
    fuzz_tool_test.go:18: str:"good bye!",first:"!eyb doog",second:"good bye!"
--- PASS: FuzzReverse (0.00s)
    --- PASS: FuzzReverse/seed#0 (0.00s)
    --- PASS: FuzzReverse/seed#1 (0.00s)
    --- PASS: FuzzReverse/seed#2 (0.00s)
PASS
ok      golearn/tool_test       0.539s

When the parameter does not carry -fuzz, random test data will not be generated. Only data in the corpus will be passed to the test function. From the results, you can see that all tests passed. Using it this way is equivalent to unit testing, but there is actually a problem. Below, add the -fuzz parameter and execute again:

$ go test -fuzz . -fuzztime 30s -run Fuzz -v
=== RUN   FuzzReverse
fuzz: elapsed: 0s, gathering baseline coverage: 0/217 completed
fuzz: minimizing 91-byte failing input file
fuzz: elapsed: 0s, gathering baseline coverage: 15/217 completed
--- FAIL: FuzzReverse (0.13s)
    --- FAIL: FuzzReverse (0.00s)
        fuzz_tool_test.go:18: str:"𐑄",first:"\x84\x91\x90\xf0",second:"𐑄"
        fuzz_tool_test.go:23: Reverse produced invalid UTF-8 string "𐑄" "\x84\x91\x90\xf0"

    Failing input written to testdata\fuzz\FuzzReverse\d856c981b6266ba2
    To re-run:
    go test -run=FuzzReverse/d856c981b6266ba2
=== NAME
FAIL
exit status 1
FAIL    golearn/tool_test       0.697s

TIP

Failed test cases in fuzz testing will be output to a corpus file under the testdata directory of the current test folder. For example, in the above example:

Failing input written to testdata\fuzz\FuzzReverse\d856c981b6266ba2
To re-run:
go test -run=FuzzReverse/d856c981b6266ba2

testdata\fuzz\FuzzReverse\d856c981b6266ba2 is the output corpus file path. The file content is as follows:

go test fuzz v1
string("𐑄")

You can see that this time it did not pass. The reason is that the string became non-UTF8 format after reversal. So through fuzz testing, this problem was discovered. Since some characters occupy more than one byte, if they are reversed in bytes, it will definitely be garbled. Therefore, modify the source code to be tested as follows. Convert the string to []rune, so that the above problem can be avoided:

func Reverse(s string) string {
    r := []rune(s)
    for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
        r[i], r[j] = r[j], r[i]
    }
    return string(r)
}

Next, directly run the test case that failed in the last fuzz test:

$ go test -run=FuzzReverse/d856c981b6266ba2 -v
=== RUN   FuzzReverse
=== RUN   FuzzReverse/d856c981b6266ba2
    fuzz_tool_test.go:18: str:"𐑄",first:"𐑄",second:"𐑄"
--- PASS: FuzzReverse (0.00s)
    --- PASS: FuzzReverse/d856c981b6266ba2 (0.00s)
PASS
ok      golearn/tool_test       0.033s

You can see that this time the test passed. Execute fuzz testing again to see if there are any more problems:

$ go test -fuzz . -fuzztime 30s -run Fuzz -v
=== RUN   FuzzReverse
fuzz: elapsed: 0s, gathering baseline coverage: 0/219 completed
fuzz: minimizing 70-byte failing input file
failure while testing seed corpus entry: FuzzReverse/d97214ce235bfcf5
fuzz: elapsed: 0s, gathering baseline coverage: 2/219 completed
--- FAIL: FuzzReverse (0.15s)
    --- FAIL: FuzzReverse (0.00s)
        fuzz_tool_test.go:18: str:"\xe4",first:"",second:""
        fuzz_tool_test.go:20: before: "\xe4", after: ""

=== NAME
FAIL
exit status 1
FAIL    golearn/tool_test       0.184s

You can find that there is an error again. This time the problem is that the string is not equal after two reversals. The original character is \xe4, the expected result is 4ex\, but the result is garbled, as follows:

func main() {
  fmt.Println("\xe4")
  fmt.Println([]byte("\xe4"))
  fmt.Println([]rune("\xe4"))
  fmt.Printf("%q\n", "\xe4")
  fmt.Printf("%x\n", "\xe4")
}

Its execution result is:

[65533]
"\xe4"
e4

The reason is that \xe4 represents one byte, but it is not a valid UTF-8 sequence (in UTF-8 encoding, \xe4 is the start of a three-byte character, but the following two bytes are missing). When converting to []rune, Golang automatically changes it to []rune{"\uFFFD"} containing a single Unicode character. After reversal, it is still []rune{"\uFFFD"}. When converting back to string, the Unicode character is replaced with its UTF-8 encoding \xef\xbf\xbd. Therefore, one solution is that if the input is a non-UTF8 string, directly return an error:

func Reverse(s string) (string, error) {
    if !utf8.ValidString(s) {
        return s, errors.New("input is not valid UTF-8")
    }
    r := []rune(s)
    for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
        r[i], r[j] = r[j], r[i]
    }
    return string(r), nil
}

The test code also needs to be slightly modified:

func FuzzReverse(f *testing.F) {
  testdata := []string{"hello world!", "nice to meet you", "good bye!"}
  for _, data := range testdata {
    f.Add(data)
  }

  f.Fuzz(func(t *testing.T, str string) {
    first, err := tool.Reverse(str)
    if err != nil {
      t.Skip()
    }
    second, err := tool.Reverse(first)
    if err != nil {
      t.Skip()
    }
    t.Logf("str:%q,first:%q,second:%q", str, first, second)
    if str != second {
      t.Errorf("before: %q, after: %q", str, second)
    }
    if utf8.ValidString(str) && !utf8.ValidString(first) {
      t.Errorf("Reverse produced invalid UTF-8 string %q %q", str, first)
    }
  })
}

When the reverse function returns error, skip the test, and then perform fuzz testing:

$ go test -fuzz . -fuzztime 30s -run Fuzz -v
=== RUN   FuzzReverse
fuzz: elapsed: 0s, gathering baseline coverage: 0/219 completed
fuzz: elapsed: 0s, gathering baseline coverage: 219/219 completed, now fuzzing with 16 workers
fuzz: elapsed: 3s, execs: 895571 (297796/sec), new interesting: 32 (total: 251)
fuzz: elapsed: 6s, execs: 1985543 (363120/sec), new interesting: 37 (total: 256)
fuzz: elapsed: 9s, execs: 3087837 (367225/sec), new interesting: 38 (total: 257)
fuzz: elapsed: 12s, execs: 4090817 (335167/sec), new interesting: 40 (total: 259)
fuzz: elapsed: 15s, execs: 5132580 (346408/sec), new interesting: 44 (total: 263)
fuzz: elapsed: 18s, execs: 6248486 (372185/sec), new interesting: 45 (total: 264)
fuzz: elapsed: 21s, execs: 7366827 (373305/sec), new interesting: 46 (total: 265)
fuzz: elapsed: 24s, execs: 8439803 (358059/sec), new interesting: 47 (total: 266)
fuzz: elapsed: 27s, execs: 9527671 (361408/sec), new interesting: 47 (total: 266)
fuzz: elapsed: 30s, execs: 10569473 (348056/sec), new interesting: 48 (total: 267)
fuzz: elapsed: 30s, execs: 10569473 (0/sec), new interesting: 48 (total: 267)
--- PASS: FuzzReverse (30.16s)
=== NAME
PASS
ok      golearn/tool_test       30.789s

Then this time you can get a more complete fuzz test output log. Some concept explanations are as follows:

• elapsed: Time elapsed after a round is completed
• execs: Total number of inputs run, 297796/sec indicates how many inputs per second
• new interesting: In testing, the total number of "interesting" inputs added to the corpus. (Interesting inputs refer to inputs that can expand code coverage beyond the scope that the existing corpus can cover. As coverage continues to expand, its growth trend will generally continue to slow down)

TIP

If there is no -fuzztime parameter to limit time, fuzz testing will run forever.

Type Support

The types supported in Go Fuzz are as follows:

• string, []byte
• int, int8, int16, int32/rune, int64
• uint, uint8/byte, uint16, uint32, uint64
• float32, float64
• bool