Go Language encode Encoding and Decoding

In the current internet era, the most commonly used language-independent data formats are xml, Yaml, json, and protobuf. Go also supports related operations for these data formats. The following is a comparison table.

Name	XML	YAML	JSON	Protocol Buffers
Data Structure	Complex	Simpler	Simple	Complex
Storage Method	Text	Text	Text	Binary
Storage Size	Large	Medium	Medium	Small
Parsing Efficiency	Slow	Medium	Medium	Fast
Language Support	Very Many	Many	Many	Relatively Many
Development Difficulty	Cumbersome	Simpler	Simple	Simple
Learning Cost	Low	Low	Low	Low
Applicable Scope	Data Exchange	Configuration Files	Data Exchange	Data Exchange

TIP

In Go, if you want to serialize and deserialize a struct, the fields must be exported, i.e., start with an uppercase letter.

Additionally, TOML is gradually becoming popular. Syntactically, it's like an improvement of .ini. If interested, you can learn more at TOML: Tom's Obvious, Minimal Language.

XML

xml, also known as eXtensible Markup Language, is a format for storing data. It originated in the 1960s and is the oldest of the above data formats. It has a wide range of uses, including network transmission, data exchange, configuration files, data storage, etc. However, with the passage of time, it is gradually being replaced by new markup languages.

First, define a struct:

type Person struct {
   UserId   string `xml:"id"`
   Username string `xml:"name"`
   Age      int    `xml:"age"`
   Address  string `xml:"address"`
}

func Marshal(v any) ([]byte, error) // XML serialization

func MarshalIndent(v any, prefix, indent string) ([]byte, error) // Formatting

func Unmarshal(data []byte, v any) error // Deserialization

Serialization

func main() {
   person := Person{
      UserId:   "120",
      Username: "jack",
      Age:      18,
      Address:  "usa",
   }

   bytes, err := xml.MarshalIndent(person, "", "\t")
   if err != nil {
      fmt.Println(err)
      return
   }
   fmt.Println(string(bytes))
}

Output:

<Person>
        <id>120</id>
        <name>jack</name>
        <age>18</age>
        <address>usa</address>
</Person>

Deserialization

func main() {
   var person = Person{
      UserId:   "",
      Username: "",
      Age:      0,
      Address:  "",
   }

   xmlStr := "<Person>                      \n        <id>120</id>          \n        <name>jack</name>     \n        <age>18</age>         \n        <address>usa</address>\n</Person>  "

   err := xml.Unmarshal([]byte(xmlStr), &person)
   if err != nil {
      fmt.Println(err)
      return
   }
}

Output:

{UserId:120 Username:jack Age:18 Address:usa}

However, traditional XML parsing often requires creating structs, which can be very cumbersome. Nowadays, most parsing involves simple XML structures. If complex structures are used, it can be very troublesome. Therefore, most people use a third-party open-source library etree to parse XML. If interested, you can learn more yourself: Go relatively easy-to-use XML parsing plugin etree - Juejin.

YML

YAML's syntax is similar to other high-level languages, and it can simply express lists, hash tables, scalars, and other data forms. It uses whitespace indentation and relies heavily on appearance features, making it particularly suitable for expressing or editing data structures and various configuration files. YML also exists in many projects in the form of configuration files, with a more concise content structure that is clear at a glance. Go official does not provide support for YML, so we need to use a third-party package.

go get github.com/go-yaml/yaml

Main Methods

func Marshal(in interface{}) (out []byte, err error) // Serialization

func Unmarshal(in []byte, out interface{}) (err error) // Deserialization

First, prepare a struct:

type Config struct {
   Database string `yaml:"database"`
   Url      string `yaml:"url"`
   Port     int    `yaml:"port"`
   Username string `yaml:"username"`
   Password string `yaml:"password"`
}

Configuration file:

database: mysql
url: 127.0.0.1
port: 3306
username: root
password: 123456

Serialization

func main() {
   config := Config{
      Database: "oracle",
      Url:      "localhost",
      Port:     3326,
      Username: "root",
      Password: "123456",
   }

   out, err := yaml.Marshal(config)
   if err != nil {
      fmt.Println(err)
      return
   }
   fmt.Println(string(out))
}

Output:

database: oracle
url: localhost
port: 3326
username: root
password: "123456"

However, since yml itself has strict indentation syntax, there is no issue with serialization formatting.

Deserialization

func main() {
   bytes, err := os.ReadFile("./src/config.yml")
   if err != nil {
      fmt.Println(err)
      return
   }
   var config Config
   err = yaml.Unmarshal(bytes, &config)
   if err != nil {
      fmt.Println(err)
      return
   }

   fmt.Println(config)
}

Output:

{mysql 127.0.0.1 3306 root 123456}

JSON

json is commonly used in Restful style interface communication. Compared to xml, it has a lighter size and lower learning cost, making it the mainstream data exchange format in the web field.

In Go, the encoding/json package provides corresponding functions for JSON serialization and deserialization. The main functions used are as follows:

func Marshal(v any) ([]byte, error) // Serialize Go object to JSON string

func Unmarshal(data []byte, v any) error // Deserialize JSON string to Go object

First, define a struct:

type Person struct {
   UserId   string
   Username string
   Age      int
   Address  string
}

Serialization

func main() {
  person := Person{
    UserId:   "120",
    Username: "jack",
    Age:      18,
    Address:  "usa",
  }

  bytes, err := json.Marshal(person)
  if err != nil {
    fmt.Println(err)
    return
  }
  fmt.Println(string(bytes))
}

Result:

{ "UserId": "120", "Username": "jack", "Age": 18, "Address": "usa" }

Field Renaming

We can achieve renaming effects through struct tags.

type Person struct {
   UserId   string `json:"id"`
   Username string `json:"name"`
   Age      int    `json:"age"`
   Address  string `json:"address"`
}

At this point, the output is:

{ "id": "1202", "name": "jack", "age": 19, "address": "USA" }

Indentation

By default, serialization has no indentation. This is to reduce space consumption during transmission, but it is not conducive to human observation. In some cases, we need to serialize it into a form that humans can observe. For this, we just need to use a different function:

func MarshalIndent(v any, prefix, indent string) ([]byte, error)

func main() {
   person := Person{
      UserId:   "1202",
      Username: "jack",
      Age:      19,
      Address:  "USA",
   }
   bytes, err := json.MarshalIndent(person, "", "\t")
   if err != nil {
      fmt.Println(err)
      return
   }
   fmt.Println(string(bytes))
}

Output:

{
  "id": "1202",
  "name": "jack",
  "age": 19,
  "address": "USA"
}

Deserialization

When deserializing, note that if the struct has JSON tags, the field names take precedence based on the JSON tags; otherwise, the struct property names are used.

func main() {
  person := Person{}

  jsonStr := "{\"id\":\"120\",\"name\":\"jack\",\"age\":18,\"address\":\"usa\"}\n"

  err := json.Unmarshal([]byte(jsonStr), &person)
  if err != nil {
    fmt.Println(err)
    return
  }
  fmt.Printf("%+v", person)
}

Output:

{UserId:120 Username:jack Age:18 Address:usa}

Protocol Buffers

Protocol Buffers is a language-neutral, protocol-neutral, extensible structured data serialization mechanism open-sourced by Google in 2008. Compared to the above three, it is lighter and faster when unpacking and packing, mostly used in RPC communication. For explanations about Protobuf, please refer to Protobuf.

Install dependencies:

go get github.com/golang/protobuf/proto

person.proto file:

syntax = "proto3";

option go_package = "./;person";

package proto;

enum Gender{
  MAIL = 0;
  FE_MAIL = 1;
}

message person {
  string name = 1;
  int32 age = 2;
  Gender gender = 3;
}

After generating the file:

package main

import (
   p "GoProject/src/proto"
   "fmt"
   "github.com/golang/protobuf/proto"
)

func main() {
  person := p.Person{
    Name:   "wyh",
    Age:    12,
    Gender: p.Gender_FE_MAIL,
  }

  data, err := proto.Marshal(&person) // Serialization
  if err != nil {
    fmt.Println(err)
    return
  }
  temp := &p.Person{}
  fmt.Println("proto buffer len: ", len(data), "bytes:", data)
  err = proto.Unmarshal(data, temp) // Deserialization
  if err != nil {
    fmt.Println(err)
    return
  }
  fmt.Println(temp)
}

Output:

proto buffer len:  9 bytes: [10 3 119 121 104 16 12 24 1]
name:"wyh"  age:12  gender:FE_MAIL

However, we usually don't serialize manually. The protoc compiler can generate source code in the corresponding language based on our defined proto files.