File

Go language provides several standard libraries for file handling:

• os library, responsible for specific implementations of OS file system interaction
• io library, an abstraction layer for reading and writing IO
• fs library, an abstraction layer for file systems

This article explains how to perform basic file handling in Go language.

Opening

Two common ways to open files are using two functions provided by the os package. The Open function returns a file pointer and an error:

func Open(name string) (*File, error)

The latter OpenFile can provide more fine-grained control. The Open function is just a simple wrapper around the OpenFile function.

func OpenFile(name string, flag int, perm FileMode) (*File, error)

Let's introduce the first usage method first. Just provide the corresponding file name directly. The code is as follows:

func main() {
   file, err := os.Open("README.txt")
   fmt.Println(file, err)
}

The file lookup path defaults to the path where the project's go.mod file is located. Since there is no file README.txt under the project, it will naturally return an error.

<nil> open README.txt: The system cannot find the file specified.

Because there are many types of IO errors, you need to manually check whether the file exists. Similarly, the os package also provides convenient functions for this. The modified code is as follows:

func main() {
  file, err := os.Open("README.txt")
  if os.IsNotExist(err) {
    fmt.Println("File does not exist")
  } else if err != nil {
    fmt.Println("File access exception")
  } else {
    fmt.Println("File read successfully", file)
  }
}

Run again, output as follows:

File does not exist

In fact, the file read by the first function is read-only and cannot be modified:

func Open(name string) (*File, error) {
  return OpenFile(name, O_RDONLY, 0)
}

Through the OpenFile function, you can control more details, such as modifying file descriptors and file permissions. Regarding file descriptors, the os package provides the following constants for use:

const (
   // Must specify one of: read-only, write-only, read-write
   O_RDONLY int = syscall.O_RDONLY // Open file in read-only mode
   O_WRONLY int = syscall.O_WRONLY // Open file in write-only mode
   O_RDWR   int = syscall.O_RDWR   // Open file in read-write mode
   // Remaining values are used to control behavior
   O_APPEND int = syscall.O_APPEND // When writing to file, append data to end of file
   O_CREATE int = syscall.O_CREAT  // Create file if it doesn't exist
   O_EXCL   int = syscall.O_EXCL   // Used with O_CREATE, file must not exist
   O_SYNC   int = syscall.O_SYNC   // Open file with synchronous IO
   O_TRUNC  int = syscall.O_TRUNC  // Truncate writable file when opening
)

Regarding file permissions, the following constants are provided:

const (
   ModeDir        = fs.ModeDir        // d: directory
   ModeAppend     = fs.ModeAppend     // a: append only
   ModeExclusive  = fs.ModeExclusive  // l: exclusive
   ModeTemporary  = fs.ModeTemporary  // T: temporary file
   ModeSymlink    = fs.ModeSymlink    // L: symbolic link
   ModeDevice     = fs.ModeDevice     // D: device file
   ModeNamedPipe  = fs.ModeNamedPipe  // p: named pipe (FIFO)
   ModeSocket     = fs.ModeSocket     // S: Unix domain socket
   ModeSetuid     = fs.ModeSetuid     // u: setuid
   ModeSetgid     = fs.ModeSetgid     // g: setgid
   ModeCharDevice = fs.ModeCharDevice // c: Unix character device, requires ModeDevice to be set
   ModeSticky     = fs.ModeSticky     // t: sticky bit
   ModeIrregular  = fs.ModeIrregular  // ?: irregular file

   // Mask for type bits. For regular files, nothing is set.
   ModeType = fs.ModeType

   ModePerm = fs.ModePerm // Unix permission bits, 0o777
)

Below is a code example of opening a file in read-write mode with permission 0666, meaning everyone can read and write to the file, and it will be created automatically if it doesn't exist:

func main() {
  file, err := os.OpenFile("README.txt", os.O_RDWR|os.O_CREATE, 0666)
  if os.IsNotExist(err) {
    fmt.Println("File does not exist")
  } else if err != nil {
    fmt.Println("File access exception")
  } else {
    fmt.Println("File opened successfully", file.Name())
    file.Close()
  }
}

Output as follows:

File opened successfully README.txt

If you just want to get some information about the file without reading it, you can use the os.Stat() function. Code example as follows:

func main() {
  fileInfo, err := os.Stat("README.txt")
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println(fmt.Sprintf("%+v", fileInfo))
  }
}

Output as follows:

&{name:README.txt FileAttributes:32 CreationTime:{LowDateTime:3603459389 HighDateTime:31016791} LastAccessTime:{LowDateTime:3603459389 HighDateTime:31016791} LastWriteTime:{LowDateTime:3603459389 HighDateTime:31016791} FileSizeHigh
:0 FileSizeLow:0 Reserved0:0 filetype:0 Mutex:{state:0 sema:0} path:README.txt vol:0 idxhi:0 idxlo:0 appendNameToPath:false}

WARNING

After opening a file, always remember to close it. Usually, the close operation is placed in a defer statement:

defer file.Close()

Reading

When a file is successfully opened, you can perform read operations. For file reading operations, the *os.File type provides the following public methods:

// Read file into the passed byte slice
func (f *File) Read(b []byte) (n int, err error)

// Compared to the first one, can read from specified offset
func (f *File) ReadAt(b []byte, off int64) (n int, err error)

In most cases, the first one is used more often. For the first method, you need to write your own logic for dynamic expansion of the slice during reading. The code is as follows:

func ReadFile(file *os.File) ([]byte, error) {
  buffer := make([]byte, 0, 512)
  for {
    // When capacity is insufficient
    if len(buffer) == cap(buffer) {
      // Expand capacity
      buffer = append(buffer, 0)[:len(buffer)]
    }
    // Continue reading file
    offset, err := file.Read(buffer[len(buffer):cap(buffer)])
    // Add written data to slice
    buffer = buffer[:len(buffer)+offset]
    // When error occurs
    if err != nil {
      if errors.Is(err, io.EOF) {
        err = nil
      }
      return buffer, err
    }
  }
}

The remaining logic is as follows:

func main() {
   file, err := os.OpenFile("README.txt", os.O_RDWR|os.O_CREATE, 0666)
   if err != nil {
      fmt.Println("File access exception")
   } else {
      fmt.Println("File opened successfully", file.Name())
      bytes, err := ReadFile(file)
      if err != nil {
         fmt.Println("File read exception", err)
      } else {
         fmt.Println(string(bytes))
      }
      file.Close()
   }
}

Output is:

File opened successfully README.txt
hello world!

Besides, you can also use two convenience functions for file reading: the ReadFile function in the os package, and the ReadAll function in the io package. For os.ReadFile, you only need to provide the file path. For io.ReadAll, you need to provide an implementation of io.Reader type.

os.ReadFile

func ReadFile(name string) ([]byte, error)

Usage example as follows:

func main() {
  bytes, err := os.ReadFile("README.txt")
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println(string(bytes))
  }
}

Output as follows:

hello world!

io.ReadAll

func ReadAll(r Reader) ([]byte, error)

Usage example as follows:

func main() {

   file, err := os.OpenFile("README.txt", os.O_RDWR|os.O_CREATE, 0666)
   if err != nil {
      fmt.Println("File access exception")
   } else {
      fmt.Println("File opened successfully", file.Name())
      bytes, err := io.ReadAll(file)
      if err != nil {
         fmt.Println(err)
      } else {
         fmt.Println(string(bytes))
      }
      file.Close()
   }
}

File opened successfully README.txt
hello world!

Writing

The os.File struct provides the following methods for writing data:

// Write byte slice
func (f *File) Write(b []byte) (n int, err error)

// Write string
func (f *File) WriteString(s string) (n int, err error)

// Write from specified position, returns error when opened with os.O_APPEND mode
func (f *File) WriteAt(b []byte, off int64) (n int, err error)

If you want to write data to a file, you must open it in O_WRONLY or O_RDWR mode, otherwise you cannot successfully write to the file. Below is an example of opening a file in os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC mode with permission 0666 to write data:

func main() {
  file, err := os.OpenFile("README.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC, 0666)
  if err != nil {
    fmt.Println("File access exception")
  } else {
    fmt.Println("File opened successfully", file.Name())
    for i := 0; i < 5; i++ {
      offset, err := file.WriteString("hello world!\n")
      if err != nil {
        fmt.Println(offset, err)
      }
    }
    fmt.Println(file.Close())
  }
}

Since the file is opened in os.O_APPEND mode, data will be appended to the end of the file when writing. After execution, the file content is as follows:

hello world!
hello world!
hello world!
hello world!
hello world!

Writing byte slices to a file is a similar operation, so I won't repeat it. For file writing operations, the standard library also provides convenience functions: os.WriteFile and io.WriteString.

os.WriteFile

func WriteFile(name string, data []byte, perm FileMode) error

Usage example as follows:

func main() {
  err := os.WriteFile("README.txt", []byte("hello world!\n"), 0666)
  if err != nil {
    fmt.Println(err)
  }
}

At this point, the file content is as follows:

hello world!

io.WriteString

func WriteString(w Writer, s string) (n int, err error)

Usage example as follows:

func main() {
   file, err := os.OpenFile("README.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC, 0666)
   if err != nil {
      fmt.Println("File access exception")
   } else {
      fmt.Println("File opened successfully", file.Name())
      for i := 0; i < 5; i++ {
         offset, err := io.WriteString(file, "hello world!\n")
         if err != nil {
            fmt.Println(offset, err)
         }
      }
      fmt.Println(file.Close())
   }
}

hello world!
hello world!
hello world!
hello world!
hello world!

The os.Create function is used to create files, which is essentially a wrapper around OpenFile:

func Create(name string) (*File, error) {
   return OpenFile(name, O_RDWR|O_CREATE|O_TRUNC, 0666)
}

WARNING

When creating a file, if its parent directory doesn't exist, creation will fail and return an error.

Copying

For copying files, you need to open two files at the same time. The first method is to read data from the original file and then write it to the target file. Code example as follows:

func main() {
    // Read data from original file
  data, err := os.ReadFile("README.txt")
  if err != nil {
    fmt.Println(err)
    return
  }
    // Write to target file
  err = os.WriteFile("README(1).txt", data, 0666)
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println("Copy successful")
  }
}

*os.File.ReadFrom

Another method is to use the ReadFrom method provided by os.File. When opening files, one is read-only, one is write-only.

func (f *File) ReadFrom(r io.Reader) (n int64, err error)

Usage example as follows:

func main() {
  // Open original file in read-only mode
  origin, err := os.OpenFile("README.txt", os.O_RDONLY, 0666)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer origin.Close()
  // Open copy file in write-only mode
  target, err := os.OpenFile("README(1).txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer target.Close()
  // Read data from original file, then write to copy file
  offset, err := target.ReadFrom(origin)
  if err != nil {
    fmt.Println(err)
    return
  }
  fmt.Println("File copy successful", offset)
}

This copy method requires reading all the content of the source file into memory first, then writing to the target file. This is not recommended for very large files.

io.Copy

Another method is to use the io.Copy function, which reads and writes at the same time, first reading content into a buffer, then writing to the target file. The default buffer size is 32KB.

func Copy(dst Writer, src Reader) (written int64, err error)

Usage example as follows:

func main() {
  // Open original file in read-only mode
  origin, err := os.OpenFile("README.txt", os.O_RDONLY, 0666)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer origin.Close()
  // Open copy file in write-only mode
  target, err := os.OpenFile("README(1).txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
  if err != nil {
    fmt.Println(err)
    return
  }
  defer target.Close()
  // Copy
  written, err := io.Copy(target, origin)
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println(written)
  }
}

You can also use io.CopyBuffer to specify the buffer size.

Renaming

Renaming can also be understood as moving files. You'll use the Rename function from the os package.

func Rename(oldpath, newpath string) error

Example as follows:

func main() {
  err := os.Rename("README.txt", "readme.txt")
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println("Rename successful")
  }
}

This function has the same effect for folders.

Deleting

Delete operations are much simpler compared to other operations. You'll only use two functions from the os package:

// Delete a single file or empty directory, returns error when directory is not empty
func Remove(name string) error

// Delete all files and directories in the specified directory including subdirectories and subfiles
func RemoveAll(path string) error

Very simple to use. Below is an example of deleting a directory:

func main() {
  // Delete all files and subdirectories in current directory
  err := os.RemoveAll(".")
  if err != nil {
    fmt.Println(err)
  }else {
    fmt.Println("Delete successful")
  }
}

Below is an example of deleting a single file:

func main() {
  // Delete all files and subdirectories in current directory
  err := os.Remove("README.txt")
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println("Delete successful")
  }
}

Flushing

The os.Sync function wraps the underlying system call Fsync, used to flush cached IO writes to disk:

func main() {
  create, err := os.Create("test.txt")
  if err != nil {
    panic(err)
  }
  defer create.Close()

  _, err = create.Write([]byte("hello"))
  if err != nil {
    panic(err)
  }

    // Flush to disk
  if err := create.Sync();err != nil {
    return
  }
}

Directories

Many directory operations are similar to file operations.

Reading

For directories, there are two ways to open them:

os.ReadDir

The first way is to use the os.ReadDir function:

func ReadDir(name string) ([]DirEntry, error)

func main() {
   // Current directory
   dir, err := os.ReadDir(".")
   if err != nil {
      fmt.Println(err)
   } else {
      for _, entry := range dir {
         fmt.Println(entry.Name())
      }
   }
}

*os.File.ReadDir

The second way is to use the *os.File.ReadDir function. os.ReadDir is essentially just a simple wrapper around *os.File.ReadDir.

// When n < 0, read all contents under the directory
func (f *File) ReadDir(n int) ([]DirEntry, error)

func main() {
   // Current directory
   dir, err := os.Open(".")
   if err != nil {
      fmt.Println(err)
   }
   defer dir.Close()
   dirs, err := dir.ReadDir(-1)
   if err != nil {
      fmt.Println(err)
   } else {
      for _, entry := range dirs {
         fmt.Println(entry.Name())
      }
   }
}

Creating

Directory creation operations use two functions from the os package:

// Create a directory with specified name and specified permissions
func Mkdir(name string, perm FileMode) error

// Compared to the former, this function creates all necessary parent directories
func MkdirAll(path string, perm FileMode) error

Example as follows:

func main() {
  err := os.Mkdir("src", 0666)
  if err != nil {
    fmt.Println(err)
  } else {
    fmt.Println("Create successful")
  }
}

Copying

We can write our own function to recursively traverse the entire directory. However, the filepath standard library already provides similar functionality, so we can use it directly. A simple directory copy code example is as follows:

func CopyDir(src, dst string) error {
    // Check source directory status
  _, err := os.Stat(src)
  if err != nil {
    return err
  }

  return filepath.Walk(src, func(path string, info fs.FileInfo, err error) error {
    if err != nil {
      return err
    }

        // Calculate relative path
    rel, err := filepath.Rel(src, path)
    if err != nil {
      return err
    }

        // Build target path
    destpath := filepath.Join(dst, rel)

        // Create directory
    var dirpath string
    var mode os.FileMode = 0755
    if info.IsDir() {
      dirpath = destpath
      mode = info.Mode()
    } else if info.Mode().IsRegular() {
      dirpath = filepath.Dir(destpath)
    }

    if err := os.MkdirAll(dirpath, mode); err != nil {
      return err
    }

        // Create file
    if info.Mode().IsRegular() {
      srcfile, err := os.Open(path)
      if err != nil {
        return err
      }
            // Always remember to close the file
      defer srcfile.Close()
      destfile, err := os.OpenFile(destpath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, info.Mode())
      if err != nil {
        return err
      }
      defer destfile.Close()

            // Copy file content
      if _, err := io.Copy(destfile, srcfile); err != nil {
        return err
      }
      return nil
    }

    return nil
  })
}

filepath.Walk will recursively traverse the entire directory. During the process, when encountering a directory, it creates a directory; when encountering a file, it creates a new file and copies it. The code is a bit more than copying files but not complicated.