Writing a reliable data store in GoLang

I have been building a simple data store as part of my module for a document database system which I’m going to build for educational purposes.

In order to store data reliably, I have to abide the ACID property. Shown below is my save method.

  • Generic version control strategy for select table data within a heavily normalized database
  • Object wiht multi-level nest relationship versioning
  • The best way to manage database changes
  • Where are all the native revisioned databases?
  • func (document Document) Save() (hash string, err error) {
        if err := os.MkdirAll(document.FileDirectory(), 0600); err != nil {
            return "", err
        }
    
        file, err := os.Create(document.TmpFile())
        if err != nil {
            return "", err
        }
    
        file.Write(document.Data)
        if err := file.Sync(); err != nil {
            return "", err
        }
    
        file.Close()
    
        if err := os.Rename(document.TmpFile(), document.File()); err != nil {
            return "", err
        }
    
        return document.Hash(), nil
    }
    

    First the data (in []byte) is saved to a temporary file. The file is then synced with file.Sync() to ensure the data is written to the persistent storage. Then the temporary file is renamed into the new file.

    Note:
    The way i chose to store the data file is in spoolDir format. Meaning the first two character of the hash generated from the data is used as parent directory name. The following two character of the hash is used as the subsequent directory name. The filename will be the 36 character left over. The temporary file only has a suffix .tmp with the file path and file name the same. This design is inspired by how git store data.

    Question:
    Is the way I implement the data storing algorithm sufficient to ensure the data is reliably persisted.

    Answer so far:
    Something about directory syncing to ensure data durability (I’m not sure)

    Thanks in advance


    Updated code as per suggested by rightfold:

    func (document Document) Save() (hash string, err error) {
        if err := os.MkdirAll(document.FileDirectory(), 0600); err != nil {
            return "", err
        }
    
        file, err := os.Create(document.TmpFile())
        if err != nil {
            return "", err
        }
    
        file.Write(document.Data)
        if err := file.Sync(); err != nil {
            return "", err
        }
    
        file.Close()
    
        if err := os.Rename(document.TmpFile(), document.File()); err != nil {
            os.Remove(document.TmpFile())
            return "", err
        }
    
        return document.Hash(), nil
    }
    

  • Roll back my github fork to an earlier commit
  • Local Client server Git setup for Xcode projects
  • Diff between two commits with two different parent commits
  • Moving from CVS to git: $Id:$ equivalent?
  • How do I git-add with wildcards when ignored files are present?
  • Is there a way to get the number of additional commits in git without awk?
  • One Solution collect form web for “Writing a reliable data store in GoLang”

    What you are doing guarantees durability to the degree the OS and hardware guarantee it (which is the best you can get).

    It is also atomic; incomplete writes don’t leave incomplete data, even when the CPU catches fire.

    You may want to delete the temporary file when renaming fails:

    if err := os.Rename(document.TmpFile(), document.File()); err != nil {
        os.Remove(document.TmpFile()) // ignore errors
        return "", err
    }
    
    Git Baby is a git and github fan, let's start git clone.