GitHub Action: Why is appending checksum after each step to checksums.txt overwriting the file each time?

Understanding Shell Redirection Operators

In GitHub Actions, each job step runs within a shell environment, typically bash on Linux and macOS runners or pwsh (PowerShell) on Windows runners. The way these shells handle output streams determines whether new data is added to a file or replaces it entirely. When you run a command like echo "data" > file.txt, the shell opens the file for writing. If the file already exists, its existing content is truncated (deleted) immediately, and then the new data is written. This behavior is standard across almost all Unix-like systems and is the root cause of the missing checksums.

To fix this, you need to switch from write mode to append mode. By using two greater-than signs (>) instead of one, you instruct the shell to open the file and move the cursor to the very end of the current content. Any subsequent data written to the file will be placed after the last byte of existing text. This preserves the history of all checksums generated by previous steps in the same job.

Implementing the Fix in Bash Workflows

Most GitHub Actions workflows use bash as the default shell on Ubuntu runners. If your workflow is currently failing to accumulate checksums, check your YAML configuration for lines that look like this:

steps:
  - name: Generate Checksum
    run: |
      sha256sum ./my_file.bin > checksums.txt
      
  - name: Append More Checksums
    run: |
      sha256sum ./another_file.bin > checksums.txt

In the example above, the second step will delete the entry for my_file.bin because it uses the single > operator. To correct this, update the second step to use the append operator:

steps:
  - name: Generate Checksum
    run: |
      sha256sum ./my_file.bin > checksums.txt
      
  - name: Append More Checksums
    run: |
      sha256sum ./another_file.bin >> checksums.txt

This change ensures that checksums.txt contains both entries after the job completes. You can verify this by adding a debug step at the end of your job that prints the file contents using cat checksums.txt.

Handling PowerShell Environments

If you are running your GitHub Action on a Windows runner, the default shell might be PowerShell. PowerShell handles redirection slightly differently but follows the same logical principle. In PowerShell, the > operator (or its alias Out-File) also overwrites files by default. To append, you should use the -Append parameter with Out-File or use the >> operator if supported in your specific PowerShell version context.

steps:
  - name: Generate Checksum
    shell: pwsh
    run: |
      Get-FileHash .\my_file.bin | Out-File -FilePath checksums.txt
      
  - name: Append More Checksums
    shell: pwsh
    run: |
      Get-FileHash .\another_file.bin | Out-File -FilePath checksums.txt -Append

Notice the explicit -Append flag in the second command. Without it, the second step would wipe out the first hash. Using >> in PowerShell can sometimes lead to encoding issues or unexpected behavior depending on the version, so using Out-File -Append is generally the more robust and explicit approach for Windows-based workflows.

Best Practices for Checksum Management

Beyond just fixing the redirection operator, there are several best practices to ensure your checksum files remain reliable and useful. First, always specify the full path to the file if you are not sure which directory the action is running in. Relative paths can behave unpredictably if a step changes the working directory using the working-directory property.

Second, consider using the GITHUB_WORKSPACE environment variable to ensure you are writing to the correct location. This variable points to the root of your repository clone. For example:

run: echo "$hash" >> $GITHUB_WORKSPACE/checksums.txt

This prevents accidental writes to temporary directories or other locations that might be discarded when the job ends. Additionally, if you are generating many checksums, consider sorting them alphabetically after generation to make manual inspection easier. You can do this by piping the file through sort after all steps have completed, though this requires careful handling to avoid modifying the file while it is being read.

Finally, remember that artifacts uploaded via actions/upload-artifact are tied to the job. If you upload checksums.txt after each step, you will only get the latest version. Instead, upload the artifact once at the end of the job to capture the complete accumulated list. This aligns with how CI/CD pipelines typically function, where final outputs are collected after all processing steps are finished.

Frequently Asked Questions

Why does my checksum file disappear between jobs?

Each job in a GitHub Actions workflow runs on a fresh virtual machine environment. Files created in one job do not persist to the next unless you explicitly save them as an artifact. To share a checksum file between jobs, you must upload it in the first job using actions/upload-artifact and download it in the subsequent job using actions/download-artifact.

Can I use tee to append to a file?

Yes, the tee command can be used to both display output to the console and save it to a file. However, tee overwrites the file by default. To append, you must use the -a flag, like this: echo "data" | tee -a checksums.txt. While functional, using >> is simpler and more common for simple redirection tasks.

What happens if the checksums.txt file does not exist yet?

Both > and >> operators will create the file if it does not already exist. The difference lies only in how they handle existing content. If the file is new, both will start writing from the beginning. Therefore, switching to >> is safe even for the very first step in your workflow.

How do I verify the content of checksums.txt during a run?

You can add a step that runs cat checksums.txt (on Linux/macOS) or type checksums.txt (on Windows PowerShell) after the appending steps. This will print the current contents of the file to the workflow logs, allowing you to visually confirm that all expected hashes are present before uploading the artifact.

Are there any security risks with appending to files in CI/CD?

Appending itself is not a security risk, but ensuring that the file path is controlled is important. Always use absolute paths or variables like $GITHUB_WORKSPACE to prevent path traversal attacks or accidental writes to sensitive system directories. Never rely solely on relative paths if your workflow involves changing directories frequently.