- Go 99.6%
- Makefile 0.4%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Thorsten Klein
0e43a729d7
All checks were successful
test / test (push) Successful in 1m55s
## Problem The `directory` workspace provider wrote files via `os.OpenFile(..., O_TRUNC)` + `io.Copy` (`pkg/client/directory.go`). The target was truncated to zero **before** the new content streamed in, so a process killed mid-write left a truncated/partial file. For knowledge data sources this corrupted `.metadata.json`, and the corrupt file then aborted the next sync. ## Change `directoryProvider.writeFile` now writes atomically: 1. Stream into a temp file in the **same directory** as the target (same filesystem) via `safeopen.OpenFileBeneath` (preserving the path-traversal guard) with an `O_EXCL` + uuid name. 2. `os.Rename` over the target — atomic on POSIX within one filesystem, so a reader sees either the complete old file or the complete new file, never a torn one. Temp files use a reserved `.wsp-tmp.` prefix that the recursive `ls` skips, so an in-flight (or crash-leftover) temp never appears in a listing. `s3`/`azure` providers already write atomically via a single upload and are untouched. This is an unconditional fix — no new API or flag. Every directory-provider write (including revision snapshots) becomes atomic transparently. ## Notes / tradeoffs - **Windows:** `os.Rename` over an existing file is not atomic on Windows; deployment is Linux (documented in a code comment). - **Stale temps on hard crash:** a SIGKILL/power-loss between temp-create and rename leaves a `.wsp-tmp.<uuid>` file. It is hidden from listings by the `ls`-skip and reclaimed on workspace removal — accepted tradeoff, no reaper added. ## Tests TDD: a new `TestWriteFile_AtomicNoTornFileOnError` fails against the old `O_TRUNC` code (a mid-write reader error leaves a torn file) and passes after the change (original content survives). `TestLs_SkipsTempFiles` covers the listing skip. `go build ./...`, `go vet ./...`, and the full `pkg/client` suite pass. ## Related - Companion to the bpai-tools `.metadata.json` hardening (throttled writes + rebuild-on-corrupt) — the tool-side counterpart. - Controller-side counterpart: bpai #320 (queue-starvation fix). |
||
| .github/workflows | ||
| pkg | ||
| .gitignore | ||
| .golangci.yml | ||
| CLAUDE.md | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| main.go | ||
| Makefile | ||
| README.md | ||
| tool.gpt | ||
workspace-provider
There are three providers that can be used to create and manage workspaces: directory, S3, and Azure.
Directory
The directory provider provides a directory-based workspace. This provider is used by default.
S3
The S3 provider provides a S3-based workspace.
This provider supports loading the default AWS configuration. You can control this configuration using the following environment variables:
AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_REGION
You must set the following environment variables:
WORKSPACE_PROVIDER_S3_BUCKET
Usage with S3-compatible providers (e.g. Cloudflare R2)
You can use the above referenced AWS environment variables to configure the S3 provider, setting the value of the environment variable to the corresponding value from your provider.
Additionally, you should also set the WORKSPACE_PROVIDER_S3_BASE_ENDPOINT environment variable to the endpoint of your provider. For example, if you are using Cloudflare R2, you can set WORKSPACE_PROVIDER_S3_BASE_ENDPOINT to https://<ACCOUNT_ID>.r2.cloudflarestorage.com.
Some S3-compatible providers do not support virtual-hosted-style requests and require path-style addressing instead (e.g. Garage and MinIO). For these, also set WORKSPACE_PROVIDER_S3_USE_PATH_STYLE=true. Make sure AWS_REGION matches the region configured on your provider (for Garage this is the s3_region value in garage.toml, which defaults to garage).
Azure
The Azure provider provides an Azure Blob Storage-based workspace.
Setup
- Create an Azure Storage Account in the Azure Portal
- Create a container in your storage account
- Get the connection string from your storage account (under "Access keys")
Configuration
You must set the following environment variables:
WORKSPACE_PROVIDER_AZURE_CONTAINER- The name of your Azure Storage containerWORKSPACE_PROVIDER_AZURE_CONNECTION_STRING- The connection string for your Azure Storage account
For example:
export WORKSPACE_PROVIDER_AZURE_CONTAINER="your-container-name"
export WORKSPACE_PROVIDER_AZURE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"