fix(openai): trim context for small-window models instead of hard-failing #2

Merged

thklein merged 3 commits from fix/context-window-overflow into main 2026-07-10 04:46:35 +00:00

Conversation 1 Commits 3 Files changed 4 +187 -6

thklein commented 2026-07-10 04:35:20 +00:00

Owner

Copy link

Problem

Self-hosted RAG bots (dor-ai-stack, vLLM behind KubeAI) frequently hit:

failed calling model for completion: request failed on non-retriable status-code 400:
{"object":"error","message":"max_tokens must be at least 1, got -3461","type":"BadRequestError","code":400}

Root cause is a context-window overflow that surfaces as a confusing vLLM error:

1. The prompt (system + retrieved knowledge chunks + question + tool schemas) exceeds the served model's --max-model-len (e.g. 32768 for mistral-small-3.1-24b).
2. bpai agents don't set maxTokens, so getBudget() fell back to the hardcoded DefaultMaxTokens = 400_000 ("GPT-5") and dropMessagesOverCount never trimmed the oversized context — it believed there was a 400k budget.
3. gptscript sends no max_tokens (0 → omitempty), so vLLM defaults it to max_model_len − prompt_tokens. When the prompt already overflows the window, that default is negative (-3461 is literally the overage) and vLLM rejects.
4. The context-limit retry loop only fired on OpenAI's context_length_exceeded code, so vLLM's BadRequestError slipped through as non-retriable → hard fail.

Notably, dor-ai-stack already ships GPTSCRIPT_DEFAULT_MAX_TOKENS: "32000" in its env, but the code never read it — it was dead config, which is why trimming never happened.

Fix

Two changes in pkg/openai:

1. getBudget() honors GPTSCRIPT_DEFAULT_MAX_TOKENS (positive int) as the default budget when a tool/model declares no maxTokens. This activates the already-deployed config so trimming assumes the real window instead of 400k. An explicitly-set per-tool maxTokens still wins.
2. isContextLengthError() recognizes vLLM's overflow errors (max_tokens must be at least 1, maximum context length, reduce the length) in addition to OpenAI's context_length_exceeded, so overflow trims-and-retries instead of failing non-retriably.

Both are needed: (1) makes the first attempt fit; (2) self-heals when the tiktoken (o200k_base) estimate diverges from the server's real tokenizer (German on Mistral runs ~15-25% higher than tiktoken).

Per-model note

The retry loop is inherently per-request/per-model — it reacts to the actual overflow from whichever model a thread uses, so mixed models across agents/threads are handled automatically. GPTSCRIPT_DEFAULT_MAX_TOKENS is a coarse per-process floor (set it to the smallest window in use). A future bpai change could feed each model's real context window into tool.MaxTokens (the plumbing already exists via engine.go populateMessageParams → getBudget(tool.MaxTokens)) to make the proactive budget exact per model; the retry loop remains the durable correctness layer regardless.

Tests

pkg/openai/contextlimit_test.go — env override / fallback / explicit-wins for getBudget, and error-detection matrix (vLLM negative max_tokens, wrapped errors, OpenAI code, maximum-context message, unrelated/nil/plain negatives). Full pkg/openai package green; go vet, gofmt, and whole-module go build clean.

## Problem Self-hosted RAG bots (dor-ai-stack, vLLM behind KubeAI) frequently hit: ``` failed calling model for completion: request failed on non-retriable status-code 400: {"object":"error","message":"max_tokens must be at least 1, got -3461","type":"BadRequestError","code":400} ``` Root cause is a context-window overflow that surfaces as a confusing vLLM error: 1. The prompt (system + retrieved knowledge chunks + question + tool schemas) exceeds the served model's `--max-model-len` (e.g. 32768 for `mistral-small-3.1-24b`). 2. bpai agents don't set `maxTokens`, so `getBudget()` fell back to the hardcoded `DefaultMaxTokens = 400_000` ("GPT-5") and `dropMessagesOverCount` never trimmed the oversized context — it believed there was a 400k budget. 3. gptscript sends **no** `max_tokens` (0 → `omitempty`), so vLLM defaults it to `max_model_len − prompt_tokens`. When the prompt already overflows the window, that default is **negative** (`-3461` is literally the overage) and vLLM rejects. 4. The context-limit retry loop only fired on OpenAI's `context_length_exceeded` code, so vLLM's `BadRequestError` slipped through as **non-retriable** → hard fail. Notably, dor-ai-stack already ships `GPTSCRIPT_DEFAULT_MAX_TOKENS: "32000"` in its env, but **the code never read it** — it was dead config, which is why trimming never happened. ## Fix Two changes in `pkg/openai`: 1. **`getBudget()` honors `GPTSCRIPT_DEFAULT_MAX_TOKENS`** (positive int) as the default budget when a tool/model declares no `maxTokens`. This activates the already-deployed config so trimming assumes the real window instead of 400k. An explicitly-set per-tool `maxTokens` still wins. 2. **`isContextLengthError()`** recognizes vLLM's overflow errors (`max_tokens must be at least 1`, `maximum context length`, `reduce the length`) in addition to OpenAI's `context_length_exceeded`, so overflow **trims-and-retries** instead of failing non-retriably. Both are needed: (1) makes the first attempt fit; (2) self-heals when the tiktoken (o200k_base) estimate diverges from the server's real tokenizer (German on Mistral runs ~15-25% higher than tiktoken). ### Per-model note The retry loop is inherently per-request/per-model — it reacts to the actual overflow from whichever model a thread uses, so mixed models across agents/threads are handled automatically. `GPTSCRIPT_DEFAULT_MAX_TOKENS` is a coarse per-process floor (set it to the smallest window in use). A future bpai change could feed each model's real context window into `tool.MaxTokens` (the plumbing already exists via `engine.go` `populateMessageParams` → `getBudget(tool.MaxTokens)`) to make the proactive budget exact per model; the retry loop remains the durable correctness layer regardless. ## Tests `pkg/openai/contextlimit_test.go` — env override / fallback / explicit-wins for `getBudget`, and error-detection matrix (vLLM negative max_tokens, wrapped errors, OpenAI code, maximum-context message, unrelated/nil/plain negatives). Full `pkg/openai` package green; `go vet`, `gofmt`, and whole-module `go build` clean.

thklein added 2 commits 2026-07-10 04:35:21 +00:00

chore: gitignore .worktrees/ 8b667d3356

fix(openai): trim context for small-window models instead of hard-failing ... d526005392

Self-hosted models (vLLM) have far smaller context windows than the
400k GPT default. Two problems caused hard 400 failures:

- getBudget() ignored the deployed GPTSCRIPT_DEFAULT_MAX_TOKENS env var,
  so message trimming assumed a 400k budget and never trimmed oversized
  RAG prompts. Honor the env var (positive int) as the default budget.

- The context-limit retry loop only fired on OpenAI's
  "context_length_exceeded" code. vLLM returns a plain 400 with message
  "max_tokens must be at least 1, got -N" (negative default completion
  length when the prompt already overflows the window) or "maximum
  context length ...". Detect these via isContextLengthError() so the
  request trims-and-retries instead of failing non-retriably.

thklein referenced this pull request from a commit 2026-07-10 04:37:44 +00:00

chore(deps): bump gptscript to context-overflow fix

thklein added 1 commit 2026-07-10 04:46:06 +00:00

fix(openai): harden context-limit classification and retry loop ... 06769b7bae

Address review findings:
- Gate isContextLengthError message matching on HTTP 400 and drop the
  overly-generic "reduce the length" needle, so unrelated 400s (param
  validation) and non-400 errors are never misclassified into the
  trim-and-retry loop (which would silently discard chat history).
- Bail early with an actionable error when the history can no longer be
  trimmed, instead of spinning all 10 retries on an identical request.
- Log and wrap the retry-exhaustion error so the failure is diagnosable.
- Warn once on an invalid GPTSCRIPT_DEFAULT_MAX_TOKENS value.

Adds regression tests for non-context 400s and non-400 context messages.

thklein commented 2026-07-10 04:46:22 +00:00

Author

Owner

Copy link

Fresh-context review (two independent subagents: general code-review + silent-failure hunter). Findings addressed in 06769b7:

Fixed

• Misclassification (critical): isContextLengthError matched the generic "reduce the length" substring with no status guard, so unrelated 400s (parameter validation) — and even non-400 errors — could be funneled into the trim-and-retry loop, silently discarding chat history. Now gated on HTTP 400 and the redundant "reduce the length" needle removed (real overflow messages already contain "maximum context length"). +2 regression tests (non-context 400, non-400 context message → both expect false).
• Retry spin: the loop could burn all 10 attempts re-issuing an identical failing request when a single message can't be trimmed further. Added a no-progress guard that bails immediately with an actionable error.
• Silent exhaustion: the loop-exhaustion path now logs (final maxTokens + remaining message count) and wraps the error so the failure is diagnosable.
• Misconfig visibility: warn once on an invalid GPTSCRIPT_DEFAULT_MAX_TOKENS.

Verified: go test ./pkg/openai -count=1, go vet, gofmt, and whole-module go build all clean. (Repo CI is red pre-existing on main — gofmt/staticcheck debt from the hard-fork rename, unrelated to this PR; none of the flagged files are touched here.)

Dismissed: none — all findings were real and fixed.

Fresh-context review (two independent subagents: general code-review + silent-failure hunter). Findings addressed in `06769b7`: **Fixed** - **Misclassification (critical):** `isContextLengthError` matched the generic `"reduce the length"` substring with no status guard, so unrelated 400s (parameter validation) — and even non-400 errors — could be funneled into the trim-and-retry loop, silently discarding chat history. Now gated on HTTP 400 and the redundant `"reduce the length"` needle removed (real overflow messages already contain "maximum context length"). +2 regression tests (`non-context 400`, `non-400 context message` → both expect `false`). - **Retry spin:** the loop could burn all 10 attempts re-issuing an identical failing request when a single message can't be trimmed further. Added a no-progress guard that bails immediately with an actionable error. - **Silent exhaustion:** the loop-exhaustion path now logs (final maxTokens + remaining message count) and wraps the error so the failure is diagnosable. - **Misconfig visibility:** warn once on an invalid `GPTSCRIPT_DEFAULT_MAX_TOKENS`. **Verified:** `go test ./pkg/openai -count=1`, `go vet`, `gofmt`, and whole-module `go build` all clean. (Repo CI is red pre-existing on `main` — gofmt/staticcheck debt from the hard-fork rename, unrelated to this PR; none of the flagged files are touched here.) **Dismissed:** none — all findings were real and fixed.

thklein merged commit 6af728555a into main 2026-07-10 04:46:35 +00:00

thklein referenced this pull request from a commit 2026-07-10 04:46:36 +00:00

fix(openai): trim context for small-window models instead of hard-failing (#2)

thklein deleted branch fix/context-window-overflow 2026-07-10 04:46:36 +00:00

thklein referenced this pull request from a commit 2026-07-10 04:48:49 +00:00

chore(deps): bump gptscript to context-overflow fix

thklein referenced this pull request from a commit 2026-07-10 04:49:36 +00:00

chore(deps): bump gptscript to context-overflow fix (#423)

thklein referenced this pull request 2026-10-18 10:29:21 +00:00

fix(deps): bump basepeak-ai modules to patched mains #23

thklein referenced this pull request from a commit 2026-10-18 10:43:52 +00:00

fix(deps): bump basepeak-ai modules to patched mains (#23)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/engine

  

clarity/clear

  

enhancement

  

security

  

size/M

No labels

area/engine

clarity/clear

enhancement

security

size/M

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

basepeak-ai/gptscript!2

No description provided.