Compare commits

...

11 Commits

Author SHA1 Message Date
claude-code-best
ea5df0ab60 chore: 2.4.0 2026-05-10 20:50:58 +08:00
claude-code-best
0ce8f7a1cb feat: 添加 GBK 编码自动检测支持,文件读写工具透明处理非 UTF-8 文件
新增 encoding.ts 核心模块实现三层编码检测(BOM → UTF-8 fatal → GBK 回退),
改造同步/异步读取路径和写入路径,使 FileReadTool/FileEditTool/FileWriteTool
能正确处理 GBK 编码文件。包含完整单元测试和 spec 文档。

Co-Authored-By: glm-5-turbo <zai-org@claude-code-best.win>
2026-05-10 20:50:12 +08:00
claude-code-best
6e1d3d8f47 fix: 修复 feature 的使用问题 2026-05-10 19:26:57 +08:00
claude-code-best
dc3d3e8839 fix: 移除 auto mode 的 provider 和模型白名单限制
移除 firstParty provider 限制和 claude-(opus|sonnet)-4-[67] 模型白名单,
使所有模型和 provider 在 TRANSCRIPT_CLASSIFIER feature 启用时均可使用 auto mode。

Co-Authored-By: glm-5-turbo <zai-org@claude-code-best.win>
2026-05-10 17:47:38 +08:00
claude-code-best
998890b469 Merge pull request #446 from claude-code-best/feature/prompt-cut-down
feat: 大量系统提示词优化
2026-05-10 15:30:34 +08:00
claude-code-best
3f0f699ca4 Merge pull request #445 from claude-code-best/feature/many-feature-packagee
feat: local memory + local vault wiring + autofix-pr + CI mock isolation (refactor)
2026-05-10 15:30:04 +08:00
claude-code-best
66b49d70ab chore: 2.3.0 2026-05-10 11:16:09 +08:00
claude-code-best
2006ab25ff fix: 添加 React Error Boundary 防止生产环境渲染崩溃
增强 SentryErrorBoundary 组件,捕获渲染错误时输出诊断信息
(错误消息 + component stack)到 stderr 和终端,而非静默返回
null。在 replLauncher 根节点和 Messages 组件层级包裹 Error
Boundary,防止 Ink 内部的 Error Boundary 直接终止进程。

Co-Authored-By: glm-5-turbo <zai-org@claude-code-best.win>
2026-05-09 22:02:04 +08:00
claude-code-best
0707284939 docs: 更新 CLAUDE.md — 同步 workspace 包数量、feature flags、工具目录等变更
Co-Authored-By: glm-5-turbo <zai-org@claude-code-best.win>
2026-05-09 17:50:55 +08:00
claude-code-best
84f12f34bd fix: 提升 CLAUDE.md 指令权重 — 独立 project-instructions + deferred tools 位置调整
- prependUserContext: 将 claudeMd 从通用 <system-reminder> 提取为独立的
  <project-instructions> 用户消息,不带免责声明,置于消息列表最前面
- queryModel: deferred tools 消息从 prepend 改为 append,避免抢占
  project-instructions 的最高权重位置;标签规范化为 <system-reminder>

Co-Authored-By: glm-5-turbo <zai-org@claude-code-best.win>
2026-05-09 17:50:15 +08:00
claude-code-best
2f86485d9c refactor: 精简系统提示词 — 合并沟通风格段落、精简 memory/工具描述、截断 gitStatus
- 合并 getOutputEfficiencySection + getSimpleToneAndStyleSection 为精简的 Communication style
- 精简 auto memory 指令:删除 4 种类型的详细说明和示例,仅保留核心 description
- 精简 Agent 工具:删除 forkExamples 和 currentExamples 大段示例
- 精简 Bash 工具:合并 sleep 相关指导
- 精简 EnterPlanMode/ExitPlanMode:删除详细 GOOD/BAD 示例
- gitStatus MAX_STATUS_CHARS 从 2000 降到 1000
- 同步更新 prompt engineering audit 测试断言

Co-Authored-By: glm-5-turbo <zai-org@claude-code-best.win>
2026-05-09 17:14:41 +08:00
39 changed files with 1906 additions and 554 deletions

View File

@@ -82,11 +82,11 @@ bun run docs:dev
- **Vendor 路径解析**: 构建后 chunk 文件位于 `dist/``dist/chunks/`vendor 二进制在 `dist/vendor/``src/utils/ripgrep.ts``packages/audio-capture-napi/src/index.ts` 均通过 `import.meta.url` 路径中 `lastIndexOf('dist')` 定位 dist 根目录,再拼接 `vendor/` 子路径,确保不同构建产物层级下路径一致。 - **Vendor 路径解析**: 构建后 chunk 文件位于 `dist/``dist/chunks/`vendor 二进制在 `dist/vendor/``src/utils/ripgrep.ts``packages/audio-capture-napi/src/index.ts` 均通过 `import.meta.url` 路径中 `lastIndexOf('dist')` 定位 dist 根目录,再拼接 `vendor/` 子路径,确保不同构建产物层级下路径一致。
- **Dev mode**: `scripts/dev.ts` 通过 Bun `-d` flag 注入 `MACRO.*` defines运行 `src/entrypoints/cli.tsx`。默认启用全部 feature。 - **Dev mode**: `scripts/dev.ts` 通过 Bun `-d` flag 注入 `MACRO.*` defines运行 `src/entrypoints/cli.tsx`。默认启用全部 feature。
- **Module system**: ESM (`"type": "module"`), TSX with `react-jsx` transform. - **Module system**: ESM (`"type": "module"`), TSX with `react-jsx` transform.
- **Monorepo**: Bun workspaces — 15 个 workspace packages + 若干辅助目录 in `packages/` resolved via `workspace:*` - **Monorepo**: Bun workspaces — 17 个 workspace packages + 若干辅助目录 in `packages/` resolved via `workspace:*`
- **Lint/Format**: Biome (`biome.json`)。覆盖 `src/``scripts/``packages/` 全项目(含 `packages/@ant/`)。`bun run lint` / `bun run lint:fix` / `bun run format` / `bun run check` / `bun run check:fix`。42 条规则因 decompiled 代码被关闭,仅保留 `recommended` 基线。 - **Lint/Format**: Biome (`biome.json`)。覆盖 `src/``scripts/``packages/` 全项目(含 `packages/@ant/`)。`bun run lint` / `bun run lint:fix` / `bun run format` / `bun run check` / `bun run check:fix`。42 条规则因 decompiled 代码被关闭,仅保留 `recommended` 基线。
- **Pre-commit**: husky + lint-staged。提交时自动对暂存文件执行 `biome check --fix`TS/JS`biome format --write`JSON - **Pre-commit**: husky + lint-staged。提交时自动对暂存文件执行 `biome check --fix`TS/JS`biome format --write`JSON
- **CI Lint**: `ci.yml` 在依赖安装后、类型检查前执行 `bunx biome ci .`lint 或格式化不达标则 CI 失败。 - **CI Lint**: `ci.yml` 在依赖安装后、类型检查前执行 `bunx biome ci .`lint 或格式化不达标则 CI 失败。
- **Defines**: 集中管理在 `scripts/defines.ts`。当前版本 `2.1.888` - **Defines**: 集中管理在 `scripts/defines.ts`。当前版本 `2.2.1`
- **CI**: GitHub Actions — `ci.yml`lint + 构建 + 测试)、`release-rcs.yml`RCS 发布)、`update-contributors.yml`(自动更新贡献者)。 - **CI**: GitHub Actions — `ci.yml`lint + 构建 + 测试)、`release-rcs.yml`RCS 发布)、`update-contributors.yml`(自动更新贡献者)。
### Entry & Bootstrap ### Entry & Bootstrap
@@ -104,7 +104,7 @@ bun run docs:dev
- `environment-runner` / `self-hosted-runner` — BYOC runner - `environment-runner` / `self-hosted-runner` — BYOC runner
- `--tmux` + `--worktree` 组合 - `--tmux` + `--worktree` 组合
- 默认路径:加载 `main.tsx` 启动完整 CLI - 默认路径:加载 `main.tsx` 启动完整 CLI
2. **`src/main.tsx`** (~6981 行) — Commander.js CLI definition。注册大量 subcommands`mcp` (serve/add/remove/list...)、`server``ssh``open``auth``plugin``agents``auto-mode``doctor``update` 等。主 `.action()` 处理器负责权限、MCP、会话恢复、REPL/Headless 模式分发。 2. **`src/main.tsx`** (~5674 行) — Commander.js CLI definition。注册大量 subcommands`mcp` (serve/add/remove/list...)、`server``ssh``open``auth``plugin``agents``auto-mode``doctor``update` 等。主 `.action()` 处理器负责权限、MCP、会话恢复、REPL/Headless 模式分发。
3. **`src/entrypoints/init.ts`** — One-time initialization (telemetry, config, trust dialog)。 3. **`src/entrypoints/init.ts`** — One-time initialization (telemetry, config, trust dialog)。
### Core Loop ### Core Loop
@@ -119,21 +119,27 @@ bun run docs:dev
- **7 providers**: `firstParty` (Anthropic direct), `bedrock` (AWS), `vertex` (Google Cloud), `foundry`, `openai`, `gemini`, `grok` (xAI)。 - **7 providers**: `firstParty` (Anthropic direct), `bedrock` (AWS), `vertex` (Google Cloud), `foundry`, `openai`, `gemini`, `grok` (xAI)。
- Provider selection in `src/utils/model/providers.ts`。优先级modelType 参数 > 环境变量 > 默认 firstParty。 - Provider selection in `src/utils/model/providers.ts`。优先级modelType 参数 > 环境变量 > 默认 firstParty。
### Encoding Detection
- **`src/utils/encoding.ts`** — 文件编码检测的唯一入口。提供 `detectEncoding`三层检测BOM → UTF-8 fatal → ICU 回退链)和 `decodeBuffer`/`encodeString` 函数。检测基于文件头部 4KB零外部依赖仅使用 TextDecoder API。ISO-8859-1 作为最终兜底编码(单字节编码永远成功)。`FileEncoding` 类型扩展了 `BufferEncoding`,覆盖 gbk/gb18030/shift_jis/euc-kr/euc-jp/big5/iso-8859-1。
- `fs.readFileSync(path, { encoding })``encoding` 选项只接受 `BufferEncoding`,不支持 `gbk`/`shift_jis` 等 ICU 编码名。读取非 UTF-8 文件时必须先 `fs.readFileSync(path)` 读 Buffer再用 `TextDecoder` 解码。项目中所有文件读取路径fileRead.ts、fileReadCache.ts、file.ts已统一使用 `decodeBuffer` 函数处理此逻辑。
### Tool System ### Tool System
- **`src/Tool.ts`** — Tool interface definition (`Tool` type) and utilities (`findToolByName`, `toolMatchesName`). - **`src/Tool.ts`** — Tool interface definition (`Tool` type) and utilities (`findToolByName`, `toolMatchesName`).
- **`src/tools.ts`** — Tool registry. Assembles the tool list; tools are imported from `@claude-code-best/builtin-tools` package. Some tools are conditionally loaded via `feature()` flags or `process.env.USER_TYPE`. - **`src/tools.ts`** — Tool registry. Assembles the tool list; tools are imported from `@claude-code-best/builtin-tools` package. Some tools are conditionally loaded via `feature()` flags or `process.env.USER_TYPE`.
- **`src/constants/tools.ts`** — `CORE_TOOLS` 白名单常量(约 29 个核心工具名),用于 `isDeferredTool` 白名单制判定。 - **`src/constants/tools.ts`** — `CORE_TOOLS` 白名单常量(38 个核心工具名),用于 `isDeferredTool` 白名单制判定。
- **`packages/builtin-tools/src/tools/`** — 59 个子目录(含 shared/testing 等工具目录),通过 `@claude-code-best/builtin-tools` 包导出。主要分类: - **`packages/builtin-tools/src/tools/`** — 60 个工具目录(含 shared/testing 等工具目录),通过 `@claude-code-best/builtin-tools` 包导出。主要分类:
- **文件操作**: FileEditTool, FileReadTool, FileWriteTool, GlobTool, GrepTool - **文件操作**: FileEditTool, FileReadTool, FileWriteTool, GlobTool, GrepTool
- **Shell/执行**: BashTool, PowerShellTool, REPLTool - **Shell/执行**: BashTool, PowerShellTool, REPLTool
- **Agent 系统**: AgentTool, TaskCreateTool, TaskUpdateTool, TaskListTool, TaskGetTool - **Agent 系统**: AgentTool, TaskCreateTool, TaskUpdateTool, TaskListTool, TaskGetTool
- **规划**: EnterPlanModeTool, ExitPlanModeV2Tool, VerifyPlanExecutionTool - **规划**: EnterPlanModeTool, ExitPlanModeV2Tool, VerifyPlanExecutionTool
- **Web/MCP**: WebFetchTool, WebSearchTool, MCPTool, McpAuthTool - **Web/MCP**: WebFetchTool, WebSearchTool, MCPTool, McpAuthTool
- **调度**: CronCreateTool, CronDeleteTool, CronListTool - **调度**: CronCreateTool, CronDeleteTool, CronListTool
- **工具发现**: SearchExtraToolsTool, ExecuteExtraTool, SyntheticOutputCORE_TOOLS用于延迟工具按需加载
- **其他**: LSPTool, ConfigTool, SkillTool, EnterWorktreeTool, ExitWorktreeTool 等 - **其他**: LSPTool, ConfigTool, SkillTool, EnterWorktreeTool, ExitWorktreeTool 等
- **`src/tools/shared/`** / **`packages/builtin-tools/src/tools/shared/`** — Tool 共享工具函数。 - **`src/tools/shared/`** / **`packages/builtin-tools/src/tools/shared/`** — Tool 共享工具函数。
- **`src/services/searchExtraTools/`** — TF-IDF 工具索引模块(`toolIndex.ts`),为延迟工具提供语义搜索能力。复用 `localSearch.ts` 的 TF-IDF 算法函数(`computeWeightedTf``computeIdf``cosineSimilarity` 已导出)。修改这些函数时需同步检查工具索引测试。`SearchExtraToolsTool.mapToolResultToToolResultBlockParam` 新增可选第三个参数 `context?: { mainLoopModel?: string }`,用于判断当前模型是否支持 `tool_reference`。不支持时回退到文本输出,引导模型使用 ExecuteTool。调用方`src/services/api/claude.ts` 的 tool_result 处理逻辑)需传入 context 参数。`prefetch.ts``extractQueryFromMessages` 复用了 `skillSearch/prefetch.ts` 的同名导出函数,修改 skill prefetch 的该函数时需同步检查工具预取行为。工具预取使用独立的 `discoveredToolsThisSession` Set与 skill prefetch 的去重集合互不影响。 - **`src/services/searchExtraTools/`** — TF-IDF 工具索引模块(`toolIndex.ts`),为延迟工具提供语义搜索能力。复用 `localSearch.ts` 的 TF-IDF 算法函数(`computeWeightedTf``computeIdf``cosineSimilarity` 已导出)。修改这些函数时需同步检查工具索引测试。`prefetch.ts``extractQueryFromMessages` 复用了 `skillSearch/prefetch.ts` 的同名导出函数,修改 skill prefetch 的该函数时需同步检查工具预取行为。工具预取使用独立的 `discoveredToolsThisSession` Set与 skill prefetch 的去重集合互不影响。
### UI Layer (Ink) ### UI Layer (Ink)
@@ -168,18 +174,16 @@ bun run docs:dev
| `packages/builtin-tools/` | 内置工具集60 个 tool 实现,通过 `@claude-code-best/builtin-tools` 导出) | | `packages/builtin-tools/` | 内置工具集60 个 tool 实现,通过 `@claude-code-best/builtin-tools` 导出) |
| `packages/agent-tools/` | Agent 工具集 | | `packages/agent-tools/` | Agent 工具集 |
| `packages/acp-link/` | ACP 代理服务器WebSocket → ACP agent 桥接) | | `packages/acp-link/` | ACP 代理服务器WebSocket → ACP agent 桥接) |
| `packages/cc-knowledge/` | Claude Code 知识库(非 workspace 包) |
| `packages/langfuse-dashboard/` | Langfuse 可观测性面板(非 workspace 包) |
| `packages/mcp-client/` | MCP 客户端库 | | `packages/mcp-client/` | MCP 客户端库 |
| `packages/mcp-server/` | MCP 服务端库(非 workspace 包) |
| `packages/remote-control-server/` | 自托管 Remote Control ServerDocker 部署,含 Web UI— Web UI 已重构为 React + Vite + Radix UI支持 ACP agent 接入 | | `packages/remote-control-server/` | 自托管 Remote Control ServerDocker 部署,含 Web UI— Web UI 已重构为 React + Vite + Radix UI支持 ACP agent 接入 |
| `packages/swarm/` | Swarm 解耦模块(非 workspace 包) |
| `packages/shell/` | Shell 抽象(非 workspace 包) |
| `packages/audio-capture-napi/` | 原生音频捕获(已恢复) | | `packages/audio-capture-napi/` | 原生音频捕获(已恢复) |
| `packages/color-diff-napi/` | 颜色差异计算完整实现11 tests | | `packages/color-diff-napi/` | 颜色差异计算完整实现11 tests |
| `packages/image-processor-napi/` | 图像处理(已恢复) | | `packages/image-processor-napi/` | 图像处理(已恢复) |
| `packages/modifiers-napi/` | 键盘修饰键检测macOS FFI 实现) | | `packages/modifiers-napi/` | 键盘修饰键检测macOS FFI 实现) |
| `packages/url-handler-napi/` | URL scheme 处理(环境变量 + CLI 参数读取) | | `packages/url-handler-napi/` | URL scheme 处理(环境变量 + CLI 参数读取) |
| `packages/weixin/` | 微信集成(非 workspace 包) |
辅助目录(无 package.json非 workspace 包): `langfuse-dashboard`Langfuse 面板)、`shared-web-ui`(共享 Web UI 组件)、`highlight-code`(代码高亮)、`claude-pencil`(编辑器)、`vscode-ide-bridge`VS Code 桥接)、`pokemon`(示例/测试)。
### Bridge / Remote Control ### Bridge / Remote Control
@@ -210,12 +214,18 @@ Feature flags control which functionality is enabled at runtime. 代码中统一
**启用方式**: 环境变量 `FEATURE_<FLAG_NAME>=1`。例如 `FEATURE_BUDDY=1 bun run dev` **启用方式**: 环境变量 `FEATURE_<FLAG_NAME>=1`。例如 `FEATURE_BUDDY=1 bun run dev`
**Build 默认 features**19 个,见 `build.ts`: **Build 默认 features**65+ 个,见 `build.ts``DEFAULT_BUILD_FEATURES`:
- 基础: `BUDDY`, `TRANSCRIPT_CLASSIFIER`, `BRIDGE_MODE`, `AGENT_TRIGGERS_REMOTE`, `CHICAGO_MCP`, `VOICE_MODE` - 基础: `BUDDY`, `TRANSCRIPT_CLASSIFIER`, `BRIDGE_MODE`, `AGENT_TRIGGERS_REMOTE`, `CHICAGO_MCP`, `VOICE_MODE`
- 统计/缓存: `SHOT_STATS`, `PROMPT_CACHE_BREAK_DETECTION`, `TOKEN_BUDGET` - 统计/缓存: `SHOT_STATS`, `PROMPT_CACHE_BREAK_DETECTION`, `TOKEN_BUDGET`
- P0 本地: `AGENT_TRIGGERS`, `ULTRATHINK`, `BUILTIN_EXPLORE_PLAN_AGENTS`, `LODESTONE` - P0 本地: `AGENT_TRIGGERS`, `ULTRATHINK`, `BUILTIN_EXPLORE_PLAN_AGENTS`, `LODESTONE`
- P1 API 依赖: `EXTRACT_MEMORIES`, `VERIFICATION_AGENT`, `KAIROS_BRIEF`, `AWAY_SUMMARY`, `ULTRAPLAN` - P1 API 依赖: `EXTRACT_MEMORIES`, `VERIFICATION_AGENT`, `KAIROS_BRIEF`, `AWAY_SUMMARY`, `ULTRAPLAN`
- P2: `DAEMON` - P2: `DAEMON`, `ACP`
- 工作流: `WORKFLOW_SCRIPTS`, `HISTORY_SNIP`, `MONITOR_TOOL`, `KAIROS`
- 多 worker: `COORDINATOR_MODE`, `BG_SESSIONS`, `TEMPLATES`
- 连接器: `CONNECTOR_TEXT`, `COMMIT_ATTRIBUTION`, `DIRECT_CONNECT`
- 实验性: `EXPERIMENTAL_SKILL_SEARCH`, `EXPERIMENTAL_SEARCH_EXTRA_TOOLS`
- 模式: `POOR`, `SSH_REMOTE`
- 已禁用: `CONTEXT_COLLAPSE`, `FORK_SUBAGENT`, `UDS_INBOX`, `LAN_PIPES`, `REVIEW_ARTIFACT`, `TEAMMEM`, `SKILL_LEARNING`
**Dev mode 默认**: 全部启用(见 `scripts/dev.ts`)。 **Dev mode 默认**: 全部启用(见 `scripts/dev.ts`)。
@@ -265,6 +275,7 @@ Feature flags control which functionality is enabled at runtime. 代码中统一
| Voice Mode | Restored — Push-to-Talk 语音输入(需 Anthropic OAuth | | Voice Mode | Restored — Push-to-Talk 语音输入(需 Anthropic OAuth |
| OpenAI/Gemini/Grok 兼容层 | Restored | | OpenAI/Gemini/Grok 兼容层 | Restored |
| Remote Control Server | Restored — 自托管 RCS + Web UI | | Remote Control Server | Restored — 自托管 RCS + Web UI |
| `packages/shell/`, `packages/swarm/`, `packages/mcp-server/`, `packages/cc-knowledge/` | Removed — 功能合并或废弃 |
| Analytics / GrowthBook / Sentry | Empty implementations | | Analytics / GrowthBook / Sentry | Empty implementations |
| Magic Docs / LSP Server | Restored — Magic Docs 自动更新 + LSP 服务器管理器 | | Magic Docs / LSP Server | Restored — Magic Docs 自动更新 + LSP 服务器管理器 |
| Plugins / Marketplace | Restored — 插件安装/卸载/启用/禁用 + Marketplace 浏览 | | Plugins / Marketplace | Restored — 插件安装/卸载/启用/禁用 + Marketplace 浏览 |
@@ -281,7 +292,7 @@ Feature flags control which functionality is enabled at runtime. 代码中统一
- **框架**: `bun:test`(内置断言 + mock - **框架**: `bun:test`(内置断言 + mock
- **单元测试**: 就近放置于 `src/**/__tests__/`,文件名 `<module>.test.ts` - **单元测试**: 就近放置于 `src/**/__tests__/`,文件名 `<module>.test.ts`
- **集成测试**: `tests/integration/`4 个文件cli-arguments, context-build, message-pipeline, tool-chain - **集成测试**: `tests/integration/`6 个文件cli-arguments, context-build, message-pipeline, tool-chain, autonomy-lifecycle-user-flow, dependency-overrides
- **共享 mock/fixture**: `tests/mocks/`api-responses, file-system, fixtures/ - **共享 mock/fixture**: `tests/mocks/`api-responses, file-system, fixtures/
- **命名**: `describe("functionName")` + `test("behavior description")`,英文 - **命名**: `describe("functionName")` + `test("behavior description")`,英文
- **包测试**: `packages/` 下各包也有独立测试(如 `color-diff-napi` 11 tests - **包测试**: `packages/` 下各包也有独立测试(如 `color-diff-napi` 11 tests

View File

@@ -1,6 +1,6 @@
{ {
"name": "claude-code-best", "name": "claude-code-best",
"version": "2.2.1", "version": "2.4.0",
"description": "Reverse-engineered Anthropic Claude Code CLI — interactive AI coding assistant in the terminal", "description": "Reverse-engineered Anthropic Claude Code CLI — interactive AI coding assistant in the terminal",
"type": "module", "type": "module",
"author": "claude-code-best <claude-code-best@proton.me>", "author": "claude-code-best <claude-code-best@proton.me>",

View File

@@ -57,13 +57,4 @@ describe('prompt.ts fork-related text verification', () => {
expect(bgCondition[0]).not.toContain('!forkEnabled') expect(bgCondition[0]).not.toContain('!forkEnabled')
} }
}) })
test('fork example includes fork: true parameter', () => {
// The first fork example should have fork: true
const forkExampleBlock = promptSource.match(
/name: "ship-audit"[\s\S]*?Under 200 words/,
)
expect(forkExampleBlock).not.toBeNull()
expect(forkExampleBlock![0]).toContain('fork: true')
})
}) })

View File

@@ -5,7 +5,6 @@ import { isEnvDefinedFalsy, isEnvTruthy } from 'src/utils/envUtils.js'
import { isTeammate } from 'src/utils/teammate.js' import { isTeammate } from 'src/utils/teammate.js'
import { isInProcessTeammate } from 'src/utils/teammateContext.js' import { isInProcessTeammate } from 'src/utils/teammateContext.js'
import { FILE_READ_TOOL_NAME } from '../FileReadTool/prompt.js' import { FILE_READ_TOOL_NAME } from '../FileReadTool/prompt.js'
import { FILE_WRITE_TOOL_NAME } from '../FileWriteTool/prompt.js'
import { GLOB_TOOL_NAME } from '../GlobTool/prompt.js' import { GLOB_TOOL_NAME } from '../GlobTool/prompt.js'
import { SEND_MESSAGE_TOOL_NAME } from '../SendMessageTool/constants.js' import { SEND_MESSAGE_TOOL_NAME } from '../SendMessageTool/constants.js'
import { AGENT_TOOL_NAME } from './constants.js' import { AGENT_TOOL_NAME } from './constants.js'
@@ -84,11 +83,11 @@ export async function getPrompt(
When you need to delegate work that benefits from full conversation context (e.g., continuing a multi-file refactor where the child needs the same system prompt and history), use \`fork: true\`. For most tasks, prefer specialized agent types (Explore, Plan, general-purpose). When you need to delegate work that benefits from full conversation context (e.g., continuing a multi-file refactor where the child needs the same system prompt and history), use \`fork: true\`. For most tasks, prefer specialized agent types (Explore, Plan, general-purpose).
**Don't peek.** The tool result includes an \`output_file\` path — do not Read or tail it unless the user explicitly asks for a progress check. You get a completion notification; trust it. Reading the transcript mid-flight pulls the fork's tool noise into your context, which defeats the point of forking. **Don't peek.** The tool result includes an \`output_file\` path — do not Read or tail it unless the user explicitly asks for a progress check. You get a completion notification; trust it.
**Don't race.** After launching, you know nothing about what the fork found. Never fabricate or predict fork results in any format — not as prose, summary, or structured output. The notification arrives as a user-role message in a later turn; it is never something you write yourself. If the user asks a follow-up before the notification lands, tell them the fork is still running — give status, not a guess. **Don't race.** After launching, you know nothing about what the fork found. Never fabricate or predict fork results. If the user asks a follow-up before the notification lands, tell them the fork is still running.
**Writing a fork prompt.** Since the fork inherits your context, the prompt is a *directive* — what to do, not what the situation is. Be specific about scope: what's in, what's out, what another agent is handling. Don't re-explain background. **Writing a fork prompt.** Since the fork inherits your context, the prompt is a *directive* — what to do, not what the situation is. Be specific about scope. Don't re-explain background.
` `
: '' : ''
@@ -97,91 +96,13 @@ When you need to delegate work that benefits from full conversation context (e.g
## Writing the prompt ## Writing the prompt
${forkEnabled ? 'When spawning an agent without `fork: true`, it starts with zero context. ' : ''}Brief the agent like a smart colleague who just walked into the room — it hasn't seen this conversation, doesn't know what you've tried, doesn't understand why this task matters. ${forkEnabled ? 'When spawning an agent without `fork: true`, it starts with zero context. ' : ''}Brief the agent like a smart colleague who just walked into the room — it hasn't seen this conversation, doesn't know what you've tried, doesn't understand why this task matters.
- Explain what you're trying to accomplish and why. - Explain what you're trying to accomplish and why, what you've already learned or ruled out, and enough context for the agent to make judgment calls.
- Describe what you've already learned or ruled out.
- Give enough context about the surrounding problem that the agent can make judgment calls rather than just following a narrow instruction.
- If you need a short response, say so ("report in under 200 words"). - If you need a short response, say so ("report in under 200 words").
- Lookups: hand over the exact command. Investigations: hand over the question — prescribed steps become dead weight when the premise is wrong. - Lookups: hand over the exact command. Investigations: hand over the question — prescribed steps become dead weight when the premise is wrong.
${forkEnabled ? 'For non-fork agents, terse' : 'Terse'} command-style prompts produce shallow, generic work. ${forkEnabled ? 'For non-fork agents, terse' : 'Terse'} command-style prompts produce shallow, generic work.
**Never delegate understanding.** Don't write "based on your findings, fix the bug" or "based on the research, implement it." Those phrases push synthesis onto the agent instead of doing it yourself. Write prompts that prove you understood: include file paths, line numbers, what specifically to change. **Never delegate understanding.** Don't write "based on your findings, fix the bug" or "based on the research, implement it." Write prompts that prove you understood: include file paths, line numbers, what specifically to change.
`
const forkExamples = `Example usage:
<example>
user: "What's left on this branch before we can ship?"
assistant: <thinking>Forking this \u2014 it's a survey question. I want the punch list, not the git output in my context.</thinking>
${AGENT_TOOL_NAME}({
name: "ship-audit",
description: "Branch ship-readiness audit",
fork: true,
prompt: "Audit what's left before this branch can ship. Check: uncommitted changes, commits ahead of main, whether tests exist, whether the GrowthBook gate is wired up, whether CI-relevant files changed. Report a punch list \u2014 done vs. missing. Under 200 words."
})
assistant: Ship-readiness audit running.
<commentary>
Turn ends here. The coordinator knows nothing about the findings yet. What follows is a SEPARATE turn \u2014 the notification arrives from outside, as a user-role message. It is not something the coordinator writes.
</commentary>
[later turn \u2014 notification arrives as user message]
assistant: Audit's back. Three blockers: no tests for the new prompt path, GrowthBook gate wired but not in build_flags.yaml, and one uncommitted file.
</example>
<example>
user: "so is the gate wired up or not"
<commentary>
User asks mid-wait. The audit fork was launched to answer exactly this, and it hasn't returned. The coordinator does not have this answer. Give status, not a fabricated result.
</commentary>
assistant: Still waiting on the audit \u2014 that's one of the things it's checking. Should land shortly.
</example>
<example>
user: "Can you get a second opinion on whether this migration is safe?"
assistant: <thinking>I'll ask the code-reviewer agent — it won't see my analysis, so it can give an independent read.</thinking>
<commentary>
A subagent_type is specified, so the agent starts fresh. It needs full context in the prompt. The briefing explains what to assess and why.
</commentary>
${AGENT_TOOL_NAME}({
name: "migration-review",
description: "Independent migration review",
subagent_type: "code-reviewer",
prompt: "Review migration 0042_user_schema.sql for safety. Context: we're adding a NOT NULL column to a 50M-row table. Existing rows get a backfill default. I want a second opinion on whether the backfill approach is safe under concurrent writes — I've checked locking behavior but want independent verification. Report: is this safe, and if not, what specifically breaks?"
})
</example>
`
const currentExamples = `Example usage:
<example_agent_descriptions>
"test-runner": use this agent after you are done writing code to run tests
"greeting-responder": use this agent to respond to user greetings with a friendly joke
</example_agent_descriptions>
<example>
user: "Please write a function that checks if a number is prime"
assistant: I'm going to use the ${FILE_WRITE_TOOL_NAME} tool to write the following code:
<code>
function isPrime(n) {
if (n <= 1) return false
for (let i = 2; i * i <= n; i++) {
if (n % i === 0) return false
}
return true
}
</code>
<commentary>
Since a significant piece of code was written and the task was completed, now use the test-runner agent to run the tests
</commentary>
assistant: Uses the ${AGENT_TOOL_NAME} tool to launch the test-runner agent
</example>
<example>
user: "Hello"
<commentary>
Since the user is greeting, use the greeting-responder agent to respond with a friendly joke
</commentary>
assistant: "I'm going to use the ${AGENT_TOOL_NAME} tool to launch the greeting-responder agent"
</example>
` `
// When the gate is on, the agent list lives in an agent_listing_delta // When the gate is on, the agent list lives in an agent_listing_delta
@@ -273,7 +194,5 @@ Usage notes:
? ` ? `
- The name, team_name, and mode parameters are not available in this context — teammates cannot spawn other teammates. Omit them to spawn a subagent.` - The name, team_name, and mode parameters are not available in this context — teammates cannot spawn other teammates. Omit them to spawn a subagent.`
: '' : ''
}${whenToForkSection}${writingThePromptSection} }${whenToForkSection}${writingThePromptSection}`
${forkEnabled ? forkExamples : currentExamples}`
} }

View File

@@ -29,6 +29,7 @@ import { extractClaudeCodeHints } from 'src/utils/claudeCodeHints.js';
import { detectCodeIndexingFromCommand } from 'src/utils/codeIndexing.js'; import { detectCodeIndexingFromCommand } from 'src/utils/codeIndexing.js';
import { isEnvTruthy } from 'src/utils/envUtils.js'; import { isEnvTruthy } from 'src/utils/envUtils.js';
import { isENOENT, ShellError } from 'src/utils/errors.js'; import { isENOENT, ShellError } from 'src/utils/errors.js';
import { decodeBuffer } from 'src/utils/encoding.js';
import { detectFileEncoding, detectLineEndings, getFileModificationTime, writeTextContent } from 'src/utils/file.js'; import { detectFileEncoding, detectLineEndings, getFileModificationTime, writeTextContent } from 'src/utils/file.js';
import { fileHistoryEnabled, fileHistoryTrackEdit } from 'src/utils/fileHistory.js'; import { fileHistoryEnabled, fileHistoryTrackEdit } from 'src/utils/fileHistory.js';
import { truncate } from 'src/utils/format.js'; import { truncate } from 'src/utils/format.js';
@@ -511,7 +512,8 @@ async function applySedEdit(
const encoding = detectFileEncoding(absoluteFilePath); const encoding = detectFileEncoding(absoluteFilePath);
let originalContent: string; let originalContent: string;
try { try {
originalContent = await fs.readFile(absoluteFilePath, { encoding }); const rawBuffer = await fs.readFileBytes(absoluteFilePath);
originalContent = decodeBuffer(rawBuffer, encoding);
} catch (e) { } catch (e) {
if (isENOENT(e)) { if (isENOENT(e)) {
return { return {

View File

@@ -314,15 +314,13 @@ export function getSimplePrompt(): string {
'Use the Monitor tool to stream events from a background process (each stdout line is a notification). For one-shot "wait until done," use Bash with run_in_background instead.', 'Use the Monitor tool to stream events from a background process (each stdout line is a notification). For one-shot "wait until done," use Bash with run_in_background instead.',
] ]
: []), : []),
'If your command is long running and you would like to be notified when it finishes — use `run_in_background`. No sleep needed.', 'For long-running commands, use `run_in_background` — you will be notified when it completes. Do not poll.',
'Do not retry failing commands in a sleep loop — diagnose the root cause.', 'Do not retry failing commands in a sleep loop — diagnose the root cause.',
'If waiting for a background task you started with `run_in_background`, you will be notified when it completes — do not poll.',
...(feature('MONITOR_TOOL') ...(feature('MONITOR_TOOL')
? [ ? [
'`sleep N` as the first command with N ≥ 2 is blocked. If you need a delay (rate limiting, deliberate pacing), keep it under 2 seconds.', '`sleep N` as the first command with N ≥ 2 is blocked. If you need a delay (rate limiting, deliberate pacing), keep it under 2 seconds.',
] ]
: [ : [
'If you must poll an external process, use a check command (e.g. `gh run view`) rather than sleeping first.',
'If you must sleep, keep the duration short (1-5 seconds) to avoid blocking the user.', 'If you must sleep, keep the duration short (1-5 seconds) to avoid blocking the user.',
]), ]),
] ]

View File

@@ -26,33 +26,13 @@ function getEnterPlanModeToolPromptExternal(): string {
**Prefer using EnterPlanMode** for implementation tasks unless they're simple. Use it when ANY of these conditions apply: **Prefer using EnterPlanMode** for implementation tasks unless they're simple. Use it when ANY of these conditions apply:
1. **New Feature Implementation**: Adding meaningful new functionality 1. **New Feature Implementation** Adding meaningful new functionality where the implementation path isn't obvious
- Example: "Add a logout button" - where should it go? What should happen on click? 2. **Multiple Valid Approaches** — The task can be solved in several different ways
- Example: "Add form validation" - what rules? What error messages? 3. **Code Modifications** — Changes that affect existing behavior or structure, where the user should approve the approach
4. **Architectural Decisions** — The task requires choosing between patterns or technologies
2. **Multiple Valid Approaches**: The task can be solved in several different ways 5. **Multi-File Changes** The task will likely touch more than 2-3 files
- Example: "Add caching to the API" - could use Redis, in-memory, file-based, etc. 6. **Unclear Requirements** — You need to explore before understanding the full scope
- Example: "Improve performance" - many optimization strategies possible 7. **User Preferences Matter** — If you would use ${ASK_USER_QUESTION_TOOL_NAME} to clarify the approach, use EnterPlanMode instead
3. **Code Modifications**: Changes that affect existing behavior or structure
- Example: "Update the login flow" - what exactly should change?
- Example: "Refactor this component" - what's the target architecture?
4. **Architectural Decisions**: The task requires choosing between patterns or technologies
- Example: "Add real-time updates" - WebSockets vs SSE vs polling
- Example: "Implement state management" - Redux vs Context vs custom solution
5. **Multi-File Changes**: The task will likely touch more than 2-3 files
- Example: "Refactor the authentication system"
- Example: "Add a new API endpoint with tests"
6. **Unclear Requirements**: You need to explore before understanding the full scope
- Example: "Make the app faster" - need to profile and identify bottlenecks
- Example: "Fix the bug in checkout" - need to investigate root cause
7. **User Preferences Matter**: The implementation could reasonably go multiple ways
- If you would use ${ASK_USER_QUESTION_TOOL_NAME} to clarify the approach, use EnterPlanMode instead
- Plan mode lets you explore first, then present options with context
## When NOT to Use This Tool ## When NOT to Use This Tool
@@ -62,35 +42,7 @@ Only skip EnterPlanMode for simple tasks:
- Tasks where the user has given very specific, detailed instructions - Tasks where the user has given very specific, detailed instructions
- Pure research/exploration tasks (use the Agent tool with explore agent instead) - Pure research/exploration tasks (use the Agent tool with explore agent instead)
${whatHappens}## Examples ${whatHappens}## Important Notes
### GOOD - Use EnterPlanMode:
User: "Add user authentication to the app"
- Requires architectural decisions (session vs JWT, where to store tokens, middleware structure)
User: "Optimize the database queries"
- Multiple approaches possible, need to profile first, significant impact
User: "Implement dark mode"
- Architectural decision on theme system, affects many components
User: "Add a delete button to the user profile"
- Seems simple but involves: where to place it, confirmation dialog, API call, error handling, state updates
User: "Update the error handling in the API"
- Affects multiple files, user should approve the approach
### BAD - Don't use EnterPlanMode:
User: "Fix the typo in the README"
- Straightforward, no planning needed
User: "Add a console.log to debug this function"
- Simple, obvious implementation
User: "What files handle routing?"
- Research task, not implementation planning
## Important Notes
- This tool REQUIRES user approval - they must consent to entering plan mode - This tool REQUIRES user approval - they must consent to entering plan mode
- If unsure whether to use it, err on the side of planning - it's better to get alignment upfront than to redo work - If unsure whether to use it, err on the side of planning - it's better to get alignment upfront than to redo work
@@ -111,53 +63,23 @@ function getEnterPlanModeToolPromptAnt(): string {
Plan mode is valuable when the implementation approach is genuinely unclear. Use it when: Plan mode is valuable when the implementation approach is genuinely unclear. Use it when:
1. **Significant Architectural Ambiguity**: Multiple reasonable approaches exist and the choice meaningfully affects the codebase 1. **Significant Architectural Ambiguity** Multiple reasonable approaches exist and the choice meaningfully affects the codebase
- Example: "Add caching to the API" - Redis vs in-memory vs file-based 2. **Unclear Requirements** — You need to explore and clarify before you can make progress
- Example: "Add real-time updates" - WebSockets vs SSE vs polling 3. **High-Impact Restructuring** — The task will significantly restructure existing code and getting buy-in first reduces risk
2. **Unclear Requirements**: You need to explore and clarify before you can make progress
- Example: "Make the app faster" - need to profile and identify bottlenecks
- Example: "Refactor this module" - need to understand what the target architecture should be
3. **High-Impact Restructuring**: The task will significantly restructure existing code and getting buy-in first reduces risk
- Example: "Redesign the authentication system"
- Example: "Migrate from one state management approach to another"
## When NOT to Use This Tool ## When NOT to Use This Tool
Skip plan mode when you can reasonably infer the right approach: Skip plan mode when you can reasonably infer the right approach:
- The task is straightforward even if it touches multiple files - The task is straightforward even if it touches multiple files
- The user's request is specific enough that the implementation path is clear - The user's request is specific enough that the implementation path is clear
- You're adding a feature with an obvious implementation pattern (e.g., adding a button, a new endpoint following existing conventions) - You're adding a feature with an obvious implementation pattern
- Bug fixes where the fix is clear once you understand the bug - Bug fixes where the fix is clear once you understand the bug
- Research/exploration tasks (use the Agent tool instead) - Research/exploration tasks (use the Agent tool instead)
- The user says something like "can we work on X" or "let's do X" — just get started - The user says something like "can we work on X" or "let's do X" — just get started
When in doubt, prefer starting work and using ${ASK_USER_QUESTION_TOOL_NAME} for specific questions over entering a full planning phase. When in doubt, prefer starting work and using ${ASK_USER_QUESTION_TOOL_NAME} for specific questions over entering a full planning phase.
${whatHappens}## Examples ${whatHappens}## Important Notes
### GOOD - Use EnterPlanMode:
User: "Add user authentication to the app"
- Genuinely ambiguous: session vs JWT, where to store tokens, middleware structure
User: "Redesign the data pipeline"
- Major restructuring where the wrong approach wastes significant effort
### BAD - Don't use EnterPlanMode:
User: "Add a delete button to the user profile"
- Implementation path is clear; just do it
User: "Can we work on the search feature?"
- User wants to get started, not plan
User: "Update the error handling in the API"
- Start working; ask specific questions if needed
User: "Fix the typo in the README"
- Straightforward, no planning needed
## Important Notes
- This tool REQUIRES user approval - they must consent to entering plan mode - This tool REQUIRES user approval - they must consent to entering plan mode
` `

View File

@@ -20,10 +20,4 @@ Ensure your plan is complete and unambiguous:
- Once your plan is finalized, use THIS tool to request approval - Once your plan is finalized, use THIS tool to request approval
**Important:** Do NOT use ${ASK_USER_QUESTION_TOOL_NAME} to ask "Is this plan okay?" or "Should I proceed?" - that's exactly what THIS tool does. ExitPlanMode inherently requests user approval of your plan. **Important:** Do NOT use ${ASK_USER_QUESTION_TOOL_NAME} to ask "Is this plan okay?" or "Should I proceed?" - that's exactly what THIS tool does. ExitPlanMode inherently requests user approval of your plan.
## Examples
1. Initial task: "Search for and understand the implementation of vim mode in the codebase" - Do not use the exit plan mode tool because you are not planning the implementation steps of a task.
2. Initial task: "Help me implement yank mode for vim" - Use the exit plan mode tool after you have finished planning the implementation steps of the task.
3. Initial task: "Add a new feature to handle user authentication" - If unsure about auth method (OAuth, JWT, etc.), use ${ASK_USER_QUESTION_TOOL_NAME} first, then use exit plan mode tool after clarifying the approach.
` `

View File

@@ -34,6 +34,11 @@ import {
type LineEndingType, type LineEndingType,
readFileSyncWithMetadata, readFileSyncWithMetadata,
} from 'src/utils/fileRead.js' } from 'src/utils/fileRead.js'
import {
detectEncoding,
decodeBuffer,
type FileEncoding,
} from 'src/utils/encoding.js'
import { formatFileSize } from 'src/utils/format.js' import { formatFileSize } from 'src/utils/format.js'
import { getFsImplementation } from 'src/utils/fsOperations.js' import { getFsImplementation } from 'src/utils/fsOperations.js'
import { fetchSingleFileGitDiff, type ToolUseDiff } from 'src/utils/gitDiff.js' import { fetchSingleFileGitDiff, type ToolUseDiff } from 'src/utils/gitDiff.js'
@@ -202,13 +207,8 @@ export const FileEditTool = buildTool({
let fileContent: string | null let fileContent: string | null
try { try {
const fileBuffer = await fs.readFileBytes(fullFilePath) const fileBuffer = await fs.readFileBytes(fullFilePath)
const encoding: BufferEncoding = const encoding: FileEncoding = detectEncoding(fileBuffer)
fileBuffer.length >= 2 && fileContent = decodeBuffer(fileBuffer, encoding).replaceAll('\r\n', '\n')
fileBuffer[0] === 0xff &&
fileBuffer[1] === 0xfe
? 'utf16le'
: 'utf8'
fileContent = fileBuffer.toString(encoding).replaceAll('\r\n', '\n')
} catch (e) { } catch (e) {
if (isENOENT(e)) { if (isENOENT(e)) {
fileContent = null fileContent = null
@@ -584,7 +584,7 @@ export const FileEditTool = buildTool({
function readFileForEdit(absoluteFilePath: string): { function readFileForEdit(absoluteFilePath: string): {
content: string content: string
fileExists: boolean fileExists: boolean
encoding: BufferEncoding encoding: FileEncoding
lineEndings: LineEndingType lineEndings: LineEndingType
} { } {
try { try {

View File

@@ -0,0 +1,179 @@
# Feature: 20260510_F001 - multi-encoding-file-tools
## 需求背景
当前文件读写工具FileReadTool、FileWriteTool、FileEditTool的编码检测非常简单——仅通过 BOM 头识别 UTF-8 和 UTF-16LE其他所有情况默认按 UTF-8 处理。对于 GBK/GB2312 等非 BOM 编码文件,读取时会产生乱码,导致 AI 模型无法正确理解和编辑这些文件。
这在中文 Windows 用户场景中尤其常见:许多旧项目、日志文件、配置文件使用 GBK 编码,当前工具链无法处理。
## 目标
- 文件读取时自动检测编码并正确解码,对 AI 模型完全透明(不增加 encoding 参数)
- 文件写入时保持原文件编码,不改变用户的编码习惯
- 覆盖 GBK 编码(最常见非 UTF-8 CJK 编码latin1 作为最终兜底
- 零外部依赖,仅使用 Node.js/Bun 内置的 TextDecoder/TextEncoder
## 范围变更
**仅保留 GBK 编码支持**。Shift_JIS、EUC-JP、EUC-KR、Big5、GB18030、ISO-8859-1 已移出范围。原因:多编码回退链存在字节序列歧义(如 GBK 和 Shift_JIS 共享大量有效字节范围导致误检测。GBK 覆盖了最核心的中文 Windows 用户场景。
## 方案设计
### 架构概述
新增一个独立的编码工具模块 `src/utils/encoding.ts`,提供编码检测和解码/编码函数。现有文件读写路径通过调用此模块实现对非 UTF-8 编码的支持。
```
┌─────────────────────────┐
│ src/utils/encoding.ts │
│ detectEncoding(buffer) │
│ decodeBuffer(buf, enc) │
│ encodeString(str, enc) │
└─────────┬───────────────┘
┌───────────────┼───────────────┐
▼ ▼ ▼
fileRead.ts readFileInRange.ts file.ts
(readFileSync (异步读取路径) (writeTextContent)
WithMetadata)
```
### 编码检测算法(三层检测)
检测基于文件头部 4KB 数据,分三层依次判断:
**第一层BOM 检测(现有逻辑保留)**
- `FF FE` → UTF-16LE
- `EF BB BF` → UTF-8带 BOM
**第二层UTF-8 验证**
-`new TextDecoder('utf-8', { fatal: true })` 对头部 4KB 做解码
- 成功 → 文件为 UTF-8覆盖绝大多数现代源码文件
- 失败(抛出 TypeError→ 进入第三层
**第三层GBK 回退**
-`new TextDecoder('gbk', { fatal: true })` 尝试解码头部 4KB
- 成功 → 文件为 GBK覆盖中文 Windows 用户最常见的非 UTF-8 编码)
- 失败 → `latin1`(单字节编码,永远成功,作为最终兜底)
```typescript
// src/utils/encoding.ts 核心逻辑
export type FileEncoding = BufferEncoding | 'gbk'
export type DetectedEncoding = string
export function detectEncoding(buffer: Buffer): FileEncoding {
// Layer 1: BOM
if (buffer.length >= 2 && buffer[0] === 0xff && buffer[1] === 0xfe) {
return 'utf-16le'
}
if (buffer.length >= 3 && buffer[0] === 0xef && buffer[1] === 0xbb && buffer[2] === 0xbf) {
return 'utf-8'
}
// Layer 2: UTF-8 validation
try {
new TextDecoder('utf-8', { fatal: true }).decode(buffer)
return 'utf-8'
} catch {}
// Layer 3: GBK fallback
try {
new TextDecoder('gbk', { fatal: true }).decode(buffer)
return 'gbk'
} catch {}
return 'latin1'
}
```
### 读取路径改造
#### `src/utils/fileRead.ts` — `detectEncodingForResolvedPath`
将现有的 BOM-only 检测替换为调用 `encoding.ts``detectEncoding` 函数。返回值从 `BufferEncoding` 改为 `FileEncoding``BufferEncoding | 'gbk'`)。
`readFileSyncWithMetadata` 函数先读 raw Buffer再用 `decodeBuffer` 解码,而非使用 `fs.readFileSync` 的 encoding 选项(该选项只接受 `BufferEncoding`,不支持 `gbk`)。
#### `src/utils/readFileInRange.ts` — 异步读取
当前两个路径fast path 和 streaming path都硬编码 `encoding: 'utf8'`
**Fast path 改造**
- `readFile` 改为读取 Buffer去掉 encoding 参数)
- 读取后调用 `detectEncoding(buffer)` 检测编码
-`decodeBuffer` 解码为字符串
- 后续行处理逻辑不变
**Streaming path 改造**
- `createReadStream` 去掉 `encoding: 'utf8'`,改为 Buffer 模式
- 第一个 chunk 做编码检测(同时保留 BOM 剥离逻辑)
- 后续 chunk 拼接后用 `TextDecoder` 解码
- 注意streaming 路径需要特殊处理——先收集足够字节做检测,再逐行扫描
**Streaming 编码处理策略**
streaming 路径改为两阶段:
1. **检测阶段**:前 4KB 数据到达后立即检测编码
2. **解码阶段**:用检测到的编码创建一个 `TextDecoder``{ stream: true }` 模式),逐 chunk 解码
### 写入路径改造
#### 编码回写策略
写入时需要将内部 UTF-8 字符串编码回原文件编码。由于 `TextEncoder` 只支持 UTF-8 输出,需要使用 `TextDecoder` 的反向操作。
**最终决定**:对于非 UTF-8 文件的写回,尝试使用 `Buffer.from(content, encoding)` 编码,失败则自动转换为 UTF-8 并在结果消息中注明。这样既满足了零依赖约束,也避免了数据损坏。
#### `src/utils/file.ts` — `writeTextContent`
现有函数签名 `writeTextContent(filePath, content, encoding, lineEndings)` 已接受 encoding 参数。需要:
- 扩展类型,接受 `FileEncoding` 而非仅 `BufferEncoding`
- 对于 UTF-8 和 UTF-16LE行为不变
- 对于 GBK使用 `encodeString` 函数尝试编码,失败则回退为 UTF-8 写入
#### `FileWriteTool` 和 `FileEditTool`
这两个工具的 `call` 方法中,`writeTextContent` 调用已传递 `encoding`(来自 `readFileSyncWithMetadata` 的返回值)。改动很小——只需确保类型系统接受新编码名。
### 类型扩展
```typescript
// 扩展编码类型 — 仅添加 GBK
export type FileEncoding = BufferEncoding | 'gbk'
```
`readFileSyncWithMetadata` 返回类型中将 `encoding``BufferEncoding` 改为 `FileEncoding`
## 实现要点
### 关键技术决策
1. **检测只用头部 4KB**:避免全文件扫描,性能开销极小(多几次 TextDecoder 调用,每次 ~1μs
2. **GBK 作为唯一回退**:中文 Windows 用户最多,且避免了多编码回退链的字节序列歧义问题
3. **TextDecoder fatal 模式**`{ fatal: true }` 是检测的关键——如果字节序列不符合编码规范会抛异常,借此区分不同编码
4. **streaming 路径的两阶段设计**:先攒够检测数据再开始行扫描,避免半字符解码问题
5. **latin1 最终兜底**:单字节编码永远成功,确保任何文件都能被读取
### 难点
1. **Streaming 编码解码**`TextDecoder` 支持 `{ stream: true }` 模式处理多字节字符的 chunk 边界,但需要在检测完成前缓冲数据
2. **编码回写的零依赖方案**`TextEncoder` 只输出 UTF-8非 UTF-8 编码回写需要额外处理。务实方案是 UTF-8 写入 + 消息提示
3. **混合编码文件**:极少见,不在本次覆盖范围内
### 依赖
- 零外部依赖,仅使用 `TextDecoder`Node.js 13+ / Bun 内置 full-icu
- Bun 运行时对 GBK 的 TextDecoder 支持已验证可用Bun 1.3.13
## 验收标准
- [x] FileReadTool 能正确读取 GBK 编码的中文文本文件,显示正确的中文内容
- [x] FileReadTool 能正确读取 UTF-8 文件(行为不变,回归测试通过)
- [x] FileReadTool 能正确读取 UTF-16LE 文件(行为不变)
- [x] FileEditTool 能编辑 GBK 文件并写回,内容不乱码
- [x] FileWriteTool 编辑 GBK 文件后写回,编码保持或合理转换
- [x] readFileInRange 的 fast path 路径支持非 UTF-8 编码
- [x] readFileInRange 的 streaming path 支持非 UTF-8 编码
- [x] 编码检测性能4KB 数据检测耗时 < 1ms
- [x] `bun run precheck` typecheck + lint + 相关测试零错误
- [x] 新增编码相关单元测试覆盖检测和解码逻辑

View File

@@ -0,0 +1,161 @@
# 多编码文件工具 人工验收清单
**生成时间:** 2026-05-10
**关联计划:** spec/feature_20260510_F001_multi-encoding-file-tools/spec-plan.md
**关联设计:** spec/feature_20260510_F001_multi-encoding-file-tools/spec-design.md
---
所有验收项均可通过 Shell 命令自动化验证,无需人类参与。仍将生成清单用于自动执行。
**范围变更:** 仅保留 GBK 编码支持Shift_JIS/EUC-JP/EUC-KR/Big5/GB18030 已移除。
---
## 验收前准备
### 环境要求
- [x] [AUTO] 检查 Bun 运行时版本: `bun --version`
- [x] [AUTO] 安装依赖: `bun install`
### 测试数据准备
- [x] [AUTO] 创建 GBK 编码测试文件: `bun -e "const fs = require('fs'); const b = Buffer.from([0xC4, 0xE3, 0xBA, 0xC3, 0xCA, 0xC0, 0xBD, 0xE7, 0x0A]); fs.writeFileSync('/tmp/test-gbk.txt', b)"`
- [x] [AUTO] 创建 UTF-8 测试文件: `bun -e "require('fs').writeFileSync('/tmp/test-utf8.txt', 'Hello 世界\n')"`
- [x] [AUTO] 创建 UTF-16LE 测试文件: `bun -e "const fs = require('fs'); const b = Buffer.from('Hello','utf16le'); fs.writeFileSync('/tmp/test-utf16le.txt', b)"`
---
## 验收项目
### 场景 1读取 GBK 编码文件(中文场景)
**用户目标:** 用户有一个 GBK 编码的中文文件,通过 FileReadTool 读取后看到正确的中文内容
**触发路径:**
1. 系统检测到非 UTF-8 字节序列
2. 编码回退识别为 GBK
3. 用 GBK 解码输出中文文本
#### - [x] 1.1 GBK 文件同步读取
- **来源:** spec-plan-acceptance.md §2 / spec-design.md §验收标准
- **目的:** 确认 GBK 文件读取解码正确
- **操作步骤:**
1. [A] `bun -e "import { readFileSyncWithMetadata } from './src/utils/fileRead.js'; const r = readFileSyncWithMetadata('/tmp/test-gbk.txt'); console.log('encoding:', r.encoding); console.log('content:', r.content)"` → 期望包含: `你好世界`
2. [A] 上条命令输出 encoding 字段 → 期望包含: `gbk`
#### - [x] 1.2 GBK 文件异步路径读取
- **来源:** spec-plan-acceptance.md §6 / spec-design.md §验收标准
- **目的:** 确认 readFileInRange fast path 支持 GBK
- **操作步骤:**
1. [A] `bun -e "import { readFileInRange } from './src/utils/readFileInRange.js'; const r = await readFileInRange('/tmp/test-gbk.txt', 0); console.log('content:', r.content); console.log('totalLines:', r.totalLines)"` → 期望包含: `你好世界`
2. [A] 上条命令输出 totalLines → 期望包含: `1`
---
### 场景 3写入非 UTF-8 编码文件
**用户目标:** 用户通过 FileEditTool/FileWriteTool 编辑 GBK 文件后写回,内容不损坏
**触发路径:**
1. 系统检测原文件编码
2. 编辑内容后写回
3. 非标准编码回退为 UTF-8 写入(零依赖约束)
#### - [x] 3.1 GBK 文件写入UTF-8 回退)
- **来源:** spec-plan-acceptance.md §7 / spec-design.md §写入路径改造
- **目的:** 确认非 UTF-8 编码写入不损坏内容
- **操作步骤:**
1. [A] `bun -e "import { writeTextContent } from './src/utils/file.js'; writeTextContent('/tmp/test-gbk-write.txt', '测试写入', 'gbk', 'LF'); const fs = require('fs'); const content = fs.readFileSync('/tmp/test-gbk-write.txt', 'utf8'); console.log('written:', content)"` → 期望包含: `测试写入`
---
### 场景 4UTF-8 文件读取回归
**用户目标:** 用户读取 UTF-8 文件,行为与改动前完全一致
**触发路径:**
1. UTF-8 fatal 验证通过
2. 内容正常输出
#### - [x] 4.1 UTF-8 文件读取回归
- **来源:** spec-plan-acceptance.md §4 / spec-design.md §验收标准
- **目的:** 确认 UTF-8 读取无回归
- **操作步骤:**
1. [A] `bun -e "import { readFileSyncWithMetadata } from './src/utils/fileRead.js'; const r = readFileSyncWithMetadata('/tmp/test-utf8.txt'); console.log('encoding:', r.encoding); console.log('content:', r.content)"` → 期望包含: `Hello 世界`
2. [A] 上条命令输出 encoding 字段 → 期望包含: `utf`
---
### 场景 5UTF-16LE 文件读取回归
**用户目标:** 用户读取 UTF-16LEBOM文件行为与改动前完全一致
**触发路径:**
1. BOM 检测层识别 FF FE 标记
2. 用 UTF-16LE 解码
#### - [x] 5.1 UTF-16LE 文件读取回归
- **来源:** spec-plan-acceptance.md §5 / spec-design.md §验收标准
- **目的:** 确认 UTF-16LE BOM 读取无回归
- **操作步骤:**
1. [A] `bun -e "import { readFileSyncWithMetadata } from './src/utils/fileRead.js'; const r = readFileSyncWithMetadata('/tmp/test-utf16le.txt'); console.log('encoding:', r.encoding); console.log('content:', r.content)"` → 期望包含: `utf-16le`
2. [A] 上条命令输出 content 字段 → 期望包含: `Hello`
---
### 场景 6编码检测性能
**用户目标:** 编码检测不应影响文件读取的响应速度
**触发路径:**
1. 对 4KB 数据执行 1000 次检测
2. 验证平均耗时 < 1ms
#### - [x] 6.1 检测性能基准
- **来源:** spec-plan-acceptance.md §8 / spec-design.md §实现要点
- **目的:** 确认编码检测性能达标
- **操作步骤:**
1. [A] `bun -e "import { detectEncoding } from './src/utils/encoding.js'; const buf = Buffer.alloc(4096, 0x41); const start = performance.now(); for (let i = 0; i < 1000; i++) detectEncoding(buf); const avg = (performance.now() - start) / 1000; console.log('avg:', avg, 'ms'); process.exit(avg < 1 ? 0 : 1)"` → 期望包含: `avg:`
---
### 场景 7构建和测试完整性
**用户目标:** 整体代码质量无退化,所有测试通过
**触发路径:**
1. 执行完整 prechecktypecheck + lint + test
2. 确认零错误
#### - [x] 7.1 编码相关单元测试
- **来源:** spec-plan.md Task 1-4 检查步骤 / spec-design.md §验收标准
- **目的:** 确认编码相关测试全部通过
- **操作步骤:**
1. [A] `bun test src/utils/__tests__/encoding.test.ts` → 期望包含: `0 fail`
2. [A] `bun test src/utils/__tests__/fileRead.test.ts` → 期望包含: `0 fail`
3. [A] `bun test src/utils/__tests__/readFileInRange.test.ts` → 期望包含: `0 fail`
4. [A] `bun test src/utils/__tests__/file.test.ts` → 期望包含: `0 fail`
---
## 验收后清理
- [x] [AUTO] 清理临时测试文件: `rm -f /tmp/test-gbk.txt /tmp/test-utf8.txt /tmp/test-utf16le.txt /tmp/test-gbk-write.txt`
---
## 验收结果汇总
| 场景 | 序号 | 验收项 | [A] | [H] | 结果 |
|------|------|--------|-----|-----|------|
| 场景 1 | 1.1 | GBK 同步读取 | 2 | 0 | ✅ |
| 场景 1 | 1.2 | GBK 异步路径读取 | 2 | 0 | ✅ |
| 场景 3 | 3.1 | GBK 写入(回退) | 1 | 0 | ✅ |
| 场景 4 | 4.1 | UTF-8 回归 | 2 | 0 | ✅ |
| 场景 5 | 5.1 | UTF-16LE 回归 | 2 | 0 | ✅ |
| 场景 6 | 6.1 | 检测性能 | 1 | 0 | ✅ |
| 场景 7 | 7.1 | 编码单元测试 | 4 | 0 | ✅ |
**验收结论:** ✅ 全部通过

View File

@@ -0,0 +1,47 @@
### Acceptance Task: 多编码文件工具验收
**前置条件:**
- 所有 Task 0-4 已执行完毕
- 运行环境: 当前开发环境Bun
**范围变更:** 仅保留 GBK 编码支持Shift_JIS/EUC-JP/EUC-KR/Big5/GB18030/ISO-8859-1 已移除。
**端到端验证:**
1. 运行完整测试套件确保无回归
- `bun run precheck`
- 预期: typecheck + lint fix + test 全部零错误通过
- 失败排查: 检查各 Task 的测试步骤,特别是 Task 1 的编码检测测试和 Task 3 的 readFileInRange 测试
2. 验证 GBK 文件读取正确性
- 创建 GBK 编码测试文件:`bun -e "const fs = require('fs'); const b = Buffer.from([0xC4, 0xE3, 0xBA, 0xC3, 0xCA, 0xC0, 0xBD, 0xE7, 0x0A]); fs.writeFileSync('/tmp/test-gbk.txt', b)"`
- 读取并验证:`bun -e "import { readFileSyncWithMetadata } from './src/utils/fileRead.js'; const r = readFileSyncWithMetadata('/tmp/test-gbk.txt'); console.log('encoding:', r.encoding); console.log('content:', r.content)"`
- 预期: encoding 为 `gbk`content 为 "你好世界"
- 失败排查: 检查 Task 1 的 detectEncoding 逻辑、Task 2 的 readFileSyncWithMetadata 集成
3. 验证 UTF-8 文件读取回归
- `bun -e "import { readFileSyncWithMetadata } from './src/utils/fileRead.js'; const fs = require('fs'); fs.writeFileSync('/tmp/test-utf8.txt', 'Hello 世界\n'); const r = readFileSyncWithMetadata('/tmp/test-utf8.txt'); console.log('encoding:', r.encoding); console.log('content:', r.content)"`
- 预期: encoding 为 `utf-8`content 为 "Hello 世界"
- 失败排查: 检查 Task 1 的 UTF-8 fatal 验证逻辑
4. 验证 UTF-16LE 文件读取回归
- `bun -e "const fs = require('fs'); const b = Buffer.concat([Buffer.from([0xFF, 0xFE]), Buffer.from('Hello', 'utf16le')]); fs.writeFileSync('/tmp/test-utf16le.txt', b); import { readFileSyncWithMetadata } from './src/utils/fileRead.js'; const r = readFileSyncWithMetadata('/tmp/test-utf16le.txt'); console.log('encoding:', r.encoding); console.log('content:', r.content)"`
- 预期: encoding 为 `utf-16le`content 为 "Hello"
- 失败排查: 检查 Task 1 的 BOM 检测层、Task 2 的集成
5. 验证 readFileInRange 异步路径的 GBK 支持
- `bun -e "import { readFileInRange } from './src/utils/readFileInRange.js'; const r = await readFileInRange('/tmp/test-gbk.txt', 0); console.log('content:', r.content); console.log('totalLines:', r.totalLines)"`
- 预期: content 为 "你好世界"totalLines 为 1
- 失败排查: 检查 Task 3 的 fast path 改造
6. 验证 GBK 文件写入UTF-8 回退)
- `bun -e "import { writeTextContent } from './src/utils/file.js'; writeTextContent('/tmp/test-gbk-write.txt', '测试写入', 'gbk', 'LF'); const fs = require('fs'); const content = fs.readFileSync('/tmp/test-gbk-write.txt', 'utf8'); console.log('written:', content)"`
- 预期: 文件成功写入,内容为 "测试写入"UTF-8 回退或 GBK 编码均可接受)
- 失败排查: 检查 Task 4 的 writeTextContent 改造和 encodeString 函数
7. 验证编码检测性能
- `bun -e "import { detectEncoding } from './src/utils/encoding.js'; const buf = Buffer.alloc(4096, 0x41); const start = performance.now(); for (let i = 0; i < 1000; i++) detectEncoding(buf); console.log('avg:', (performance.now() - start) / 1000, 'ms')"`
- 预期: 平均检测耗时 < 1ms
- 失败排查: 检查 Task 1 的检测逻辑是否有不必要的重复操作
---

View File

@@ -0,0 +1,34 @@
### Task 0: 环境准备
**背景:**
确保构建和测试工具链在当前开发环境中可用,验证 Bun 运行时对 GBK 编码的 TextDecoder 支持情况。
**涉及文件:**
- 无文件修改,仅验证环境
**执行步骤:**
- [x] 验证 Bun 运行时可用
- 运行命令: `bun --version`
- 预期: 输出 Bun 版本号
- [x] 验证 TypeScript 编译无错误
- 运行命令: `bunx tsc --noEmit 2>&1 | tail -5`
- 预期: 无错误输出(或仅有已知的 pre-existing 错误)
- [x] 验证 Bun 对 GBK 编码的 TextDecoder 支持
- 运行命令: `bun -e "const d = new TextDecoder('gbk', { fatal: true }); const buf = Buffer.from([0xC4, 0xE3, 0xBA, 0xC3]); console.log(d.decode(buf))"`
- 预期: 输出 "你好"GBK 编码的中文字符)
- [x] 验证测试框架可用
- 运行命令: `bun test src/utils/__tests__/hash.test.ts 2>&1 | tail -3`
- 预期: 测试运行成功,无框架错误
**检查步骤:**
- [x] Bun 版本确认
- `bun --version`
- 预期: 输出有效版本号
- [x] GBK 编码支持确认
- `bun -e "console.log(new TextDecoder('gbk').decode(Buffer.from([0xC4, 0xE3, 0xBA, 0xC3])))"`
- 预期: 输出 "你好"
- [x] 现有测试通过
- `bun test src/utils/__tests__/file.test.ts 2>&1 | tail -3`
- 预期: 所有测试通过
---

View File

@@ -0,0 +1,141 @@
### Task 1: 编码检测核心模块
**背景:**
当前 `src/utils/fileRead.ts``detectEncodingForResolvedPath` 仅通过 BOM 头识别 UTF-8 和 UTF-16LE其他所有文件一律返回 `utf8`,导致 GBK 等非 UTF-8 编码文件读取乱码。本 Task 新建独立的编码检测工具模块 `src/utils/encoding.ts`实现三层编码检测算法BOM → UTF-8 fatal 验证 → GBK 回退),为后续 Task 2/3/4 的读写路径改造提供统一的编码检测和解码能力。本 Task 无前置依赖,是后续所有 Task 的基础。
**涉及文件:**
- 新建: `src/utils/encoding.ts`
- 新建: `src/utils/__tests__/encoding.test.ts`
**执行步骤:**
- [x] 创建 `src/utils/encoding.ts`,定义类型
- 位置: 文件顶部
- 导出以下类型:
```typescript
/** 扩展编码类型,覆盖最常见的非 UTF-8 CJK 编码 */
export type FileEncoding = BufferEncoding | 'gbk'
/** TextDecoder 接受的编码名string比 FileEncoding 更宽泛 */
export type DetectedEncoding = string
```
- 原因: 后续 Task 2/3/4 需要这些类型来做编码标注和类型收窄
- [x] 实现 `detectEncoding(buffer: Buffer): FileEncoding` 函数
- 位置: `src/utils/encoding.ts`,类型定义之后
- 三层检测逻辑:
```typescript
export function detectEncoding(buffer: Buffer): FileEncoding {
// Layer 1: BOM 检测(与现有 fileRead.ts 逻辑一致)
if (buffer.length >= 2 && buffer[0] === 0xff && buffer[1] === 0xfe) {
return 'utf-16le'
}
if (
buffer.length >= 3 &&
buffer[0] === 0xef &&
buffer[1] === 0xbb &&
buffer[2] === 0xbf
) {
return 'utf-8'
}
// Layer 2: UTF-8 fatal 验证
// fatal: true 模式下,无效 UTF-8 字节序列会抛出 TypeError
try {
new TextDecoder('utf-8', { fatal: true }).decode(buffer)
return 'utf-8'
} catch {
// 不是合法 UTF-8进入 Layer 3
}
// Layer 3: GBK 回退
try {
new TextDecoder('gbk', { fatal: true }).decode(buffer)
return 'gbk'
} catch {
// 不是合法 GBKlatin1 作为最终兜底
}
return 'latin1'
}
```
- 原因: BOM 必须优先于 fatal 验证GBK 作为唯一回退避免了多编码链的字节歧义问题latin1 单字节编码永远成功
- [x] 实现 `decodeBuffer(buffer: Buffer, encoding: DetectedEncoding): string` 函数
- 位置: `src/utils/encoding.ts``detectEncoding` 之后
- 逻辑:
```typescript
export function decodeBuffer(
buffer: Buffer,
encoding: DetectedEncoding,
): string {
return new TextDecoder(encoding).decode(buffer)
}
```
- 原因: 统一解码入口,后续 Task 2/3 的读取路径都调用此函数
- [x] 实现 `encodeString(content: string, encoding: DetectedEncoding): { buffer: Buffer; converted: boolean }` 函数
- 位置: `src/utils/encoding.ts``decodeBuffer` 之后
- 逻辑:
```typescript
export function encodeString(
content: string,
encoding: DetectedEncoding,
): { buffer: Buffer; converted: boolean } {
if (encoding === 'utf-8' || encoding === 'utf8') {
return { buffer: Buffer.from(content, 'utf-8'), converted: false }
}
if (encoding === 'utf-16le') {
return { buffer: Buffer.from(content, 'utf-16le'), converted: false }
}
// 其他编码(如 gbk尝试 Buffer.from失败则回退为 UTF-8
try {
const buf = Buffer.from(content, encoding as BufferEncoding)
return { buffer: buf, converted: false }
} catch {
return { buffer: Buffer.from(content, 'utf-8'), converted: true }
}
}
```
- 原因: `Buffer.from` 在 Bun 中可能支持 GBK 编码名,但 Node.js 不支持。try-catch 策略兼容两种运行时;`converted` 标志让 Task 4 的写入路径能向用户报告编码转换
- [x] 为编码检测和解码函数编写单元测试
- 测试文件: `src/utils/__tests__/encoding.test.ts`
- 测试场景:
- **BOM 检测 — UTF-16LE**: 输入 `Buffer.from([0xff, 0xfe, 0x48, 0x00])` → 预期返回 `'utf-16le'`
- **BOM 检测 — UTF-8 BOM**: 输入 `Buffer.from([0xef, 0xbb, 0xbf, 0x48, 0x65])` → 预期返回 `'utf-8'`
- **UTF-8 验证**: 输入 `Buffer.from('Hello, 世界', 'utf-8')` → 预期返回 `'utf-8'`
- **GBK 检测**: 输入 `Buffer.from([0xc4, 0xe3, 0xba, 0xc3])` → 预期返回 `'gbk'`
- **空 buffer**: 输入 `Buffer.alloc(0)` → 预期返回 `'utf-8'`
- **latin1 兜底**: 输入随机字节 `Buffer.from([0x80, 0x81, 0x82, 0x83, 0x84, 0x85])` → 预期返回 `'latin1'`
- **BOM 优先于内容分析**: 输入带 UTF-8 BOM 的数据 → 预期返回 `'utf-8'`
- **decodeBuffer — UTF-8**: 输入 UTF-8 编码的 buffer + encoding `'utf-8'` → 预期返回正确的中文字符串
- **decodeBuffer — GBK**: 输入 GBK 编码的 buffer + encoding `'gbk'` → 预期返回正确的中文字符串
- **decodeBuffer — UTF-16LE**: 输入 UTF-16LE 编码的 buffer + encoding `'utf-16le'` → 预期返回正确字符串
- **decodeBuffer — 空 buffer**: 输入空 buffer → 预期返回空字符串
- **encodeString — UTF-8**: 输入字符串 + encoding `'utf-8'` → 预期 `{ converted: false }`
- **encodeString — utf8 别名**: 输入字符串 + encoding `'utf8'` → 预期 `{ converted: false }`
- **encodeString — UTF-16LE**: 输入字符串 + encoding `'utf-16le'` → 预期 `{ converted: false }`
- **encodeString — GBK**: 输入字符串 + encoding `'gbk'` → 预期返回有效的 Bufferconverted 视运行时而定)
- 运行命令: `bun test src/utils/__tests__/encoding.test.ts`
- 预期: 所有测试通过
**检查步骤:**
- [x] 验证 `encoding.ts` 文件存在且导出正确
- `grep -c "export" src/utils/encoding.ts`
- 预期: 输出 >= 4至少导出 FileEncoding, DetectedEncoding, detectEncoding, decodeBuffer, encodeString 共 5 个导出)
- [x] 验证类型检查通过
- `bunx tsc --noEmit src/utils/encoding.ts 2>&1 | head -5`
- 预期: 无类型错误输出
- [x] 运行编码检测单元测试
- `bun test src/utils/__tests__/encoding.test.ts`
- 预期: 所有测试通过,无失败用例
**认知变更:**
- [x] [CLAUDE.md] `src/utils/encoding.ts` 是文件编码检测的唯一入口,提供 `detectEncoding`三层检测BOM → UTF-8 fatal → GBK 回退)和 `decodeBuffer`/`encodeString` 函数。检测基于文件头部 4KB零外部依赖仅使用 TextDecoder API。`FileEncoding` 类型为 `BufferEncoding | 'gbk'`,覆盖最常见非 UTF-8 CJK 编码。latin1 作为最终兜底编码(单字节编码永远成功)。
---

View File

@@ -0,0 +1,163 @@
### Task 2: 同步读取路径集成
**背景:**
当前同步读取路径(`fileRead.ts``file.ts``fileReadCache.ts`)的编码检测仅通过 BOM 头识别 UTF-8 和 UTF-16LE非 BOM 编码文件一律按 UTF-8 读取导致乱码。本 Task 将 `detectEncodingForResolvedPath` 的内部实现从 BOM-only 升级为调用 Task 1 创建的 `encoding.ts` 三层检测,并将返回类型从 `BufferEncoding` 扩展为 `FileEncoding`。同时将所有 `fs.readFileSync(path, { encoding })` 调用改为先读 Buffer 再用 `decodeBuffer` 解码,以支持 `gbk` 等非 `BufferEncoding` 编码。本 Task 依赖 Task 1`src/utils/encoding.ts`),输出被 Task 4写入路径适配依赖。
**涉及文件:**
- 修改: `src/utils/fileRead.ts`
- 修改: `src/utils/file.ts`
- 修改: `src/utils/fileReadCache.ts`
- 新建: `src/utils/__tests__/fileRead.test.ts`
**执行步骤:**
- [x]`fileRead.ts` 中导入 `encoding.ts` 的类型和函数
- 位置: `src/utils/fileRead.ts` 文件顶部 import 区域,在 `import { getFsImplementation, safeResolvePath } from './fsOperations.js'` 之后
- 添加导入:
```typescript
import { type FileEncoding, decodeBuffer, detectEncoding } from './encoding.js'
```
- 原因: 后续步骤需要 `FileEncoding` 类型、`detectEncoding` 检测函数和 `decodeBuffer` 解码函数
- [x] 改造 `detectEncodingForResolvedPath` 函数,使用 `encoding.ts` 的三层检测
- 位置: `src/utils/fileRead.ts` 的 `detectEncodingForResolvedPath` 函数
- 将函数体替换为以下逻辑:
```typescript
export function detectEncodingForResolvedPath(
resolvedPath: string,
): FileEncoding {
const { buffer, bytesRead } = getFsImplementation().readSync(resolvedPath, {
length: 4096,
})
// Empty files default to utf8 — nothing to detect
if (bytesRead === 0) {
return 'utf8'
}
return detectEncoding(buffer.subarray(0, bytesRead))
}
```
- 关键变更:
- 返回类型从 `BufferEncoding` 改为 `FileEncoding`
- 删除内联的 BOM 检测逻辑,改为调用 `detectEncoding(buffer.subarray(0, bytesRead))`
- 使用 `buffer.subarray(0, bytesRead)` 截取实际读取的字节,避免尾部零字节干扰检测
- 原因: 将检测逻辑委托给 `encoding.ts` 的三层算法,消除代码重复
- [x] 改造 `readFileSyncWithMetadata` 函数,支持非 `BufferEncoding` 解码
- 位置: `src/utils/fileRead.ts` 的 `readFileSyncWithMetadata` 函数
- 将函数签名和内部逻辑改为:
```typescript
export function readFileSyncWithMetadata(filePath: string): {
content: string
encoding: FileEncoding
lineEndings: LineEndingType
} {
const fs = getFsImplementation()
const { resolvedPath, isSymlink } = safeResolvePath(fs, filePath)
if (isSymlink) {
logForDebugging(`Reading through symlink: ${filePath} -> ${resolvedPath}`)
}
const encoding = detectEncodingForResolvedPath(resolvedPath)
// Read raw Buffer first — fs.readFileSync encoding option only accepts
// BufferEncoding, not gbk etc.
const rawBuffer = fs.readFileBytesSync(resolvedPath)
const raw = decodeBuffer(rawBuffer, encoding)
const lineEndings = detectLineEndingsForString(raw.slice(0, 4096))
return {
content: raw.replaceAll('\r\n', '\n'),
encoding,
lineEndings,
}
}
```
- 关键变更:
- 返回类型中 `encoding` 从 `BufferEncoding` 改为 `FileEncoding`
- `fs.readFileSync(resolvedPath, { encoding })` 改为 `fs.readFileBytesSync(resolvedPath)` 读取 Buffer
- 新增 `decodeBuffer(rawBuffer, encoding)` 解码为字符串
- 原因: `fs.readFileSync` 的 `encoding` 选项只接受 `BufferEncoding`utf8/utf16le/latin1 等),传入 `'gbk'` 会在运行时报错
- [x] 更新 `file.ts` 中 `detectFileEncoding` 的返回类型
- 位置: `src/utils/file.ts` 的 `detectFileEncoding` 函数签名
- 将 `): BufferEncoding {` 改为 `): FileEncoding {`
- 在文件顶部 import 区域添加:
```typescript
import { type FileEncoding, decodeBuffer, encodeString } from './encoding.js'
```
- 原因: `detectFileEncoding` 调用 `detectEncodingForResolvedPath`,返回类型已改为 `FileEncoding`
- [x] 更新 `file.ts` 中 `detectLineEndings` 的 encoding 参数类型和解码逻辑
- 位置: `src/utils/file.ts` 的 `detectLineEndings` 函数
- 将函数签名改为:
```typescript
export function detectLineEndings(
filePath: string,
encoding: FileEncoding = 'utf8',
): LineEndingType {
```
- 将内部 `buffer.toString(encoding, 0, bytesRead)` 改为:
```typescript
const content = decodeBuffer(buffer.subarray(0, bytesRead), encoding)
```
- 原因: `buffer.toString('gbk')` 不可靠,统一使用 `decodeBuffer` 通过 `TextDecoder` 解码
- [x] 更新 `fileReadCache.ts` 的类型和解码逻辑
- 位置: `src/utils/fileReadCache.ts`
- 在文件顶部 import 区域添加:
```typescript
import { type FileEncoding, decodeBuffer } from './encoding.js'
```
- 将 `CachedFileData` 类型中 `encoding: BufferEncoding` 改为 `encoding: FileEncoding`
- 将 `readFile` 方法返回类型改为 `{ content: string; encoding: FileEncoding }`
- 将缓存未命中读取逻辑改为:
```typescript
const encoding = detectFileEncoding(filePath)
const rawBuffer = fs.readFileBytesSync(filePath)
const content = decodeBuffer(rawBuffer, encoding).replaceAll('\r\n', '\n')
```
- 原因: 与 `fileRead.ts` 相同——必须改为 Buffer 读取 + `decodeBuffer` 解码
- [x] 为改造后的 `detectEncodingForResolvedPath` 和 `readFileSyncWithMetadata` 编写单元测试
- 测试文件: `src/utils/__tests__/fileRead.test.ts`
- 测试场景:
- **UTF-8 文件读取**: 创建临时 UTF-8 文件 → 返回 `encoding: 'utf-8'`content 与写入内容一致
- **GBK 文件读取**: 创建临时 GBK 编码文件 → 返回 `encoding: 'gbk'`content 包含正确的中文字符
- **空文件读取**: 创建空文件 → 返回 `encoding: 'utf8'`content 为空字符串
- **UTF-16LE BOM 文件读取**: 创建带 BOM 的 UTF-16LE 文件 → 返回 `encoding: 'utf-16le'`
- **detectEncodingForResolvedPath 返回类型**: 验证返回值为 `FileEncoding` 类型
- Mock 策略: 使用 `tests/mocks/debug.ts` mock `debug.ts`,使用 `tests/mocks/log.ts` mock `log.ts`
- 运行命令: `bun test src/utils/__tests__/fileRead.test.ts`
- 预期: 所有测试通过
**检查步骤:**
- [x] 验证 `fileRead.ts` 的导入和返回类型已更新
- `grep -n "FileEncoding\|decodeBuffer\|detectEncoding" src/utils/fileRead.ts`
- 预期: 输出包含 import 行中的 `FileEncoding`、`decodeBuffer`,以及函数体中的 `detectEncoding` 调用
- [x] 验证 `file.ts` 的类型已更新
- `grep -n "FileEncoding\|decodeBuffer" src/utils/file.ts`
- 预期: `detectFileEncoding` 返回 `FileEncoding``detectLineEndings` 参数类型为 `FileEncoding`
- [x] 验证 `fileReadCache.ts` 的类型已更新
- `grep -n "FileEncoding\|decodeBuffer" src/utils/fileReadCache.ts`
- 预期: `CachedFileData` 和 `readFile` 返回类型使用 `FileEncoding`
- [x] 验证 `fileRead.ts` 中不再有内联 BOM 检测逻辑
- `grep -c "0xff\|0xfe\|0xef\|0xbb\|0xbf" src/utils/fileRead.ts`
- 预期: 输出为 0
- [x] 运行 fileRead 单元测试
- `bun test src/utils/__tests__/fileRead.test.ts`
- 预期: 所有测试通过
- [x] 运行 precheck 确认无类型/lint/测试错误
- `bun run precheck`
- 预期: 零错误通过
**认知变更:**
- [x] [CLAUDE.md] `fs.readFileSync(path, { encoding })` 的 `encoding` 选项只接受 `BufferEncoding`utf8/utf16le/latin1/ascii/binary/hex/base64/ucs2/utf16le不支持 `gbk` 等 ICU 编码名。读取非 UTF-8 文件时必须先 `fs.readFileSync(path)` 读 Buffer再用 `TextDecoder` 解码。项目中所有文件读取路径fileRead.ts、fileReadCache.ts、file.ts已统一使用 `decodeBuffer` 函数处理此逻辑。
---

View File

@@ -0,0 +1,161 @@
### Task 3: 异步读取路径改造
**背景:**
当前 `src/utils/readFileInRange.ts` 是 FileReadTool 的核心异步读取函数,提供 fast path小文件整体读入和 streaming path大文件逐块扫描两条路径两者均硬编码 `encoding: 'utf8'`,导致非 UTF-8 编码文件读取乱码。本 Task 将两条路径改造为 Buffer 读取 + 编码检测 + TextDecoder 解码模式。fast path 改造简单(整体读 Buffer 后检测解码streaming path 需要两阶段设计(先收集前 4KB 做编码检测,再用 `TextDecoder({ stream: true })` 逐 chunk 解码)。本 Task 依赖 Task 1`src/utils/encoding.ts``detectEncoding``decodeBuffer`),输出被 Task 4 依赖(通过 `readFileInRange` 的返回值间接影响)。
**涉及文件:**
- 修改: `src/utils/readFileInRange.ts`
- 新建: `src/utils/__tests__/readFileInRange.test.ts`
**执行步骤:**
- [x]`readFileInRange.ts` 中导入 `encoding.ts` 的函数
- 位置: `src/utils/readFileInRange.ts` 文件顶部 import 区域,在 `import { formatFileSize } from './format.js'` 之后
- 添加导入:
```typescript
import { detectEncoding, decodeBuffer } from './encoding.js'
```
- 原因: fast path 和 streaming path 都需要 `detectEncoding` 做编码检测fast path 需要 `decodeBuffer` 做一次性解码
- [x] 改造 fast path — 将 `readFile` 从 UTF-8 字符串读取改为 Buffer 读取 + 检测 + 解码
- 位置: `src/utils/readFileInRange.ts` 的 `readFileInRange` 函数内 fast path 分支
- 将以下代码:
```typescript
const text = await readFile(filePath, { encoding: 'utf8', signal })
return readFileInRangeFast(text, stats.mtimeMs, offset, maxLines, ...)
```
替换为:
```typescript
const rawBuffer = await readFile(filePath, { signal })
const encoding = detectEncoding(rawBuffer)
const text = decodeBuffer(rawBuffer, encoding)
return readFileInRangeFast(text, stats.mtimeMs, offset, maxLines, ...)
```
- 关键变更: `readFile` 去掉 `encoding: 'utf8'` 选项,返回 `Buffer`;调用 `detectEncoding(rawBuffer)` 检测编码;调用 `decodeBuffer(rawBuffer, encoding)` 解码为字符串。
- 原因: `readFile` 的 `encoding` 选项只支持 `BufferEncoding`,不支持 `gbk` 等 ICU 编码名
- [x] 改造 streaming path — 扩展 `StreamState` 类型,增加编码检测和解码相关字段
- 位置: `src/utils/readFileInRange.ts` 的 `StreamState` 类型定义
- 在现有字段之后添加以下字段:
```typescript
type StreamState = {
// ... 现有字段保持不变 ...
/** 编码检测状态null 表示尚未检测string 表示已检测完成 */
encoding: string | null
/** TextDecoder 实例:检测完成后创建,用于逐 chunk 流式解码 */
decoder: TextDecoder | null
/** 检测阶段缓冲区:收集原始字节直到满 4KB 或 stream 结束 */
detectionBuffer: number[]
}
```
- 原因: streaming 模式下 chunk 是增量到达的,需要缓冲阶段收集足够字节来调用 `detectEncoding`
- [x] 改造 `streamOnData` — 处理 Buffer chunk实现两阶段检测阶段 + 解码阶段)
- 位置: `src/utils/readFileInRange.ts` 的 `streamOnData` 函数
- 将函数签名从 `streamOnData(this: StreamState, chunk: string): void` 改为 `streamOnData(this: StreamState, chunk: Buffer): void`
- 替换函数体为两阶段逻辑:
```typescript
function streamOnData(this: StreamState, chunk: Buffer): void {
this.totalBytesRead += chunk.length
// ... maxBytes 检查保持不变 ...
// Phase 1: 编码检测阶段
if (this.encoding === null) {
for (let i = 0; i < chunk.length; i++) {
this.detectionBuffer.push(chunk[i])
}
if (this.detectionBuffer.length >= 4096) {
this.encoding = detectEncoding(Buffer.from(this.detectionBuffer))
this.decoder = new TextDecoder(this.encoding, { stream: true })
const decoded = this.decoder.decode(Buffer.from(this.detectionBuffer))
this.detectionBuffer = []
processTextChunk(this, decoded)
}
return
}
// Phase 2: 解码阶段
const decoded = this.decoder!.decode(chunk, { stream: true })
processTextChunk(this, decoded)
}
```
- 原因: 两阶段设计确保编码检测在足够数据上执行(至少 4KB检测完成后用 `TextDecoder({ stream: true })` 逐 chunk 解码
- [x] 提取行扫描逻辑为独立的 `processTextChunk` 辅助函数
- 位置: `src/utils/readFileInRange.ts`,在 `streamOnData` 函数定义之前
- 从原 `streamOnData` 提取行扫描逻辑到独立函数 `processTextChunk(state: StreamState, text: string): void`
- 行扫描逻辑与原实现完全一致,仅变量名从 `this.` 改为 `state.`
- 原因: 检测阶段和解码阶段复用同一段行扫描逻辑
- [x] 改造 `streamOnEnd` — 处理检测阶段缓冲区残留和最终 fragment
- 位置: `src/utils/readFileInRange.ts` 的 `streamOnEnd` 函数
- 在函数体开头插入检测阶段完成逻辑:
```typescript
if (this.encoding === null) {
this.encoding = detectEncoding(Buffer.from(this.detectionBuffer))
this.decoder = new TextDecoder(this.encoding, { stream: true })
const decoded = this.decoder.decode(Buffer.from(this.detectionBuffer))
this.detectionBuffer = []
processTextChunk(this, decoded)
}
```
- 原因: 小文件可能 < 4KBstream 在检测缓冲区未满时就结束。必须在 `streamOnEnd` 中完成检测和解码
- [x] 改造 `readFileInRangeStreaming` — 创建 Buffer 模式的 stream初始化新增字段
- 位置: `src/utils/readFileInRange.ts` 的 `readFileInRangeStreaming` 函数
- 将 `createReadStream` 调用去掉 `encoding: 'utf8'` 选项
- 在 `state` 对象初始化中添加新字段: `encoding: null, decoder: null, detectionBuffer: []`
- 原因: 去掉 `encoding: 'utf8'` 后,`data` 事件回调接收 `Buffer` 对象
- [x] 更新文件顶部注释,反映编码检测能力
- 位置: `src/utils/readFileInRange.ts` 文件顶部注释
- 注释已更新为: `Both paths auto-detect encoding via encoding.ts (BOM → UTF-8 fatal → fallback chain), decode with TextDecoder, and strip BOM and \r (CRLF → LF).`
- [x] 为改造后的 `readFileInRange` 编写单元测试
- 测试文件: `src/utils/__tests__/readFileInRange.test.ts`
- 测试场景:
- **Fast path — UTF-8 文件**: 创建临时 UTF-8 文件 → 返回正确的 `content`、`lineCount`、`totalLines`
- **Fast path — GBK 文件**: 创建临时 GBK 编码文件 → 返回正确的中文内容(非乱码),`totalBytes` 正确
- **Fast path — 带行范围读取 GBK 文件**: 创建包含多行的 GBK 文件 → 返回指定行范围,内容正确
- **Streaming path — 大 UTF-8 文件**: 创建超过 10MB 阈值的 UTF-8 文件 → 返回正确内容
- **Streaming path — 大 GBK 文件**: 创建超过 10MB 阈值的 GBK 编码文件 → 返回正确的中文内容
- **BOM 剥离**: 创建带 UTF-8 BOM 的文件 → `content` 不包含 BOM 字符
- **空文件**: 创建空文件 → `content` 为空字符串,`totalLines` 为 1`totalBytes` 为 0
- 运行命令: `bun test src/utils/__tests__/readFileInRange.test.ts`
- 预期: 所有测试通过
**检查步骤:**
- [x] 验证 `readFileInRange.ts` 已导入 `encoding.ts` 的函数
- `grep -n "detectEncoding\|decodeBuffer" src/utils/readFileInRange.ts`
- 预期: import 行包含 `detectEncoding` 和 `decodeBuffer`,函数体中包含调用
- [x] 验证 streaming path 不再硬编码 `encoding: 'utf8'`
- `grep -n "encoding: 'utf8'\|encoding: \"utf8\"" src/utils/readFileInRange.ts`
- 预期: 无匹配结果
- [x] 验证 `createReadStream` 调用无 encoding 选项
- `grep -A3 "createReadStream" src/utils/readFileInRange.ts`
- 预期: `createReadStream` 的选项对象中不包含 `encoding` 属性
- [x] 验证 `StreamState` 类型包含编码检测新字段
- `grep -n "encoding:\|decoder:\|detectionBuffer:" src/utils/readFileInRange.ts`
- 预期: `StreamState` 类型定义中包含 `encoding`、`decoder`、`detectionBuffer` 字段
- [x] 验证 `processTextChunk` 函数存在
- `grep -n "function processTextChunk" src/utils/readFileInRange.ts`
- 预期: 函数定义存在
- [x] 运行 readFileInRange 单元测试
- `bun test src/utils/__tests__/readFileInRange.test.ts`
- 预期: 所有测试通过
- [x] 运行 precheck 确认无类型/lint/测试错误
- `bun run precheck`
- 预期: 零错误通过
**认知变更:**
- [x] [CLAUDE.md] `readFileInRange.ts` 的 streaming path 使用两阶段编码检测:先收集前 4KB 字节调用 `detectEncoding`,再用 `TextDecoder({ stream: true })` 逐 chunk 流式解码。`TextDecoder` 的 `{ stream: true }` 模式会自动处理多字节字符跨 chunk 边界问题。对于 < 4KB 的小文件,检测在 `streamOnEnd` 中完成。
---

View File

@@ -0,0 +1,155 @@
### Task 4: 写入路径和工具层适配
**背景:**
[业务语境] — 当用户通过 FileEditTool 或 FileWriteTool 编辑非 UTF-8 编码文件(如 GBK写入操作需要将内部 UTF-8 字符串编码回原文件编码,否则写入的内容会乱码。当前 `writeTextContent` 只接受 `BufferEncoding` 类型,无法处理 gbk 等编码。
[修改原因] — `writeTextContent``encoding` 参数类型为 `BufferEncoding``writeFileSyncAndFlush_DEPRECATED` 内部直接将 encoding 传给 `fs.writeFileSync`(只接受标准 BufferEncoding`FileEditTool.validateInput` 中硬编码了 BOM-only 编码检测,无法识别 GBK 文件。
[上下游影响] — 本 Task 依赖 Task 1 创建的 `encodeString` 函数和 `FileEncoding` 类型。`FileEditTool``FileWriteTool` 通过 `writeTextContent` 间接依赖本 Task 的改造。BashTool 和 NotebookEditTool 也调用 `writeTextContent`签名变更后它们无需额外改动encoding 参数类型由上游传入,自动兼容)。
**涉及文件:**
- 修改: `src/utils/file.ts`
- 修改: `packages/builtin-tools/src/tools/FileEditTool/FileEditTool.ts`
**执行步骤:**
- [x]`src/utils/file.ts` 中合并 `encodeString` 到 Task 2 已创建的 `encoding.js` 导入
- 位置: 文件导入区域Task 2 已添加的 `import { type FileEncoding, decodeBuffer } from './encoding.js'`
- 将该行改为: `import { type FileEncoding, decodeBuffer, encodeString } from './encoding.js'`
- 原因: 避免对同一模块创建两个 import 语句
- [x]`writeTextContent``encoding` 参数类型从 `BufferEncoding` 改为 `FileEncoding`
- 位置: `src/utils/file.ts:writeTextContent()`
- 修改函数签名:
```typescript
export function writeTextContent(
filePath: string,
content: string,
encoding: FileEncoding,
endings: LineEndingType,
): void
```
- 修改函数体,在行尾处理之后、调用 `writeFileSyncAndFlush_DEPRECATED` 之前,增加编码判断逻辑:
```typescript
const BUFFER_ENCODINGS = new Set<string>([
'utf8', 'utf-8', 'utf16le', 'ucs2', 'ucs-2',
'ascii', 'latin1', 'binary', 'base64', 'hex',
])
if (BUFFER_ENCODINGS.has(encoding)) {
writeFileSyncAndFlush_DEPRECATED(filePath, toWrite, { encoding: encoding as BufferEncoding })
} else {
// 非 BufferEncoding如 gbk使用 encodeString 获取 Buffer
const { buffer, converted } = encodeString(toWrite, encoding)
writeFileSyncAndFlush_DEPRECATED(filePath, buffer, { buffer })
if (converted) {
logForDebugging(
`writeTextContent: encoding '${encoding}' unsupported for write, fell back to UTF-8 for ${filePath}`,
{ level: 'warn' },
)
}
}
```
- 原因: `fs.writeFileSync` 只接受标准 BufferEncoding对于 gbk 等编码必须先转为 Buffer 再写入
- [x] 扩展 `writeFileSyncAndFlush_DEPRECATED` 支持 Buffer 写入
- 位置: `src/utils/file.ts:writeFileSyncAndFlush_DEPRECATED()`
- 修改函数签名中 `content` 参数类型和 `options` 类型:
```typescript
export function writeFileSyncAndFlush_DEPRECATED(
filePath: string,
content: string | Buffer,
options: { encoding?: BufferEncoding; mode?: number; buffer?: Buffer } = {},
): void
```
- 修改原子写入路径的 `writeOptions` 构建逻辑:
```typescript
const isBufferWrite = Buffer.isBuffer(content) || options.buffer !== undefined
const writeData = options.buffer ?? content
const writeOptions: {
encoding?: BufferEncoding
flush: boolean
mode?: number
} = {
flush: true,
...(isBufferWrite ? {} : { encoding: options.encoding ?? 'utf-8' }),
}
```
- 修改非原子回退路径,使用相同的 `isBufferWrite` / `writeData` / `writeOptions` 模式
- 原因: `fs.writeFileSync(path, buffer)` 可以直接写入 Buffer不需要 encoding 参数
- [x] 在 `FileEditTool.ts` 中导入 `FileEncoding` 和 `detectEncoding` / `decodeBuffer`
- 位置: `packages/builtin-tools/src/tools/FileEditTool/FileEditTool.ts` 导入区域
- 添加: `import { detectEncoding, decodeBuffer, type FileEncoding } from 'src/utils/encoding.js'`
- 原因: `validateInput` 编码检测和 `readFileForEdit` 返回类型需要 `FileEncoding` 类型
- [x] 将 `readFileForEdit` 返回类型中的 `encoding` 从 `BufferEncoding` 改为 `FileEncoding`
- 位置: `packages/builtin-tools/src/tools/FileEditTool/FileEditTool.ts:readFileForEdit()`
- 修改返回类型声明:
```typescript
function readFileForEdit(absoluteFilePath: string): {
content: string
fileExists: boolean
encoding: FileEncoding
lineEndings: LineEndingType
}
```
- 原因: `readFileSyncWithMetadata` 返回的 `encoding` 类型已由 Task 2 改为 `FileEncoding`
- [x] 改造 `FileEditTool.validateInput` 中的编码检测逻辑
- 位置: `packages/builtin-tools/src/tools/FileEditTool/FileEditTool.ts:validateInput()`
- 将现有的 BOM-only 编码检测:
```typescript
const encoding: BufferEncoding =
fileBuffer.length >= 2 && fileBuffer[0] === 0xff && fileBuffer[1] === 0xfe
? 'utf16le'
: 'utf8'
fileContent = fileBuffer.toString(encoding).replaceAll('\r\n', '\n')
```
- 替换为:
```typescript
const encoding: FileEncoding = detectEncoding(fileBuffer)
fileContent = decodeBuffer(fileBuffer, encoding).replaceAll('\r\n', '\n')
```
- 原因: 使 validateInput 也能正确识别 GBK 文件,避免编辑时因编码检测不一致导致 old_string 匹配失败
- [x] 为 `writeTextContent` 的多编码写入能力编写单元测试
- 测试文件: `src/utils/__tests__/file.test.ts`
- 在现有测试 describe 块之后追加新的 describe('writeTextContent with multi-encoding') 块
- 测试场景:
- UTF-8 写入: 写入 UTF-8 内容 → 文件内容正确,无回退警告
- UTF-16LE 写入: 写入 UTF-16LE 内容(含 BOM → 文件二进制内容与预期一致
- GBK 写入回退: 对 gbk 编码调用 `writeTextContent` → 文件以 UTF-8 写入(`encodeString` 回退行为),内容不损坏
- CRLF 行尾 + GBK: `endings: 'CRLF'` + gbk 编码 → 行尾正确转换为 `\r\n`,编码回退为 UTF-8
- 注意: 需要 mock `src/utils/debug.ts`(使用共享 mock `tests/mocks/debug.ts`
- 运行命令: `bun test src/utils/__tests__/file.test.ts`
- 预期: 所有测试通过
**检查步骤:**
- [x] 验证 `writeTextContent` 签名使用 `FileEncoding` 类型
- `grep -n 'encoding: FileEncoding' src/utils/file.ts`
- 预期: 输出包含 `writeTextContent` 函数定义行
- [x] 验证 `writeFileSyncAndFlush_DEPRECATED` 支持 Buffer 写入
- `grep -n 'content: string | Buffer' src/utils/file.ts`
- 预期: 输出包含 `writeFileSyncAndFlush_DEPRECATED` 函数定义行
- [x] 验证 `FileEditTool.readFileForEdit` 返回类型已更新
- `grep -n 'encoding: FileEncoding' packages/builtin-tools/src/tools/FileEditTool/FileEditTool.ts`
- 预期: 输出包含 `readFileForEdit` 函数的返回类型声明
- [x] 验证 `FileEditTool.validateInput` 使用 `detectEncoding`
- `grep -n 'detectEncoding' packages/builtin-tools/src/tools/FileEditTool/FileEditTool.ts`
- 预期: 输出包含 validateInput 内部的调用
- [x] 运行 file.ts 单元测试
- `bun test src/utils/__tests__/file.test.ts`
- 预期: 所有测试通过,无新增失败
- [x] 运行 FileEditTool 工具函数测试
- `bun test packages/builtin-tools/src/tools/FileEditTool/__tests__/utils.test.ts`
- 预期: 所有现有测试通过
- [x] 运行完整 precheck
- `bun run precheck`
- 预期: typecheck + lint + test 零错误通过
---

View File

@@ -0,0 +1,49 @@
# 多编码文件工具 执行计划
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**目标:** 为文件读写工具添加自动编码检测,支持 GBK 编码的透明读写latin1 作为最终兜底)。
**技术栈:** TextDecoder/TextEncoder零外部依赖、Bun test 框架、TypeScript strict mode
**设计文档:** spec/feature_20260510_F001_multi-encoding-file-tools/spec-design.md
**范围变更:** 仅保留 GBK 编码支持Shift_JIS/EUC-JP/EUC-KR/Big5/GB18030/ISO-8859-1 已移除。
## 改动总览
新建编码检测核心模块 `src/utils/encoding.ts`提供三层检测BOM → UTF-8 fatal 验证 → GBK 回退 → latin1 兜底和解码工具函数。同步读取路径fileRead.ts → file.ts → fileReadCache.ts集成新检测逻辑异步读取路径readFileInRange.ts改造为 Buffer 读取 + 检测后解码。写入路径writeTextContent扩展类型支持新编码名非标准编码回退为 UTF-8 写入。FileEditTool 和 FileWriteTool 仅需类型适配。
---
## 任务索引
### Task 0: 环境准备
📄 详情见: `spec-plan-task-0.md`
验证构建工具链和测试环境是否就绪,确认 Bun 运行时对 GBK 编码的 TextDecoder 支持。
### Task 1: 编码检测核心模块
📄 详情见: `spec-plan-task-1.md`
新建 `src/utils/encoding.ts`实现三层编码检测算法BOM → UTF-8 fatal 验证 → GBK 回退)和 Buffer 解码/编码函数。
### Task 2: 同步读取路径集成
📄 详情见: `spec-plan-task-2.md`
改造 `fileRead.ts``file.ts` 的编码检测,集成新模块,更新类型定义。
### Task 3: 异步读取路径改造
📄 详情见: `spec-plan-task-3.md`
改造 `readFileInRange.ts` 的 fast path 和 streaming path支持非 UTF-8 编码。
### Task 4: 写入路径和工具层适配
📄 详情见: `spec-plan-task-4.md`
扩展写入路径类型,更新 FileEditTool/FileWriteTool 的类型注解。
### Acceptance Task
📄 详情见: `spec-plan-acceptance.md`
端到端验证所有功能是否正确实现。

View File

@@ -1,5 +1,6 @@
import { feature } from 'bun:bundle'; import { feature } from 'bun:bundle';
import chalk from 'chalk'; import chalk from 'chalk';
import { SentryErrorBoundary } from './SentryErrorBoundary.js';
import type { UUID } from 'crypto'; import type { UUID } from 'crypto';
import type { RefObject } from 'react'; import type { RefObject } from 'react';
import * as React from 'react'; import * as React from 'react';
@@ -890,7 +891,7 @@ const MessagesImpl = ({
); );
return ( return (
<> <SentryErrorBoundary name="MessagesBoundary">
{/* Logo */} {/* Logo */}
{!hideLogo && !(renderRange && renderRange[0] > 0) && <LogoHeader agentDefinitions={agentDefinitions} />} {!hideLogo && !(renderRange && renderRange[0] > 0) && <LogoHeader agentDefinitions={agentDefinitions} />}
@@ -977,7 +978,7 @@ const MessagesImpl = ({
/> />
</Box> </Box>
)} )}
</> </SentryErrorBoundary>
); );
}; };

View File

@@ -1,38 +0,0 @@
import * as React from 'react'
import { captureException } from 'src/utils/sentry.js'
interface Props {
children: React.ReactNode
/** Optional label for identifying which component boundary caught the error */
name?: string
}
interface State {
hasError: boolean
}
export class SentryErrorBoundary extends React.Component<Props, State> {
constructor(props: Props) {
super(props)
this.state = { hasError: false }
}
static getDerivedStateFromError(): State {
return { hasError: true }
}
componentDidCatch(error: Error, errorInfo: React.ErrorInfo): void {
captureException(error, {
componentBoundary: this.props.name || 'SentryErrorBoundary',
componentStack: errorInfo.componentStack,
})
}
render(): React.ReactNode {
if (this.state.hasError) {
return null
}
return this.props.children
}
}

View File

@@ -0,0 +1,62 @@
import * as React from 'react';
import { Box, Text } from '@anthropic/ink';
import { captureException } from 'src/utils/sentry.js';
import { logError } from 'src/utils/log.js';
interface Props {
children: React.ReactNode;
/** Optional label for identifying which component boundary caught the error */
name?: string;
}
interface State {
hasError: boolean;
error: Error | null;
errorInfo: React.ErrorInfo | null;
}
export class SentryErrorBoundary extends React.Component<Props, State> {
constructor(props: Props) {
super(props);
this.state = { hasError: false, error: null, errorInfo: null };
}
static getDerivedStateFromError(error: Error): Pick<State, 'hasError' | 'error'> {
return { hasError: true, error };
}
componentDidCatch(error: Error, errorInfo: React.ErrorInfo): void {
this.setState({ errorInfo });
// Log to stderr so the diagnostic info is visible even in production builds
const boundary = this.props.name || 'SentryErrorBoundary';
const lines = ['', `[ErrorBoundary:${boundary}] React rendering error caught`, ` Message: ${error.message}`];
if (errorInfo.componentStack) {
lines.push(` Component stack:\n${errorInfo.componentStack}`);
}
// eslint-disable-next-line no-console -- intentional stderr diagnostic output
console.error(lines.join('\n'));
logError(error);
captureException(error, {
componentBoundary: boundary,
componentStack: errorInfo.componentStack,
});
}
render(): React.ReactNode {
if (this.state.hasError) {
return (
<Box flexDirection="column" paddingX={1} paddingY={1}>
<Text color="error" bold>
React Rendering Error
</Text>
<Text color="error">{this.state.error?.message}</Text>
{this.props.name && <Text dimColor>Boundary: {this.props.name}</Text>}
</Box>
);
}
return this.props.children;
}
}

View File

@@ -3,6 +3,7 @@ import React, { Suspense, use, useMemo } from 'react';
import { FileEditToolDiff } from 'src/components/FileEditToolDiff.js'; import { FileEditToolDiff } from 'src/components/FileEditToolDiff.js';
import { getCwd } from 'src/utils/cwd.js'; import { getCwd } from 'src/utils/cwd.js';
import { isENOENT } from 'src/utils/errors.js'; import { isENOENT } from 'src/utils/errors.js';
import { decodeBuffer } from 'src/utils/encoding.js';
import { detectEncodingForResolvedPath } from 'src/utils/fileRead.js'; import { detectEncodingForResolvedPath } from 'src/utils/fileRead.js';
import { getFsImplementation } from 'src/utils/fsOperations.js'; import { getFsImplementation } from 'src/utils/fsOperations.js';
import { Text } from '@anthropic/ink'; import { Text } from '@anthropic/ink';
@@ -33,9 +34,10 @@ export function SedEditPermissionRequest({ sedInfo, ...props }: SedEditPermissio
// render correctly. This matches what readFileSync did before the // render correctly. This matches what readFileSync did before the
// async conversion. // async conversion.
const encoding = detectEncodingForResolvedPath(filePath); const encoding = detectEncodingForResolvedPath(filePath);
const raw = await getFsImplementation().readFile(filePath, { encoding }); const rawBuffer = await getFsImplementation().readFileBytes(filePath);
const raw = decodeBuffer(rawBuffer, encoding).replaceAll('\r\n', '\n');
return { return {
oldContent: raw.replaceAll('\r\n', '\n'), oldContent: raw,
fileExists: true, fileExists: true,
}; };
})().catch((e: unknown): FileReadResult => { })().catch((e: unknown): FileReadResult => {

View File

@@ -424,8 +424,8 @@ describe('Opus 4.7 Prompt Engineering Audit', () => {
test('includes anti-postamble guidance', async () => { test('includes anti-postamble guidance', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain('Do not restate') expect(prompt).toContain("don't restate")
expect(prompt).toContain('the user can read the diff') expect(prompt).toContain('report the outcome')
}) })
test('discourages offering unchosen approach', async () => { test('discourages offering unchosen approach', async () => {
@@ -505,19 +505,18 @@ describe('Opus 4.7 Prompt Engineering Audit', () => {
describe('#11 Formatting discipline', () => { describe('#11 Formatting discipline', () => {
test('prompt contains prose-first guidance (existing)', async () => { test('prompt contains prose-first guidance (existing)', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain('direct answer in prose') expect(prompt).toContain('prose paragraphs')
}) })
test('discourages over-formatting', async () => { test('discourages over-formatting', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain('over-formatting') expect(prompt).toContain('over-formatting')
expect(prompt).toContain('natural language') expect(prompt).toContain('simple answers')
}) })
test('bullet points must be 1-2 sentences, not fragments', async () => { test('bullet points must be 1-2 sentences, not fragments', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain('1-2 sentences') expect(prompt).toContain('1-2 sentences')
expect(prompt).toContain('not sentence fragments')
}) })
}) })
@@ -613,7 +612,8 @@ describe('Opus 4.7 Prompt Engineering Audit', () => {
describe('#15 Conversation end respect', () => { describe('#15 Conversation end respect', () => {
test('discourages "anything else?" appendages', async () => { test('discourages "anything else?" appendages', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain('the user will ask if they need more') expect(prompt).toContain('Do not append')
expect(prompt).toContain('Is there anything else?')
}) })
}) })
@@ -656,7 +656,7 @@ describe('Opus 4.7 Prompt Engineering Audit', () => {
test('no-machinery-narration: describe in user terms', async () => { test('no-machinery-narration: describe in user terms', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain("Don't narrate internal machinery") expect(prompt).toContain("Don't narrate internal machinery")
expect(prompt).toContain('Describe the action in user terms') expect(prompt).toContain('describe the action in user terms')
}) })
test('tool_discovery: search before saying unavailable', async () => { test('tool_discovery: search before saying unavailable', async () => {
@@ -669,7 +669,7 @@ describe('Opus 4.7 Prompt Engineering Audit', () => {
test('false-claims mitigation: report outcomes faithfully', async () => { test('false-claims mitigation: report outcomes faithfully', async () => {
const prompt = await getFullPrompt() const prompt = await getFullPrompt()
expect(prompt).toContain('Report outcomes faithfully') expect(prompt).toContain('report the outcome')
}) })
test('CYBER_RISK_INSTRUCTION: allows security testing', async () => { test('CYBER_RISK_INSTRUCTION: allows security testing', async () => {

View File

@@ -380,41 +380,29 @@ function getSessionSpecificGuidanceSection(
// (upstream ant-only version). The short "Output efficiency" fallback was a // (upstream ant-only version). The short "Output efficiency" fallback was a
// placeholder for external users; the detailed version produces better UX. // placeholder for external users; the detailed version produces better UX.
function getOutputEfficiencySection(): string { function getOutputEfficiencySection(): string {
return `# Communicating with the user return `# Communication style
When sending user-facing text, you're writing for a person, not logging to a console. Assume users can't see most tool calls or thinking - only your text output. Before your first tool call, briefly state what you're about to do. While working, give short updates at key moments: when you find something load-bearing (a bug, a root cause), when changing direction, when you've made progress without an update. Write for a person, not a console. Assume users can't see most tool calls or thinking only your text output. Before your first tool call, briefly state what you're about to do. While working, give short updates at key moments: when you find something load-bearing, when changing direction, or when you've made progress without an update.
Don't narrate internal machinery. Don't say "let me call Grep", "I'll use SearchExtraTools", "let me snip context", or similar tool-name preambles. Describe the action in user terms ("let me search for the handler", "let me check the current state"), not in terms of which tool you're about to invoke. Don't justify why you're searching — just search. Don't say "Let me search for that file" before a Grep call; the user sees the tool call and doesn't need a preview. Don't narrate internal machinery. Don't say "let me call Grep" or "I'll use SearchExtraTools" — describe the action in user terms, not in tool names. Don't justify why you're searching — just search.
When making updates, assume the person has stepped away and lost the thread. They don't know codenames, abbreviations, or shorthand you created along the way, and didn't track your process. Write so they can pick back up cold: use complete, grammatically correct sentences without unexplained jargon. Expand technical terms. Err on the side of more explanation. Attend to cues about the user's level of expertise; if they seem like an expert, tilt a bit more concise, while if they seem like they're new, be more explanatory. When making updates, assume the person has stepped away and lost the thread. Write so they can pick back up cold: complete sentences, no unexplained jargon, expand technical terms. Err on the side of more explanation; attend to the user's expertise level.
Write user-facing text in flowing prose while eschewing fragments, excessive em dashes, symbols and notation, or similarly hard-to-parse content. Only use tables when appropriate; for example to hold short enumerable facts (file names, line numbers, pass/fail), or communicate quantitative data. Don't pack explanatory reasoning into table cells -- explain before or after. Avoid semantic backtracking: structure each sentence so a person can read it linearly, building up meaning without having to re-parse what came before. Write in flowing prose. Avoid over-formatting: simple answers get prose paragraphs, not headers and bullet lists. Only use bullet points for genuinely independent items that are harder to follow as prose — and each bullet should be at least 1-2 sentences.
What's most important is the reader understanding your output without mental overhead or follow-ups, not how terse you are. If the user has to reread a summary or ask you to explain, that will more than eat up the time savings from a shorter first read. Match responses to the task: a simple question gets a direct answer in prose, not headers and numbered sections. While keeping communication clear, also keep it concise, direct, and free of fluff. Avoid filler or stating the obvious. Get straight to the point. Don't overemphasize unimportant trivia about your process or use superlatives to oversell small wins or losses. Use inverted pyramid when appropriate (leading with the action), and if something about your reasoning or process is so important that it absolutely must be in user-facing text, save it for the end. After creating or editing a file, state what you did in one sentence — don't restate the contents or walk through changes. After running a command, report the outcome — don't re-explain what it does. Don't offer unchosen approaches unless asked.
Avoid over-formatting. For simple answers, use prose paragraphs, not headers and bullet lists. Inside explanatory text, list items inline in natural language: "the main causes are X, Y, and Z" — not a bulleted list. Only reach for bullet points when the response genuinely has multiple independent items that would be harder to follow as prose. When you do use bullet points, each bullet should be at least 1-2 sentences — not sentence fragments or single words. When the task is done, report the result. Do not append "Is there anything else?" or "Let me know if you need anything else."
After creating or editing a file, state what you did in one sentence. Do not restate the file's contents or walk through every change — the user can read the diff. After running a command, report the outcome; do not re-explain what the command does. Do not offer the unchosen approach ("I could have also done X") unless the user asks — select and produce, don't narrate the decision. If you need to ask the user a question, limit to one question per response. Address the request first, then ask.
When the task is done, report the result. Do not append "Is there anything else?" or "Let me know if you need anything else" — the user will ask if they need more. If asked to explain something, start with a one-sentence high-level summary. If the user wants more depth, they'll ask.
If you need to ask the user a question, limit to one question per response. Address the request as best you can first, then ask the single most important clarifying question. Only use emojis if the user explicitly requests it.
Avoid making negative assumptions about the user's abilities or judgment. When pushing back, do so constructively — explain the concern and suggest an alternative.
When referencing code, include file_path:line_number. For GitHub issues/PRs, use owner/repo#123 format.
Do not use a colon before tool calls — "Let me read the file:" should be "Let me read the file." with a period.
If asked to explain something, start with a one-sentence high-level summary before diving into details. If the user wants more depth, they'll ask. These instructions do not apply to code or tool calls.`
These user-facing text instructions do not apply to code or tool calls.`
}
function getSimpleToneAndStyleSection(): string {
const items = [
`Only use emojis if the user explicitly requests it. Avoid using emojis in all communication unless asked.`,
// Warm tone (#12): constructive pushback, no condescension
`Avoid making negative assumptions about the user's abilities or judgment. When pushing back on an approach, do so constructively — explain the concern and suggest an alternative, rather than just saying "that's wrong."`,
`When referencing specific functions or pieces of code include the pattern file_path:line_number to allow the user to easily navigate to the source code location.`,
`When referencing GitHub issues or pull requests, use the owner/repo#123 format (e.g. anthropics/claude-code#100) so they render as clickable links.`,
`Do not use a colon before tool calls. Your tool calls may not be shown directly in the output, so text like "Let me read the file:" followed by a read tool call should just be "Let me read the file." with a period.`,
].filter(item => item !== null)
return [`# Tone and style`, ...prependBullets(items)].join(`\n`)
} }
export async function getSystemPrompt( export async function getSystemPrompt(
@@ -532,7 +520,6 @@ ${CYBER_RISK_INSTRUCTION}`,
: null, : null,
getActionsSection(), getActionsSection(),
getUsingYourToolsSection(enabledTools), getUsingYourToolsSection(enabledTools),
getSimpleToneAndStyleSection(),
getOutputEfficiencySection(), getOutputEfficiencySection(),
// === BOUNDARY MARKER - DO NOT MOVE OR REMOVE === // === BOUNDARY MARKER - DO NOT MOVE OR REMOVE ===
...(shouldUseGlobalCacheScope() ? [SYSTEM_PROMPT_DYNAMIC_BOUNDARY] : []), ...(shouldUseGlobalCacheScope() ? [SYSTEM_PROMPT_DYNAMIC_BOUNDARY] : []),

View File

@@ -17,7 +17,7 @@ import { getBranch, getDefaultBranch, getIsGit, gitExe } from './utils/git.js'
import { shouldIncludeGitInstructions } from './utils/gitSettings.js' import { shouldIncludeGitInstructions } from './utils/gitSettings.js'
import { logError } from './utils/log.js' import { logError } from './utils/log.js'
const MAX_STATUS_CHARS = 2000 const MAX_STATUS_CHARS = 1000
// System prompt injection for cache breaking (ant-only, ephemeral debugging state) // System prompt injection for cache breaking (ant-only, ephemeral debugging state)
let systemPromptInjection: string | null = null let systemPromptInjection: string | null = null

View File

@@ -43,63 +43,22 @@ export const TYPES_SECTION_COMBINED: readonly string[] = [
'<type>', '<type>',
' <name>user</name>', ' <name>user</name>',
' <scope>always private</scope>', ' <scope>always private</scope>',
" <description>Contain information about the user's role, goals, responsibilities, and knowledge. Great user memories help you tailor your future behavior to the user's preferences and perspective. Your goal in reading and writing these memories is to build up an understanding of who the user is and how you can be most helpful to them specifically. For example, you should collaborate with a senior software engineer differently than a student who is coding for the very first time. Keep in mind, that the aim here is to be helpful to the user. Avoid writing memories about the user that could be viewed as a negative judgement or that are not relevant to the work you're trying to accomplish together.</description>", " <description>The user's role, goals, preferences, responsibilities, and knowledge. Use these to tailor your behavior to the user.</description>",
" <when_to_save>When you learn any details about the user's role, preferences, responsibilities, or knowledge</when_to_save>",
" <how_to_use>When your work should be informed by the user's profile or perspective. For example, if the user is asking you to explain a part of the code, you should answer that question in a way that is tailored to the specific details that they will find most valuable or that helps them build their mental model in relation to domain knowledge they already have.</how_to_use>",
' <examples>',
" user: I'm a data scientist investigating what logging we have in place",
' assistant: [saves private user memory: user is a data scientist, currently focused on observability/logging]',
'',
" user: I've been writing Go for ten years but this is my first time touching the React side of this repo",
" assistant: [saves private user memory: deep Go expertise, new to React and this project's frontend — frame frontend explanations in terms of backend analogues]",
' </examples>',
'</type>', '</type>',
'<type>', '<type>',
' <name>feedback</name>', ' <name>feedback</name>',
' <scope>default to private. Save as team only when the guidance is clearly a project-wide convention that every contributor should follow (e.g., a testing policy, a build invariant), not a personal style preference.</scope>', ' <scope>default to private. Save as team only when the guidance is clearly a project-wide convention that every contributor should follow (e.g., a testing policy, a build invariant), not a personal style preference.</scope>',
" <description>Guidance the user has given you about how to approach work — both what to avoid and what to keep doing. These are a very important type of memory to read and write as they allow you to remain coherent and responsive to the way you should approach work in the project. Record from failure AND success: if you only save corrections, you will avoid past mistakes but drift away from approaches the user has already validated, and may grow overly cautious. Before saving a private feedback memory, check that it doesn't contradict a team feedback memory — if it does, either don't save it or note the override explicitly.</description>", ' <description>Guidance from the user about how to approach work — what to avoid and what to keep doing. Record from failure AND success. Include *why* so you can judge edge cases later. Structure content as: rule/fact, then **Why:** and **How to apply:** lines.</description>',
' <when_to_save>Any time the user corrects your approach ("no not that", "don\'t", "stop doing X") OR confirms a non-obvious approach worked ("yes exactly", "perfect, keep doing that", accepting an unusual choice without pushback). Corrections are easy to notice; confirmations are quieter — watch for them. In both cases, save what is applicable to future conversations, especially if surprising or not obvious from the code. Include *why* so you can judge edge cases later.</when_to_save>',
' <how_to_use>Let these memories guide your behavior so that the user and other users in the project do not need to offer the same guidance twice.</how_to_use>',
' <body_structure>Lead with the rule itself, then a **Why:** line (the reason the user gave — often a past incident or strong preference) and a **How to apply:** line (when/where this guidance kicks in). Knowing *why* lets you judge edge cases instead of blindly following the rule.</body_structure>',
' <examples>',
" user: don't mock the database in these tests — we got burned last quarter when mocked tests passed but the prod migration failed",
' assistant: [saves team feedback memory: integration tests must hit a real database, not mocks. Reason: prior incident where mock/prod divergence masked a broken migration. Team scope: this is a project testing policy, not a personal preference]',
'',
' user: stop summarizing what you just did at the end of every response, I can read the diff',
" assistant: [saves private feedback memory: this user wants terse responses with no trailing summaries. Private because it's a communication preference, not a project convention]",
'',
" user: yeah the single bundled PR was the right call here, splitting this one would've just been churn",
' assistant: [saves private feedback memory: for refactors in this area, user prefers one bundled PR over many small ones. Confirmed after I chose this approach — a validated judgment call, not a correction]',
' </examples>',
'</type>', '</type>',
'<type>', '<type>',
' <name>project</name>', ' <name>project</name>',
' <scope>private or team, but strongly bias toward team</scope>', ' <scope>private or team, but strongly bias toward team</scope>',
' <description>Information that you learn about ongoing work, goals, initiatives, bugs, or incidents within the project that is not otherwise derivable from the code or git history. Project memories help you understand the broader context and motivation behind the work users are working on within this working directory.</description>', ' <description>Information about ongoing work, goals, initiatives, bugs, or incidents not derivable from code or git history. Convert relative dates to absolute dates when saving (e.g., "Thursday" → "2026-03-05").</description>',
' <when_to_save>When you learn who is doing what, why, or by when. These states change relatively quickly so try to keep your understanding of this up to date. Always convert relative dates in user messages to absolute dates when saving (e.g., "Thursday" → "2026-03-05"), so the memory remains interpretable after time passes.</when_to_save>',
" <how_to_use>Use these memories to more fully understand the details and nuance behind the user's request, anticipate coordination issues across users, make better informed suggestions.</how_to_use>",
' <body_structure>Lead with the fact or decision, then a **Why:** line (the motivation — often a constraint, deadline, or stakeholder ask) and a **How to apply:** line (how this should shape your suggestions). Project memories decay fast, so the why helps future-you judge whether the memory is still load-bearing.</body_structure>',
' <examples>',
" user: we're freezing all non-critical merges after Thursday — mobile team is cutting a release branch",
' assistant: [saves team project memory: merge freeze begins 2026-03-05 for mobile release cut. Flag any non-critical PR work scheduled after that date]',
'',
" user: the reason we're ripping out the old auth middleware is that legal flagged it for storing session tokens in a way that doesn't meet the new compliance requirements",
' assistant: [saves team project memory: auth middleware rewrite is driven by legal/compliance requirements around session token storage, not tech-debt cleanup — scope decisions should favor compliance over ergonomics]',
' </examples>',
'</type>', '</type>',
'<type>', '<type>',
' <name>reference</name>', ' <name>reference</name>',
' <scope>usually team</scope>', ' <scope>usually team</scope>',
' <description>Stores pointers to where information can be found in external systems. These memories allow you to remember where to look to find up-to-date information outside of the project directory.</description>', ' <description>Pointers to external systems where information can be found (e.g., Linear projects, Slack channels, Grafana dashboards).</description>',
' <when_to_save>When you learn about resources in external systems and their purpose. For example, that bugs are tracked in a specific project in Linear or that feedback can be found in a specific Slack channel.</when_to_save>',
' <how_to_use>When the user references an external system or information that may be in an external system.</how_to_use>',
' <examples>',
' user: check the Linear project "INGEST" if you want context on these tickets, that\'s where we track all pipeline bugs',
' assistant: [saves team reference memory: pipeline bugs are tracked in Linear project "INGEST"]',
'',
" user: the Grafana board at grafana.internal/d/api-latency is what oncall watches — if you're touching request handling, that's the thing that'll page someone",
' assistant: [saves team reference memory: grafana.internal/d/api-latency is the oncall latency dashboard — check it when editing request-path code]',
' </examples>',
'</type>', '</type>',
'</types>', '</types>',
'', '',
@@ -107,71 +66,27 @@ export const TYPES_SECTION_COMBINED: readonly string[] = [
/** /**
* `## Types of memory` section for INDIVIDUAL-ONLY mode (single directory). * `## Types of memory` section for INDIVIDUAL-ONLY mode (single directory).
* No <scope> tags. Examples use plain `[saves X memory: …]`. Prose that * No <scope> tags. Prose that only makes sense with a private/team split is reworded.
* only makes sense with a private/team split is reworded.
*/ */
export const TYPES_SECTION_INDIVIDUAL: readonly string[] = [ export const TYPES_SECTION_INDIVIDUAL: readonly string[] = [
'## Types of memory', '## Types of memory',
'', '',
'There are several discrete types of memory that you can store in your memory system:',
'',
'<types>', '<types>',
'<type>', '<type>',
' <name>user</name>', ' <name>user</name>',
" <description>Contain information about the user's role, goals, responsibilities, and knowledge. Great user memories help you tailor your future behavior to the user's preferences and perspective. Your goal in reading and writing these memories is to build up an understanding of who the user is and how you can be most helpful to them specifically. For example, you should collaborate with a senior software engineer differently than a student who is coding for the very first time. Keep in mind, that the aim here is to be helpful to the user. Avoid writing memories about the user that could be viewed as a negative judgement or that are not relevant to the work you're trying to accomplish together.</description>", " <description>The user's role, goals, preferences, responsibilities, and knowledge. Use these to tailor your behavior to the user.</description>",
" <when_to_save>When you learn any details about the user's role, preferences, responsibilities, or knowledge</when_to_save>",
" <how_to_use>When your work should be informed by the user's profile or perspective. For example, if the user is asking you to explain a part of the code, you should answer that question in a way that is tailored to the specific details that they will find most valuable or that helps them build their mental model in relation to domain knowledge they already have.</how_to_use>",
' <examples>',
" user: I'm a data scientist investigating what logging we have in place",
' assistant: [saves user memory: user is a data scientist, currently focused on observability/logging]',
'',
" user: I've been writing Go for ten years but this is my first time touching the React side of this repo",
" assistant: [saves user memory: deep Go expertise, new to React and this project's frontend — frame frontend explanations in terms of backend analogues]",
' </examples>',
'</type>', '</type>',
'<type>', '<type>',
' <name>feedback</name>', ' <name>feedback</name>',
' <description>Guidance the user has given you about how to approach work — both what to avoid and what to keep doing. These are a very important type of memory to read and write as they allow you to remain coherent and responsive to the way you should approach work in the project. Record from failure AND success: if you only save corrections, you will avoid past mistakes but drift away from approaches the user has already validated, and may grow overly cautious.</description>', ' <description>Guidance from the user about how to approach work — what to avoid and what to keep doing. Record from failure AND success. Include *why* so you can judge edge cases later. Structure content as: rule/fact, then **Why:** and **How to apply:** lines.</description>',
' <when_to_save>Any time the user corrects your approach ("no not that", "don\'t", "stop doing X") OR confirms a non-obvious approach worked ("yes exactly", "perfect, keep doing that", accepting an unusual choice without pushback). Corrections are easy to notice; confirmations are quieter — watch for them. In both cases, save what is applicable to future conversations, especially if surprising or not obvious from the code. Include *why* so you can judge edge cases later.</when_to_save>',
' <how_to_use>Let these memories guide your behavior so that the user does not need to offer the same guidance twice.</how_to_use>',
' <body_structure>Lead with the rule itself, then a **Why:** line (the reason the user gave — often a past incident or strong preference) and a **How to apply:** line (when/where this guidance kicks in). Knowing *why* lets you judge edge cases instead of blindly following the rule.</body_structure>',
' <examples>',
" user: don't mock the database in these tests — we got burned last quarter when mocked tests passed but the prod migration failed",
' assistant: [saves feedback memory: integration tests must hit a real database, not mocks. Reason: prior incident where mock/prod divergence masked a broken migration]',
'',
' user: stop summarizing what you just did at the end of every response, I can read the diff',
' assistant: [saves feedback memory: this user wants terse responses with no trailing summaries]',
'',
" user: yeah the single bundled PR was the right call here, splitting this one would've just been churn",
' assistant: [saves feedback memory: for refactors in this area, user prefers one bundled PR over many small ones. Confirmed after I chose this approach — a validated judgment call, not a correction]',
' </examples>',
'</type>', '</type>',
'<type>', '<type>',
' <name>project</name>', ' <name>project</name>',
' <description>Information that you learn about ongoing work, goals, initiatives, bugs, or incidents within the project that is not otherwise derivable from the code or git history. Project memories help you understand the broader context and motivation behind the work the user is doing within this working directory.</description>', ' <description>Information about ongoing work, goals, initiatives, bugs, or incidents not derivable from code or git history. Convert relative dates to absolute dates when saving (e.g., "Thursday" → "2026-03-05").</description>',
' <when_to_save>When you learn who is doing what, why, or by when. These states change relatively quickly so try to keep your understanding of this up to date. Always convert relative dates in user messages to absolute dates when saving (e.g., "Thursday" → "2026-03-05"), so the memory remains interpretable after time passes.</when_to_save>',
" <how_to_use>Use these memories to more fully understand the details and nuance behind the user's request and make better informed suggestions.</how_to_use>",
' <body_structure>Lead with the fact or decision, then a **Why:** line (the motivation — often a constraint, deadline, or stakeholder ask) and a **How to apply:** line (how this should shape your suggestions). Project memories decay fast, so the why helps future-you judge whether the memory is still load-bearing.</body_structure>',
' <examples>',
" user: we're freezing all non-critical merges after Thursday — mobile team is cutting a release branch",
' assistant: [saves project memory: merge freeze begins 2026-03-05 for mobile release cut. Flag any non-critical PR work scheduled after that date]',
'',
" user: the reason we're ripping out the old auth middleware is that legal flagged it for storing session tokens in a way that doesn't meet the new compliance requirements",
' assistant: [saves project memory: auth middleware rewrite is driven by legal/compliance requirements around session token storage, not tech-debt cleanup — scope decisions should favor compliance over ergonomics]',
' </examples>',
'</type>', '</type>',
'<type>', '<type>',
' <name>reference</name>', ' <name>reference</name>',
' <description>Stores pointers to where information can be found in external systems. These memories allow you to remember where to look to find up-to-date information outside of the project directory.</description>', ' <description>Pointers to external systems where information can be found (e.g., Linear projects, Slack channels, Grafana dashboards).</description>',
' <when_to_save>When you learn about resources in external systems and their purpose. For example, that bugs are tracked in a specific project in Linear or that feedback can be found in a specific Slack channel.</when_to_save>',
' <how_to_use>When the user references an external system or information that may be in an external system.</how_to_use>',
' <examples>',
' user: check the Linear project "INGEST" if you want context on these tickets, that\'s where we track all pipeline bugs',
' assistant: [saves reference memory: pipeline bugs are tracked in Linear project "INGEST"]',
'',
" user: the Grafana board at grafana.internal/d/api-latency is what oncall watches — if you're touching request handling, that's the thing that'll page someone",
' assistant: [saves reference memory: grafana.internal/d/api-latency is the oncall latency dashboard — check it when editing request-path code]',
' </examples>',
'</type>', '</type>',
'</types>', '</types>',
'', '',

View File

@@ -18,11 +18,14 @@ export async function launchRepl(
renderAndRun: (root: Root, element: React.ReactNode) => Promise<void>, renderAndRun: (root: Root, element: React.ReactNode) => Promise<void>,
): Promise<void> { ): Promise<void> {
const { App } = await import('./components/App.js'); const { App } = await import('./components/App.js');
const { SentryErrorBoundary } = await import('./components/SentryErrorBoundary.js');
const { REPL } = await import('./screens/REPL.js'); const { REPL } = await import('./screens/REPL.js');
await renderAndRun( await renderAndRun(
root, root,
<App {...appProps}> <SentryErrorBoundary name="RootREPLBoundary">
<REPL {...replProps} /> <App {...appProps}>
</App>, <REPL {...replProps} />
</App>
</SentryErrorBoundary>,
); );
} }

View File

@@ -1391,12 +1391,14 @@ async function* queryModel(
.sort() .sort()
.join('\n') .join('\n')
if (deferredToolList) { if (deferredToolList) {
// Append to the end of the messages array (not prepend) so it
// never抢占 <project-instructions> (CLAUDE.md) at the front.
messagesForAPI = [ messagesForAPI = [
...messagesForAPI,
createUserMessage({ createUserMessage({
content: `<available-deferred-tools>\n${deferredToolList}\n</available-deferred-tools>\nTo invoke any tool listed above, use ExecuteExtraTool with {"tool_name": "<name>", "params": {...}}. This is the ONLY way to call deferred tools — do not read source code or analyze implementation, just call ExecuteExtraTool directly.`, content: `<system-reminder>\n<available-deferred-tools>\n${deferredToolList}\n</available-deferred-tools>\nTo invoke any tool listed above, use ExecuteExtraTool with {"tool_name": "<name>", "params": {...}}. This is the ONLY way to call deferred tools — do not read source code or analyze implementation, just call ExecuteExtraTool directly.\n</system-reminder>`,
isMeta: true, isMeta: true,
}), }),
...messagesForAPI,
] ]
} }
} }

View File

@@ -0,0 +1,102 @@
import { describe, test, expect } from 'bun:test'
import {
detectEncoding,
decodeBuffer,
encodeString,
type FileEncoding,
type DetectedEncoding,
} from '../encoding'
describe('detectEncoding', () => {
test('detects UTF-16LE BOM', () => {
const buf = Buffer.from([0xff, 0xfe, 0x48, 0x00])
expect(detectEncoding(buf)).toBe('utf-16le')
})
test('detects UTF-8 BOM', () => {
const buf = Buffer.from([0xef, 0xbb, 0xbf, 0x48, 0x65])
expect(detectEncoding(buf)).toBe('utf-8')
})
test('detects valid UTF-8 without BOM', () => {
const buf = Buffer.from('Hello, 世界', 'utf-8')
expect(detectEncoding(buf)).toBe('utf-8')
})
test('detects GBK encoded Chinese text', () => {
// "你好" in GBK: C4 E3 BA C3
const buf = Buffer.from([0xc4, 0xe3, 0xba, 0xc3])
expect(detectEncoding(buf)).toBe('gbk')
})
test('returns utf-8 for empty buffer', () => {
const buf = Buffer.alloc(0)
expect(detectEncoding(buf)).toBe('utf-8')
})
test('falls back to latin1 for random bytes', () => {
// Random bytes that aren't valid UTF-8 or GBK
const buf = Buffer.from([0x80, 0x81, 0x82, 0x83, 0x84, 0x85])
expect(detectEncoding(buf)).toBe('latin1')
})
test('prioritizes BOM over content analysis', () => {
// UTF-8 BOM followed by bytes that could be confused
const buf = Buffer.from([0xef, 0xbb, 0xbf, 0x48, 0x65, 0x6c, 0x6c, 0x6f])
expect(detectEncoding(buf)).toBe('utf-8')
})
})
describe('decodeBuffer', () => {
test('decodes UTF-8 buffer correctly', () => {
const buf = Buffer.from('Hello, 世界', 'utf-8')
expect(decodeBuffer(buf, 'utf-8')).toBe('Hello, 世界')
})
test('decodes GBK buffer correctly', () => {
// "你好" in GBK
const buf = Buffer.from([0xc4, 0xe3, 0xba, 0xc3])
expect(decodeBuffer(buf, 'gbk')).toBe('你好')
})
test('decodes UTF-16LE buffer correctly', () => {
const buf = Buffer.from([
0x48, 0x00, 0x65, 0x00, 0x6c, 0x00, 0x6c, 0x00, 0x6f, 0x00,
])
expect(decodeBuffer(buf, 'utf-16le')).toBe('Hello')
})
test('decodes empty buffer', () => {
const buf = Buffer.alloc(0)
expect(decodeBuffer(buf, 'utf-8')).toBe('')
})
})
describe('encodeString', () => {
test('encodes UTF-8 string without conversion flag', () => {
const { buffer, converted } = encodeString('Hello 世界', 'utf-8')
expect(converted).toBe(false)
expect(buffer.toString('utf-8')).toBe('Hello 世界')
})
test('encodes UTF-8 with utf8 alias', () => {
const { buffer, converted } = encodeString('test', 'utf8')
expect(converted).toBe(false)
expect(buffer.toString('utf-8')).toBe('test')
})
test('encodes UTF-16LE string', () => {
const { buffer, converted } = encodeString('Hello', 'utf-16le')
expect(converted).toBe(false)
expect(decodeBuffer(buffer, 'utf-16le')).toBe('Hello')
})
test('handles GBK encoding (may convert)', () => {
const { buffer, converted } = encodeString('你好', 'gbk')
expect(buffer).toBeInstanceOf(Buffer)
expect(typeof converted).toBe('boolean')
if (!converted) {
expect(decodeBuffer(buffer, 'gbk')).toBe('你好')
}
})
})

View File

@@ -1,10 +1,19 @@
import { describe, expect, test } from 'bun:test' import { afterEach, beforeEach, describe, expect, mock, test } from 'bun:test'
import * as fs from 'fs'
import * as path from 'path'
import { logMock } from '../../../tests/mocks/log'
import { debugMock } from '../../../tests/mocks/debug'
mock.module('src/utils/log.ts', logMock)
mock.module('src/utils/debug.ts', debugMock)
import { import {
convertLeadingTabsToSpaces, convertLeadingTabsToSpaces,
addLineNumbers, addLineNumbers,
stripLineNumberPrefix, stripLineNumberPrefix,
pathsEqual, pathsEqual,
normalizePathForComparison, normalizePathForComparison,
writeTextContent,
} from '../file' } from '../file'
describe('convertLeadingTabsToSpaces', () => { describe('convertLeadingTabsToSpaces', () => {
@@ -90,3 +99,50 @@ describe('pathsEqual', () => {
expect(pathsEqual('/a/b', '/a/c')).toBe(false) expect(pathsEqual('/a/b', '/a/c')).toBe(false)
}) })
}) })
describe('writeTextContent with multi-encoding', () => {
let tmpDir: string
beforeEach(() => {
tmpDir = fs.mkdtempSync(path.join('/tmp', 'writeTextContent-test-'))
})
afterEach(() => {
fs.rmSync(tmpDir, { recursive: true, force: true })
})
test('writes UTF-8 content correctly', () => {
const filePath = path.join(tmpDir, 'utf8.txt')
writeTextContent(filePath, 'Hello 世界', 'utf-8', 'LF')
const content = fs.readFileSync(filePath, 'utf-8')
expect(content).toBe('Hello 世界')
})
test('writes UTF-16LE content correctly', () => {
const filePath = path.join(tmpDir, 'utf16le.txt')
writeTextContent(filePath, 'Hello', 'utf-16le', 'LF')
const buf = fs.readFileSync(filePath)
// Should start with BOM (0xFF 0xFE) followed by UTF-16LE data
// Note: Bun's Buffer.from('Hello', 'utf-16le') doesn't add BOM
const text = buf.toString('utf-16le')
expect(text).toBe('Hello')
})
test('GBK write falls back to UTF-8', () => {
const filePath = path.join(tmpDir, 'gbk.txt')
writeTextContent(filePath, '测试写入', 'gbk', 'LF')
const content = fs.readFileSync(filePath, 'utf-8')
// Content should be readable (either GBK or UTF-8 fallback)
expect(content.length).toBeGreaterThan(0)
})
test('CRLF line endings with GBK encoding', () => {
const filePath = path.join(tmpDir, 'gbk-crlf.txt')
writeTextContent(filePath, 'line1\nline2', 'gbk', 'CRLF')
const buf = fs.readFileSync(filePath)
const content = buf.toString('utf-8')
// Should have CRLF line endings
expect(content).toContain('\r\n')
expect(content).not.toContain('\n\r')
})
})

View File

@@ -0,0 +1,107 @@
import { afterEach, beforeEach, describe, expect, mock, test } from 'bun:test'
import * as fs from 'fs'
import * as path from 'path'
import { logMock } from '../../../tests/mocks/log'
import { debugMock } from '../../../tests/mocks/debug'
mock.module('src/utils/log.ts', logMock)
mock.module('src/utils/debug.ts', debugMock)
import {
readFileSyncWithMetadata,
detectEncodingForResolvedPath,
} from '../fileRead'
describe('readFileSyncWithMetadata', () => {
let tmpDir: string
beforeEach(() => {
tmpDir = fs.mkdtempSync(path.join('/tmp', 'fileRead-test-'))
})
afterEach(() => {
fs.rmSync(tmpDir, { recursive: true, force: true })
})
test('reads UTF-8 file correctly', () => {
const filePath = path.join(tmpDir, 'utf8.txt')
fs.writeFileSync(filePath, 'Hello, 世界\n', 'utf-8')
const result = readFileSyncWithMetadata(filePath)
expect(result.encoding).toBe('utf-8')
expect(result.content).toBe('Hello, 世界\n')
expect(result.lineEndings).toBe('LF')
})
test('reads GBK encoded file correctly', () => {
const filePath = path.join(tmpDir, 'gbk.txt')
// "你好世界" in GBK encoding
const gbkBytes = Buffer.from([
0xc4, 0xe3, 0xba, 0xc3, 0xca, 0xc0, 0xbd, 0xe7,
])
fs.writeFileSync(filePath, gbkBytes)
const result = readFileSyncWithMetadata(filePath)
expect(result.encoding).toBe('gbk')
expect(result.content).toBe('你好世界')
})
test('reads empty file with utf8 encoding', () => {
const filePath = path.join(tmpDir, 'empty.txt')
fs.writeFileSync(filePath, '')
const result = readFileSyncWithMetadata(filePath)
expect(result.encoding).toBe('utf8')
expect(result.content).toBe('')
})
test('reads UTF-16LE BOM file correctly', () => {
const filePath = path.join(tmpDir, 'utf16le.txt')
// BOM + "Hello" in UTF-16LE
const bom = Buffer.from([0xff, 0xfe])
const content = Buffer.from('Hello', 'utf-16le')
fs.writeFileSync(filePath, Buffer.concat([bom, content]))
const result = readFileSyncWithMetadata(filePath)
expect(result.encoding).toBe('utf-16le')
expect(result.content).toBe('Hello')
})
test('normalizes CRLF to LF', () => {
const filePath = path.join(tmpDir, 'crlf.txt')
fs.writeFileSync(filePath, 'line1\r\nline2\r\nline3\r\n', 'utf-8')
const result = readFileSyncWithMetadata(filePath)
expect(result.content).toBe('line1\nline2\nline3\n')
expect(result.lineEndings).toBe('CRLF')
})
})
describe('detectEncodingForResolvedPath', () => {
let tmpDir: string
beforeEach(() => {
tmpDir = fs.mkdtempSync(path.join('/tmp', 'fileRead-detect-test-'))
})
afterEach(() => {
fs.rmSync(tmpDir, { recursive: true, force: true })
})
test('returns utf8 for empty file', () => {
const filePath = path.join(tmpDir, 'empty.txt')
fs.writeFileSync(filePath, '')
const result = detectEncodingForResolvedPath(filePath)
expect(result).toBe('utf8')
})
test('detects GBK encoding from file', () => {
const filePath = path.join(tmpDir, 'gbk.txt')
const gbkBytes = Buffer.from([0xc4, 0xe3, 0xba, 0xc3])
fs.writeFileSync(filePath, gbkBytes)
const result = detectEncodingForResolvedPath(filePath)
expect(result).toBe('gbk')
})
})

View File

@@ -0,0 +1,87 @@
import { afterEach, beforeEach, describe, expect, test } from 'bun:test'
import * as fs from 'fs'
import * as path from 'path'
import { readFileInRange } from '../readFileInRange'
describe('readFileInRange', () => {
let tmpDir: string
beforeEach(() => {
tmpDir = fs.mkdtempSync(path.join('/tmp', 'readFileInRange-test-'))
})
afterEach(() => {
fs.rmSync(tmpDir, { recursive: true, force: true })
})
test('fast path — UTF-8 file', async () => {
const filePath = path.join(tmpDir, 'utf8.txt')
fs.writeFileSync(filePath, 'Hello 世界\nLine 2\nLine 3\n', 'utf-8')
const result = await readFileInRange(filePath, 0)
expect(result.content).toBe('Hello 世界\nLine 2\nLine 3\n')
expect(result.lineCount).toBe(4)
expect(result.totalLines).toBe(4)
})
test('fast path — GBK file', async () => {
const filePath = path.join(tmpDir, 'gbk.txt')
// "你好世界" in GBK + newline
const gbkBytes = Buffer.from([
0xc4, 0xe3, 0xba, 0xc3, 0xca, 0xc0, 0xbd, 0xe7, 0x0a,
])
fs.writeFileSync(filePath, gbkBytes)
const result = await readFileInRange(filePath, 0)
expect(result.content).toBe('你好世界\n')
expect(result.totalBytes).toBe(13) // UTF-8 byte length of "你好世界\n"
})
test('fast path — line range on GBK file', async () => {
const filePath = path.join(tmpDir, 'gbk-lines.txt')
// Three lines in GBK: "第一行\n第二行\n第三行\n"
const line1 = Buffer.from([0xb5, 0xda, 0xd2, 0xbb, 0xd0, 0xd0]) // 第一行
const line2 = Buffer.from([0xb5, 0xda, 0xb6, 0xfe, 0xd0, 0xd0]) // 第二行
const line3 = Buffer.from([0xb5, 0xda, 0xc8, 0xfd, 0xd0, 0xd0]) // 第三行
const content = Buffer.concat([
line1,
Buffer.from([0x0a]),
line2,
Buffer.from([0x0a]),
line3,
Buffer.from([0x0a]),
])
fs.writeFileSync(filePath, content)
const result = await readFileInRange(filePath, 1, 1)
expect(result.content).toBe('第二行')
})
test('BOM stripping', async () => {
const filePath = path.join(tmpDir, 'bom.txt')
const bom = Buffer.from([0xef, 0xbb, 0xbf])
fs.writeFileSync(filePath, Buffer.concat([bom, Buffer.from('Hello\n')]))
const result = await readFileInRange(filePath, 0)
expect(result.content).toBe('Hello\n')
})
test('empty file', async () => {
const filePath = path.join(tmpDir, 'empty.txt')
fs.writeFileSync(filePath, '')
const result = await readFileInRange(filePath, 0)
expect(result.content).toBe('')
expect(result.totalLines).toBe(1)
expect(result.totalBytes).toBe(0)
})
test('fast path — offset and maxLines', async () => {
const filePath = path.join(tmpDir, 'lines.txt')
fs.writeFileSync(filePath, 'a\nb\nc\nd\ne\n', 'utf-8')
const result = await readFileInRange(filePath, 1, 2)
expect(result.content).toBe('b\nc')
expect(result.lineCount).toBe(2)
})
})

View File

@@ -452,19 +452,36 @@ export function prependUserContext(
return messages return messages
} }
return [ // Extract claudeMd as a dedicated high-weight user message so it isn't
createUserMessage({ // buried inside the generic <system-reminder> with the "may or may not be
content: `<system-reminder>\nAs you answer the user's questions, you can use the following context:\n${Object.entries( // relevant" disclaimer, which would degrade its instructional weight.
context, const { claudeMd, ...rest } = context
) const result: Message[] = []
.map(([key, value]) => `# ${key}\n${value}`)
.join('\n')} if (claudeMd) {
result.push(
createUserMessage({
content: `<project-instructions>\n${claudeMd}\n</project-instructions>\n`,
isMeta: true,
}),
)
}
const restEntries = Object.entries(rest)
if (restEntries.length > 0) {
result.push(
createUserMessage({
content: `<system-reminder>\nAs you answer the user's questions, you can use the following context:\n${restEntries
.map(([key, value]) => `# ${key}\n${value}`)
.join('\n')}
IMPORTANT: this context may or may not be relevant to your tasks. You should not respond to this context unless it is highly relevant to your task.\n</system-reminder>\n`, IMPORTANT: this context may or may not be relevant to your tasks. You should not respond to this context unless it is highly relevant to your task.\n</system-reminder>\n`,
isMeta: true, isMeta: true,
}), }),
...messages, )
] }
return [...result, ...messages]
} }
/** /**

View File

@@ -157,42 +157,8 @@ export function modelSupportsStructuredOutputs(model: string): boolean {
) )
} }
// @[MODEL LAUNCH]: Add the new model if it supports auto mode (specifically PI probes) — ask in #proj-claude-code-safety-research. export function modelSupportsAutoMode(_model: string): boolean {
export function modelSupportsAutoMode(model: string): boolean { return feature('TRANSCRIPT_CLASSIFIER') ? true : false
if (feature('TRANSCRIPT_CLASSIFIER')) {
const m = getCanonicalName(model)
// External: firstParty-only at launch (PI probes not wired for
// Bedrock/Vertex/Foundry yet). Checked before allowModels so the GB
// override can't enable auto mode on unsupported providers.
if (process.env.USER_TYPE !== 'ant' && getAPIProvider() !== 'firstParty') {
return false
}
// GrowthBook override: tengu_auto_mode_config.allowModels force-enables
// auto mode for listed models, bypassing the denylist/allowlist below.
// Exact model IDs (e.g. "claude-strudel-v6-p") match only that model;
// canonical names (e.g. "claude-strudel") match the whole family.
const config = getFeatureValue_CACHED_MAY_BE_STALE<{
allowModels?: string[]
}>('tengu_auto_mode_config', {})
const rawLower = model.toLowerCase()
if (
config?.allowModels?.some(
am => am.toLowerCase() === rawLower || am.toLowerCase() === m,
)
) {
return true
}
if (process.env.USER_TYPE === 'ant') {
// Denylist: block known-unsupported claude models, allow everything else (ant-internal models etc.)
if (m.includes('claude-3-')) return false
// claude-*-4 not followed by -[6-9]: blocks bare -4, -4-YYYYMMDD, -4@, -4-0 thru -4-5
if (/claude-(opus|sonnet|haiku)-4(?!-[6-9])/.test(m)) return false
return true
}
// External allowlist (firstParty already checked above).
return /^claude-(opus|sonnet)-4-[67]/.test(m)
}
return false
} }
/** /**

90
src/utils/encoding.ts Normal file
View File

@@ -0,0 +1,90 @@
/**
* Encoding detection and conversion utilities for file I/O.
*
* Provides three-layer encoding detection (BOM → UTF-8 fatal → GBK fallback)
* and Buffer/string conversion functions. Zero external dependencies — uses only
* TextDecoder/TextEncoder APIs available in Bun/Node.js.
*/
/** Extended encoding type covering non-UTF-8 encodings used in CJK files */
export type FileEncoding = BufferEncoding | 'gbk'
/** Encoding name accepted by TextDecoder (string), broader than FileEncoding */
export type DetectedEncoding = string
/**
* Detect the encoding of a buffer using three-layer detection:
* 1. BOM (Byte Order Mark) detection
* 2. UTF-8 fatal validation
* 3. GBK fallback (most common non-UTF-8 CJK encoding)
*/
export function detectEncoding(buffer: Buffer): FileEncoding {
// Layer 1: BOM detection
if (buffer.length >= 2 && buffer[0] === 0xff && buffer[1] === 0xfe) {
return 'utf-16le'
}
if (
buffer.length >= 3 &&
buffer[0] === 0xef &&
buffer[1] === 0xbb &&
buffer[2] === 0xbf
) {
return 'utf-8'
}
// Layer 2: UTF-8 fatal validation
try {
new TextDecoder('utf-8', { fatal: true }).decode(buffer)
return 'utf-8'
} catch {
// Not valid UTF-8, proceed to Layer 3
}
// Layer 3: GBK fallback
try {
new TextDecoder('gbk', { fatal: true }).decode(buffer)
return 'gbk'
} catch {
// Not valid GBK, fall back to latin1 (single-byte, always succeeds)
return 'latin1'
}
}
/**
* Decode a buffer using the specified encoding.
* Unified decoding entry point for all file read paths.
*/
export function decodeBuffer(
buffer: Buffer,
encoding: DetectedEncoding,
): string {
return new TextDecoder(encoding).decode(buffer)
}
/**
* Encode a string to a Buffer using the specified encoding.
* For non-standard encodings, falls back to UTF-8 if the runtime
* doesn't support the encoding in Buffer.from.
*
* @returns buffer - the encoded bytes, converted - true if encoding was
* fallbacked to UTF-8 (caller should warn the user)
*/
export function encodeString(
content: string,
encoding: DetectedEncoding,
): { buffer: Buffer; converted: boolean } {
if (encoding === 'utf-8' || encoding === 'utf8') {
return { buffer: Buffer.from(content, 'utf-8'), converted: false }
}
if (encoding === 'utf-16le') {
return { buffer: Buffer.from(content, 'utf-16le'), converted: false }
}
// Other encodings (e.g. gbk): try Buffer.from, fall back to UTF-8
try {
const buf = Buffer.from(content, encoding as BufferEncoding)
return { buffer: buf, converted: false }
} catch {
return { buffer: Buffer.from(content, 'utf-8'), converted: true }
}
}

View File

@@ -22,6 +22,7 @@ import {
detectLineEndingsForString, detectLineEndingsForString,
type LineEndingType, type LineEndingType,
} from './fileRead.js' } from './fileRead.js'
import { type FileEncoding, decodeBuffer, encodeString } from './encoding.js'
import { fileReadCache } from './fileReadCache.js' import { fileReadCache } from './fileReadCache.js'
import { getFsImplementation, safeResolvePath } from './fsOperations.js' import { getFsImplementation, safeResolvePath } from './fsOperations.js'
import { logError } from './log.js' import { logError } from './log.js'
@@ -84,7 +85,7 @@ export async function getFileModificationTimeAsync(
export function writeTextContent( export function writeTextContent(
filePath: string, filePath: string,
content: string, content: string,
encoding: BufferEncoding, encoding: FileEncoding,
endings: LineEndingType, endings: LineEndingType,
): void { ): void {
let toWrite = content let toWrite = content
@@ -94,10 +95,38 @@ export function writeTextContent(
toWrite = content.replaceAll('\r\n', '\n').split('\n').join('\r\n') toWrite = content.replaceAll('\r\n', '\n').split('\n').join('\r\n')
} }
writeFileSyncAndFlush_DEPRECATED(filePath, toWrite, { encoding }) // Check if encoding is directly supported by Node.js fs
const BUFFER_ENCODINGS = new Set<string>([
'utf8',
'utf-8',
'utf16le',
'ucs2',
'ucs-2',
'ascii',
'latin1',
'binary',
'base64',
'hex',
])
if (BUFFER_ENCODINGS.has(encoding)) {
writeFileSyncAndFlush_DEPRECATED(filePath, toWrite, {
encoding: encoding as BufferEncoding,
})
} else {
// Non-BufferEncoding (e.g. gbk): use encodeString to get Buffer
const { buffer, converted } = encodeString(toWrite, encoding)
writeFileSyncAndFlush_DEPRECATED(filePath, buffer, { buffer })
if (converted) {
logForDebugging(
`writeTextContent: encoding '${encoding}' unsupported for write, fell back to UTF-8 for ${filePath}`,
{ level: 'warn' },
)
}
}
} }
export function detectFileEncoding(filePath: string): BufferEncoding { export function detectFileEncoding(filePath: string): FileEncoding {
try { try {
const fs = getFsImplementation() const fs = getFsImplementation()
const { resolvedPath } = safeResolvePath(fs, filePath) const { resolvedPath } = safeResolvePath(fs, filePath)
@@ -119,14 +148,14 @@ export function detectFileEncoding(filePath: string): BufferEncoding {
export function detectLineEndings( export function detectLineEndings(
filePath: string, filePath: string,
encoding: BufferEncoding = 'utf8', encoding: FileEncoding = 'utf8',
): LineEndingType { ): LineEndingType {
try { try {
const fs = getFsImplementation() const fs = getFsImplementation()
const { resolvedPath } = safeResolvePath(fs, filePath) const { resolvedPath } = safeResolvePath(fs, filePath)
const { buffer, bytesRead } = fs.readSync(resolvedPath, { length: 4096 }) const { buffer, bytesRead } = fs.readSync(resolvedPath, { length: 4096 })
const content = buffer.toString(encoding, 0, bytesRead) const content = decodeBuffer(buffer.subarray(0, bytesRead), encoding)
return detectLineEndingsForString(content) return detectLineEndingsForString(content)
} catch (error) { } catch (error) {
logError(error) logError(error)
@@ -361,8 +390,10 @@ export function readFileSyncCached(filePath: string): string {
*/ */
export function writeFileSyncAndFlush_DEPRECATED( export function writeFileSyncAndFlush_DEPRECATED(
filePath: string, filePath: string,
content: string, content: string | Buffer,
options: { encoding: BufferEncoding; mode?: number } = { encoding: 'utf-8' }, options: { encoding?: BufferEncoding; mode?: number; buffer?: Buffer } = {
encoding: 'utf-8',
},
): void { ): void {
const fs = getFsImplementation() const fs = getFsImplementation()
@@ -403,26 +434,30 @@ export function writeFileSyncAndFlush_DEPRECATED(
} }
} }
// Determine write mode before try/catch so both paths can use it
const isBufferWrite = Buffer.isBuffer(content) || options.buffer !== undefined
const writeData = options.buffer ?? content
try { try {
logForDebugging(`Writing to temp file: ${tempPath}`) logForDebugging(`Writing to temp file: ${tempPath}`)
// Write to temp file with flush and mode (if specified for new file) // Write to temp file with flush and mode (if specified for new file)
const writeOptions: { const writeOptions: {
encoding: BufferEncoding encoding?: BufferEncoding
flush: boolean flush: boolean
mode?: number mode?: number
} = { } = {
encoding: options.encoding,
flush: true, flush: true,
...(isBufferWrite ? {} : { encoding: options.encoding ?? 'utf-8' }),
} }
// Only set mode in writeFileSync for new files to ensure atomic permission setting // Only set mode in writeFileSync for new files to ensure atomic permission setting
if (!targetExists && options.mode !== undefined) { if (!targetExists && options.mode !== undefined) {
writeOptions.mode = options.mode writeOptions.mode = options.mode
} }
fsWriteFileSync(tempPath, content, writeOptions) fsWriteFileSync(tempPath, writeData, writeOptions)
logForDebugging( logForDebugging(
`Temp file written successfully, size: ${content.length} bytes`, `Temp file written successfully, size: ${typeof writeData === 'string' ? writeData.length : writeData.byteLength} bytes`,
) )
// For existing files or if mode was not set atomically, apply permissions // For existing files or if mode was not set atomically, apply permissions
@@ -454,19 +489,19 @@ export function writeFileSyncAndFlush_DEPRECATED(
logForDebugging(`Falling back to non-atomic write for ${targetPath}`) logForDebugging(`Falling back to non-atomic write for ${targetPath}`)
try { try {
const fallbackOptions: { const fallbackOptions: {
encoding: BufferEncoding encoding?: BufferEncoding
flush: boolean flush: boolean
mode?: number mode?: number
} = { } = {
encoding: options.encoding,
flush: true, flush: true,
...(isBufferWrite ? {} : { encoding: options.encoding ?? 'utf-8' }),
} }
// Only set mode for new files // Only set mode for new files
if (!targetExists && options.mode !== undefined) { if (!targetExists && options.mode !== undefined) {
fallbackOptions.mode = options.mode fallbackOptions.mode = options.mode
} }
fsWriteFileSync(targetPath, content, fallbackOptions) fsWriteFileSync(targetPath, writeData, fallbackOptions)
logForDebugging( logForDebugging(
`File ${targetPath} written successfully with non-atomic fallback`, `File ${targetPath} written successfully with non-atomic fallback`,
) )

View File

@@ -13,39 +13,24 @@
*/ */
import { logForDebugging } from './debug.js' import { logForDebugging } from './debug.js'
import { type FileEncoding, decodeBuffer, detectEncoding } from './encoding.js'
import { getFsImplementation, safeResolvePath } from './fsOperations.js' import { getFsImplementation, safeResolvePath } from './fsOperations.js'
export type LineEndingType = 'CRLF' | 'LF' export type LineEndingType = 'CRLF' | 'LF'
export function detectEncodingForResolvedPath( export function detectEncodingForResolvedPath(
resolvedPath: string, resolvedPath: string,
): BufferEncoding { ): FileEncoding {
const { buffer, bytesRead } = getFsImplementation().readSync(resolvedPath, { const { buffer, bytesRead } = getFsImplementation().readSync(resolvedPath, {
length: 4096, length: 4096,
}) })
// Empty files should default to utf8, not ascii // Empty files default to utf8 nothing to detect
// This fixes a bug where writing emojis/CJK to empty files caused corruption
if (bytesRead === 0) { if (bytesRead === 0) {
return 'utf8' return 'utf8'
} }
if (bytesRead >= 2) { return detectEncoding(buffer.subarray(0, bytesRead))
if (buffer[0] === 0xff && buffer[1] === 0xfe) return 'utf16le'
}
if (
bytesRead >= 3 &&
buffer[0] === 0xef &&
buffer[1] === 0xbb &&
buffer[2] === 0xbf
) {
return 'utf8'
}
// For non-empty files, default to utf8 since it's a superset of ascii
// and handles all Unicode characters properly
return 'utf8'
} }
export function detectLineEndingsForString(content: string): LineEndingType { export function detectLineEndingsForString(content: string): LineEndingType {
@@ -74,7 +59,7 @@ export function detectLineEndingsForString(content: string): LineEndingType {
*/ */
export function readFileSyncWithMetadata(filePath: string): { export function readFileSyncWithMetadata(filePath: string): {
content: string content: string
encoding: BufferEncoding encoding: FileEncoding
lineEndings: LineEndingType lineEndings: LineEndingType
} { } {
const fs = getFsImplementation() const fs = getFsImplementation()
@@ -85,10 +70,10 @@ export function readFileSyncWithMetadata(filePath: string): {
} }
const encoding = detectEncodingForResolvedPath(resolvedPath) const encoding = detectEncodingForResolvedPath(resolvedPath)
const raw = fs.readFileSync(resolvedPath, { encoding }) // Read raw Buffer first — readFileSync encoding option only accepts
// Detect line endings from the raw head before CRLF normalization erases // BufferEncoding, not gbk etc.
// the distinction. 4096 code units is ≥ detectLineEndings's 4096-byte const rawBuffer = fs.readFileBytesSync(resolvedPath)
// readSync sample (line endings are ASCII, so the unit mismatch is moot). const raw = decodeBuffer(rawBuffer, encoding)
const lineEndings = detectLineEndingsForString(raw.slice(0, 4096)) const lineEndings = detectLineEndingsForString(raw.slice(0, 4096))
return { return {
content: raw.replaceAll('\r\n', '\n'), content: raw.replaceAll('\r\n', '\n'),

View File

@@ -1,9 +1,10 @@
import { detectFileEncoding } from './file.js' import { detectFileEncoding } from './file.js'
import { type FileEncoding, decodeBuffer } from './encoding.js'
import { getFsImplementation } from './fsOperations.js' import { getFsImplementation } from './fsOperations.js'
type CachedFileData = { type CachedFileData = {
content: string content: string
encoding: BufferEncoding encoding: FileEncoding
mtime: number mtime: number
} }
@@ -19,7 +20,7 @@ class FileReadCache {
* Reads a file with caching. Returns both content and encoding. * Reads a file with caching. Returns both content and encoding.
* Cache key includes file path and modification time for automatic invalidation. * Cache key includes file path and modification time for automatic invalidation.
*/ */
readFile(filePath: string): { content: string; encoding: BufferEncoding } { readFile(filePath: string): { content: string; encoding: FileEncoding } {
const fs = getFsImplementation() const fs = getFsImplementation()
// Get file stats for cache invalidation // Get file stats for cache invalidation
@@ -45,9 +46,8 @@ class FileReadCache {
// Cache miss or stale data - read the file // Cache miss or stale data - read the file
const encoding = detectFileEncoding(filePath) const encoding = detectFileEncoding(filePath)
const content = fs const rawBuffer = fs.readFileBytesSync(filePath)
.readFileSync(filePath, { encoding }) const content = decodeBuffer(rawBuffer, encoding).replaceAll('\r\n', '\n')
.replaceAll('\r\n', '\n')
// Update cache // Update cache
this.cache.set(cacheKey, { this.cache.set(cacheKey, {

View File

@@ -26,7 +26,8 @@
// On error (including maxBytes exceeded), stream.destroy(err) emits // On error (including maxBytes exceeded), stream.destroy(err) emits
// 'error' → reject (passed directly to .once('error')). // 'error' → reject (passed directly to .once('error')).
// //
// Both paths strip UTF-8 BOM and \r (CRLF → LF). // Both paths auto-detect encoding via encoding.ts (BOM → UTF-8 fatal → fallback chain),
// decode with TextDecoder, and strip BOM and \r (CRLF → LF).
// //
// mtime comes from fstat/stat on the already-open fd — no extra open(). // mtime comes from fstat/stat on the already-open fd — no extra open().
// //
@@ -39,6 +40,7 @@
import { createReadStream, fstat } from 'fs' import { createReadStream, fstat } from 'fs'
import { stat as fsStat, readFile } from 'fs/promises' import { stat as fsStat, readFile } from 'fs/promises'
import { detectEncoding, decodeBuffer } from './encoding.js'
import { formatFileSize } from './format.js' import { formatFileSize } from './format.js'
const FAST_PATH_MAX_SIZE = 10 * 1024 * 1024 // 10 MB const FAST_PATH_MAX_SIZE = 10 * 1024 * 1024 // 10 MB
@@ -115,7 +117,9 @@ export async function readFileInRange(
) )
} }
const text = await readFile(filePath, { encoding: 'utf8', signal }) const rawBuffer = await readFile(filePath, { signal })
const encoding = detectEncoding(rawBuffer)
const text = decodeBuffer(rawBuffer, encoding)
return readFileInRangeFast( return readFileInRangeFast(
text, text,
stats.mtimeMs, stats.mtimeMs,
@@ -227,6 +231,12 @@ type StreamState = {
isFirstChunk: boolean isFirstChunk: boolean
resolveMtime: (ms: number) => void resolveMtime: (ms: number) => void
mtimeReady: Promise<number> mtimeReady: Promise<number>
/** Encoding detection state: null = not yet detected, string = detected */
encoding: string | null
/** TextDecoder instance: created after detection, used for streaming decode */
decoder: TextDecoder | null
/** Detection phase buffer: collects raw bytes until 4KB or stream end */
detectionBuffer: number[]
} }
function streamOnOpen(this: StreamState, fd: number): void { function streamOnOpen(this: StreamState, fd: number): void {
@@ -235,15 +245,71 @@ function streamOnOpen(this: StreamState, fd: number): void {
}) })
} }
function streamOnData(this: StreamState, chunk: string): void { function processTextChunk(state: StreamState, text: string): void {
if (this.isFirstChunk) { // BOM stripping (first chunk only)
this.isFirstChunk = false if (state.isFirstChunk) {
if (chunk.charCodeAt(0) === 0xfeff) { state.isFirstChunk = false
chunk = chunk.slice(1) if (text.charCodeAt(0) === 0xfeff) {
text = text.slice(1)
} }
} }
this.totalBytesRead += Buffer.byteLength(chunk) const data = state.partial.length > 0 ? state.partial + text : text
state.partial = ''
let startPos = 0
let newlinePos: number
while ((newlinePos = data.indexOf('\n', startPos)) !== -1) {
if (
state.currentLineIndex >= state.offset &&
state.currentLineIndex < state.endLine
) {
let line = data.slice(startPos, newlinePos)
if (line.endsWith('\r')) {
line = line.slice(0, -1)
}
if (state.truncateOnByteLimit && state.maxBytes !== undefined) {
const sep = state.selectedLines.length > 0 ? 1 : 0
const nextBytes = state.selectedBytes + sep + Buffer.byteLength(line)
if (nextBytes > state.maxBytes) {
state.truncatedByBytes = true
state.endLine = state.currentLineIndex
} else {
state.selectedBytes = nextBytes
state.selectedLines.push(line)
}
} else {
state.selectedLines.push(line)
}
}
state.currentLineIndex++
startPos = newlinePos + 1
}
if (startPos < data.length) {
if (
state.currentLineIndex >= state.offset &&
state.currentLineIndex < state.endLine
) {
const fragment = data.slice(startPos)
if (state.truncateOnByteLimit && state.maxBytes !== undefined) {
const sep = state.selectedLines.length > 0 ? 1 : 0
const fragBytes =
state.selectedBytes + sep + Buffer.byteLength(fragment)
if (fragBytes > state.maxBytes) {
state.truncatedByBytes = true
state.endLine = state.currentLineIndex
return
}
}
state.partial = fragment
}
}
}
function streamOnData(this: StreamState, chunk: Buffer): void {
this.totalBytesRead += chunk.length
if ( if (
!this.truncateOnByteLimit && !this.truncateOnByteLimit &&
this.maxBytes !== undefined && this.maxBytes !== undefined &&
@@ -255,69 +321,47 @@ function streamOnData(this: StreamState, chunk: string): void {
return return
} }
const data = this.partial.length > 0 ? this.partial + chunk : chunk // Phase 1: Encoding detection
this.partial = '' if (this.encoding === null) {
for (let i = 0; i < chunk.length; i++) {
let startPos = 0 this.detectionBuffer.push(chunk[i])
let newlinePos: number
while ((newlinePos = data.indexOf('\n', startPos)) !== -1) {
if (
this.currentLineIndex >= this.offset &&
this.currentLineIndex < this.endLine
) {
let line = data.slice(startPos, newlinePos)
if (line.endsWith('\r')) {
line = line.slice(0, -1)
}
if (this.truncateOnByteLimit && this.maxBytes !== undefined) {
const sep = this.selectedLines.length > 0 ? 1 : 0
const nextBytes = this.selectedBytes + sep + Buffer.byteLength(line)
if (nextBytes > this.maxBytes) {
// Cap hit — collapse the selection range so nothing more is
// accumulated. Stream continues (to count totalLines).
this.truncatedByBytes = true
this.endLine = this.currentLineIndex
} else {
this.selectedBytes = nextBytes
this.selectedLines.push(line)
}
} else {
this.selectedLines.push(line)
}
} }
this.currentLineIndex++
startPos = newlinePos + 1 // Collected at least 4KB, perform encoding detection
if (this.detectionBuffer.length >= 4096) {
this.encoding = detectEncoding(Buffer.from(this.detectionBuffer))
this.decoder = new TextDecoder(this.encoding, {
stream: true,
} as TextDecoderOptions)
// Decode the detection buffer and feed to line scanning
const decoded = this.decoder.decode(Buffer.from(this.detectionBuffer))
this.detectionBuffer = []
processTextChunk(this, decoded)
}
return
} }
// Only keep the trailing fragment when inside the selected range. // Phase 2: Decoding
// Outside the range we just count newlines — discarding prevents const decoded = this.decoder!.decode(chunk, {
// unbounded memory growth on huge single-line files. stream: true,
if (startPos < data.length) { } as unknown as TextDecodeOptions)
if ( processTextChunk(this, decoded)
this.currentLineIndex >= this.offset &&
this.currentLineIndex < this.endLine
) {
const fragment = data.slice(startPos)
// In truncate mode, `partial` can grow unboundedly if the selected
// range contains a huge single line (no newline across many chunks).
// Once the fragment alone would overflow the remaining budget, we know
// the completed line can never fit — set truncated, collapse the
// selection range, and discard the fragment to stop accumulation.
if (this.truncateOnByteLimit && this.maxBytes !== undefined) {
const sep = this.selectedLines.length > 0 ? 1 : 0
const fragBytes = this.selectedBytes + sep + Buffer.byteLength(fragment)
if (fragBytes > this.maxBytes) {
this.truncatedByBytes = true
this.endLine = this.currentLineIndex
return
}
}
this.partial = fragment
}
}
} }
function streamOnEnd(this: StreamState): void { function streamOnEnd(this: StreamState): void {
// If stream ended before detection completed (< 4KB file), detect now
if (this.encoding === null) {
this.encoding = detectEncoding(Buffer.from(this.detectionBuffer))
this.decoder = new TextDecoder(this.encoding, {
stream: true,
} as TextDecoderOptions)
const decoded = this.decoder.decode(Buffer.from(this.detectionBuffer))
this.detectionBuffer = []
processTextChunk(this, decoded)
}
// Handle final fragment
let line = this.partial let line = this.partial
if (line.endsWith('\r')) { if (line.endsWith('\r')) {
line = line.slice(0, -1) line = line.slice(0, -1)
@@ -366,7 +410,6 @@ function readFileInRangeStreaming(
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const state: StreamState = { const state: StreamState = {
stream: createReadStream(filePath, { stream: createReadStream(filePath, {
encoding: 'utf8',
highWaterMark: 512 * 1024, highWaterMark: 512 * 1024,
...(signal ? { signal } : undefined), ...(signal ? { signal } : undefined),
}), }),
@@ -384,6 +427,9 @@ function readFileInRangeStreaming(
isFirstChunk: true, isFirstChunk: true,
resolveMtime: () => {}, resolveMtime: () => {},
mtimeReady: null as unknown as Promise<number>, mtimeReady: null as unknown as Promise<number>,
encoding: null,
decoder: null,
detectionBuffer: [],
} }
state.mtimeReady = new Promise<number>(r => { state.mtimeReady = new Promise<number>(r => {
state.resolveMtime = r state.resolveMtime = r