AI

A seemingly small OpenClaw error can occasionally expose several layers of state that have accumulated over time: stale Codex session bindings, retired model references, legacy media metadata, database schema migrations, old plugin generations, and outdated systemd service definitions.

This article documents one such recovery on an Ubuntu host. All hostnames, usernames, domains, IP addresses, ports, account identifiers, session identifiers, and personally identifying paths have been removed or replaced with generic placeholders.

The final result was a healthy OpenClaw installation running version 2026.9.3, with a valid schema-16 database, working Gateway, current plugins, corrected systemd configuration, and no recurrence of the original Codex runner conflict.

The Initial Failure

The incident started with an OpenClaw task immediately failing with:

Codex session became active in another runner;
wait for it to finish before continuing

The same workload had been functioning normally before the failure appeared.

At the time, the installation was approximately:

OpenClaw 2026.9.1
Node.js v24.x
Ubuntu 24.04 LTS

Restarting the entire operating system did not solve the problem.

After reboot, the Gateway itself looked healthy:

Runtime: running
Connectivity probe: ok

That indicated that the failure was not simply a dead Gateway process or an abandoned operating-system process.

The problem was instead related to OpenClaw’s persistent session and state layer.

Attempting an OpenClaw Upgrade

Because newer OpenClaw releases contained additional Codex/session handling improvements, the next logical step was to upgrade from:

2026.9.1

to:

2026.9.3

The normal upgrade command was started:

openclaw update

The package-manager phase succeeded, but the Doctor phase stopped during state migration.

The important error was:

Failed reading legacy managed outgoing image state:
Error: legacy managed image original is outside managed outgoing storage

This was not a Node.js installation failure, network failure, or package-manager failure.

It was a legacy-state migration problem.

Legacy Managed-Image Metadata

Inspection showed that OpenClaw still had old JSON records representing managed outgoing images.

A typical record contained metadata similar to:

{
  "attachmentId": "<UUID>",
  "sessionKey": "<SESSION>",
  "original": {
    "path": "/home/<OLD_USER>/.openclaw/media/outgoing/originals/<UUID>.png"
  }
}

The current OpenClaw installation, however, was running under a different state location:

<OPENCLAW_STATE_DIR>/media/outgoing/originals/

Most of the old records pointed to files under an obsolete user’s home directory.

Further inspection established that:

37 records -> obsolete paths, original image already missing
3 records  -> valid current paths, original image still present

OpenClaw’s migration code intentionally rejected the invalid records rather than silently importing metadata referencing files outside the managed storage root.

That behavior is conservative but correct: state migration should fail closed rather than silently corrupt attachment ownership.

Quarantining Only Proven-Orphaned Records

Rather than deleting the entire media state or modifying SQLite manually, only the records that met all of the following conditions were removed from active migration input:

original path outside the current managed-media root
AND
referenced original file does not exist
AND
no corresponding current managed file exists

Before making the change, the complete legacy-record directory was copied to a temporary recovery location.

The invalid records were then moved into a quarantine directory, leaving the three valid records untouched.

The result was:

Moved orphaned  : 37
Kept valid      : 3
Need attention  : 0

No active database rows or current attachments were deleted.

An Interrupted Upgrade Created a Schema-Version Mismatch

The next complication was more interesting.

During the failed update, OpenClaw had already migrated its shared SQLite database from:

schema 15

to:

schema 16

But because Doctor later failed, the updater restored the installed OpenClaw package to version 2026.9.1.

The result was:

Database schema: 16
Installed OpenClaw supports: 15

OpenClaw correctly refused to start against a database newer than the binary understood:

This OpenClaw build cannot open your existing data.

OpenClaw state database uses newer schema version 16;
this build supports 15.

At this point, repeatedly running:

openclaw update

was no longer useful because the old binary could not even safely open the current state database.

The correct recovery direction was forward, not backward.

Manually Installing the Compatible OpenClaw Version

The compatible release was installed directly with npm while the Gateway remained stopped.

Conceptually:

npm install -g openclaw@2026.9.3

After installation:

openclaw --version

reported:

OpenClaw 2026.9.3

This restored binary/database compatibility.

The next step was:

openclaw doctor --fix

Doctor successfully migrated the three valid managed outgoing image records:

Migrated 3 managed outgoing image record(s) → shared SQLite state
Removed legacy managed outgoing image JSON after SQLite verification

That confirmed that the earlier image-state problem had been resolved correctly.

A Second Migration Blocker: Skill Workshop

Doctor then encountered another legacy-state problem:

Failed to migrate Skill Workshop proposal:
proposal draft hash does not match proposal metadata

An old Skill Workshop proposal contained:

PROPOSAL.md
proposal.json
rollback.json

but the stored proposal metadata did not match the current draft hash.

Because the proposal could not be verified safely, Doctor refused to continue migrating state.

The affected legacy proposal was removed from the active proposal-discovery location and placed into a recovery directory instead of editing its metadata or fabricating a matching hash.

Afterward, Doctor was run again.

This time it completed:

Doctor complete.

The temporary recovery copy was later deleted after the installation had been verified healthy and the old proposal was no longer required.

The Important Codex Repair

During the successful Doctor run, one line was especially relevant to the original problem:

Removed 1 orphaned Codex app-server session binding(s)

This was the strongest direct indication that persistent Codex session state had contributed to the original:

Codex session became active in another runner

failure.

A reboot alone had not solved the problem because the bad relationship was not merely an in-memory process lock. It existed in persistent OpenClaw state.

Doctor repaired that state.

Model Migration: gpt-5.4-mini to gpt-5.6-luna

There was also a second independent issue visible in the old Gateway logs.

Requests using the former default model produced:

The 'gpt-5.4-mini' model is not supported when using Codex with a ChatGPT account.

The old primary model had been:

openai/gpt-5.4-mini

OpenClaw 2026.9.3 automatically migrated the default to:

openai/gpt-5.6-luna

with a fallback such as:

deepseek/deepseek-v4-flash

Doctor explicitly reported the migration:

Replaced retired agents.defaults.model.primary
"openai/gpt-5.4-mini"
with
"openai/gpt-5.6-luna".

Residual active references to gpt-5.4-mini were subsequently removed.

The resulting model status was approximately:

Default:
  openai/gpt-5.6-luna

Fallback:
  deepseek/deepseek-v4-flash

OpenAI runtime:
  Codex
  authentication usable
  no route issues

A separate openai/gpt-5.4 alias remained in configuration, but it was not the retired gpt-5.4-mini model and was not the active default.

Plugin Version Alignment

Doctor initially warned that several official plugins were still associated with the previous OpenClaw generation.

After repair and restart, the important enabled plugins were verified as:

Codex             2026.9.3
DeepSeek          2026.9.3
DuckDuckGo        2026.9.3
Nextcloud Talk    2026.9.3

There were no remaining 2026.9.1 versions among those active plugins.

This mattered because running a newer core with stale managed plugin generations could otherwise introduce confusing compatibility failures.

Fixing the systemd Gateway Service

The upgraded OpenClaw binary also detected that its systemd unit was outdated.

The old service contained:

KillMode=control-group

OpenClaw 2026.9.3 required:

KillMode=mixed

The reason is operationally important.

With KillMode=mixed, OpenClaw can first allow active work to drain cleanly before systemd performs final child-process cleanup.

The service was regenerated using OpenClaw’s supported installer:

openclaw gateway install --force

Afterward:

systemctl --user show openclaw-gateway.service -p KillMode

returned:

KillMode=mixed

An existing memory-limit drop-in was preserved:

[Service]
MemoryHigh=5G
MemoryMax=6G

This demonstrated why operator-owned systemd drop-ins are preferable to manually editing generated service units.

Why Gateway Restart Initially Appeared to Fail

Starting or restarting the Gateway sometimes produced:

Gateway start timed out after 45s waiting for /healthz and /readyz

This looked alarming but was not actually a failed start.

Logs showed approximately:

systemd service started
...
Gateway loading configuration
...
database opened
...
HTTP server listening
...
Gateway ready

The host simply needed roughly one minute to become fully ready, while the CLI waited only 45 seconds.

The actual runtime state afterward was:

Runtime: running
Connectivity probe: ok

So the timeout was a readiness-wait timeout, not a service crash.

Controlled Restart and Drain Behavior

Once KillMode=mixed was active, a later Gateway restart entered a controlled drain:

pendingReplies=1
embeddedRuns=1

During this phase, temporary HTTP 503 responses and transport loss were expected.

Eventually the client reported:

gateway reconnected after transport loss

A subsequent inspection showed a new Gateway PID, zero restart-loop count, and healthy runtime state.

This confirmed that the managed restart mechanism was functioning correctly.

Database Verification

After all migrations and restarts, the shared SQLite database was checked directly in read-only mode.

Schema:

PRAGMA user_version;

Result:

16

Integrity:

PRAGMA integrity_check;

Result:

ok

This was an important final check because the recovery had crossed a database migration boundary.

No attempt was made to manually change SQLite schema-version metadata.

Verifying the Original Error Did Not Return

The most useful log check was not to search the entire historical log indiscriminately.

Instead, logs were scoped to the startup time of the new Gateway process.

Only events after the repaired Gateway started were examined for:

Codex session became active in another runner
model_not_found
orphaned binding
schema errors
database failures
plugin failures
fatal
panic
crash

No new occurrences of the original Codex runner error appeared.

The new process showed normal activity such as:

sessions.list
chat.history
models.authStatus
talk.catalog

with successful responses.

The old active in another runner entries remained in historical logs, but they belonged to the previous Gateway process and were therefore not evidence of a current fault.

This distinction between historical errors and current errors is essential during long-running troubleshooting sessions.

Backup Hygiene

The repair also exposed an important operational issue: automatically generated backup names do not always match a desired archival convention.

The preferred long-term convention was:

original-filename.YYYYMMDD-HHMMSS

Examples:

openclaw.json.20260909-123456
openclaw-gateway.service.20260909-123456

Names such as:

*.bak
*.old
*.orig
*.save

were not considered valid long-term archive names.

When OpenClaw generated a .bak file, it was compared against an existing timestamped backup using SHA-256.

If both files were byte-for-byte identical, the non-standard duplicate could be removed safely while retaining the timestamped archive.

Temporary diagnostic and recovery data created during troubleshooting were also removed after successful recovery, including temporary migration copies and support/triage artifacts that were no longer needed.

Historical timestamped backups were preserved.

Final State

The repaired installation ended in approximately the following state:

OpenClaw:          2026.9.3
Node.js:           24.x
SQLite schema:     16
SQLite integrity:  ok

Gateway:
  active
  running
  connectivity probe: ok

systemd:
  KillMode=mixed
  MemoryHigh=5G
  MemoryMax=6G

Default model:
  openai/gpt-5.6-luna

Fallback:
  deepseek/deepseek-v4-flash

Key plugins:
  Codex             2026.9.3
  DeepSeek          2026.9.3
  DuckDuckGo        2026.9.3
  Nextcloud Talk    2026.9.3

Codex:
  orphaned app-server binding repaired
  no new "active in another runner" errors observed

Lessons From the Incident

The main lesson was that several superficially unrelated errors were actually consequences of persistent state accumulated across OpenClaw versions.

The original Codex error was not solved by rebooting because at least part of the problem existed in persisted application state rather than merely in running processes.

The upgrade then exposed old media metadata and a malformed Skill Workshop proposal because the newer Doctor performed stricter migrations.

The failed migration temporarily left the database newer than the installed binary, making a forward manual package installation necessary.

Once the correct binary was installed, the migration blockers were isolated rather than destructively removed, Doctor completed, the orphaned Codex binding was repaired, retired model configuration was migrated, and the systemd service definition was brought in line with the current release.

The broader operational principle is simple:

Repair the smallest proven fault, preserve recoverable state, distinguish historical errors from current ones, and stop troubleshooting once the service is demonstrably healthy.

A working system does not become healthier merely because more commands are run against it.

Leave a Reply

Your email address will not be published. Required fields are marked *