{"id":6201,"date":"2026-09-11T22:31:44","date_gmt":"2026-09-11T13:31:44","guid":{"rendered":"https:\/\/eternalsphere.net\/echoes\/?p=6201"},"modified":"2026-09-12T17:18:05","modified_gmt":"2026-09-12T08:18:05","slug":"omp9rse74ebt0iy","status":"publish","type":"post","link":"https:\/\/blog.eternalsphere.net\/index.php\/2026\/09\/11\/omp9rse74ebt0iy\/","title":{"rendered":"Recovering an OpenClaw Installation After a Codex Session Conflict and Interrupted State Migration"},"content":{"rendered":"\n

A seemingly small OpenClaw error can occasionally expose several layers of state that have accumulated over time: stale Codex session bindings, retired model references, legacy media metadata, database schema migrations, old plugin generations, and outdated systemd service definitions.<\/p>\n\n\n\n

This article documents one such recovery on an Ubuntu host. All hostnames, usernames, domains, IP addresses, ports, account identifiers, session identifiers, and personally identifying paths have been removed or replaced with generic placeholders.<\/p>\n\n\n\n

The final result was a healthy OpenClaw installation running version 2026.9.3<\/code>, with a valid schema-16 database, working Gateway, current plugins, corrected systemd configuration, and no recurrence of the original Codex runner conflict.<\/p>\n\n\n\n

\n\n\n\n

The Initial Failure<\/h2>\n\n\n\n

The incident started with an OpenClaw task immediately failing with:<\/p>\n\n\n\n

Codex session became active in another runner;\nwait for it to finish before continuing<\/code><\/pre>\n\n\n\n
The same workload had been functioning normally before the failure appeared.<\/p>\n\n\n\n
At the time, the installation was approximately:<\/p>\n\n\n\n
OpenClaw 2026.9.1\nNode.js v24.x\nUbuntu 24.04 LTS<\/code><\/pre>\n\n\n\n
Restarting the entire operating system did not solve the problem.<\/p>\n\n\n\n
After reboot, the Gateway itself looked healthy:<\/p>\n\n\n\n
Runtime: running\nConnectivity probe: ok<\/code><\/pre>\n\n\n\n
That indicated that the failure was not simply a dead Gateway process or an abandoned operating-system process.<\/p>\n\n\n\n
The problem was instead related to OpenClaw’s persistent session and state layer.<\/p>\n\n\n\n
\n\n\n\n
Attempting an OpenClaw Upgrade<\/h2>\n\n\n\n
Because newer OpenClaw releases contained additional Codex\/session handling improvements, the next logical step was to upgrade from:<\/p>\n\n\n\n
2026.9.1<\/code><\/pre>\n\n\n\n
to:<\/p>\n\n\n\n
2026.9.3<\/code><\/pre>\n\n\n\n
The normal upgrade command was started:<\/p>\n\n\n\n
openclaw update<\/code><\/pre>\n\n\n\n
The package-manager phase succeeded, but the Doctor phase stopped during state migration.<\/p>\n\n\n\n
The important error was:<\/p>\n\n\n\n
Failed reading legacy managed outgoing image state:\nError: legacy managed image original is outside managed outgoing storage<\/code><\/pre>\n\n\n\n
This was not a Node.js installation failure, network failure, or package-manager failure.<\/p>\n\n\n\n
It was a legacy-state migration problem.<\/p>\n\n\n\n
\n\n\n\n
Legacy Managed-Image Metadata<\/h2>\n\n\n\n
Inspection showed that OpenClaw still had old JSON records representing managed outgoing images.<\/p>\n\n\n\n
A typical record contained metadata similar to:<\/p>\n\n\n\n
{\n  \"attachmentId\": \"<UUID>\",\n  \"sessionKey\": \"<SESSION>\",\n  \"original\": {\n    \"path\": \"\/home\/<OLD_USER>\/.openclaw\/media\/outgoing\/originals\/<UUID>.png\"\n  }\n}<\/code><\/pre>\n\n\n\n
The current OpenClaw installation, however, was running under a different state location:<\/p>\n\n\n\n
<OPENCLAW_STATE_DIR>\/media\/outgoing\/originals\/<\/code><\/pre>\n\n\n\n
Most of the old records pointed to files under an obsolete user’s home directory.<\/p>\n\n\n\n
Further inspection established that:<\/p>\n\n\n\n
37 records -> obsolete paths, original image already missing\n3 records  -> valid current paths, original image still present<\/code><\/pre>\n\n\n\n
OpenClaw’s migration code intentionally rejected the invalid records rather than silently importing metadata referencing files outside the managed storage root.<\/p>\n\n\n\n
That behavior is conservative but correct: state migration should fail closed rather than silently corrupt attachment ownership.<\/p>\n\n\n\n
\n\n\n\n
Quarantining Only Proven-Orphaned Records<\/h2>\n\n\n\n
Rather than deleting the entire media state or modifying SQLite manually, only the records that met all of the following conditions were removed from active migration input:<\/p>\n\n\n\n
original path outside the current managed-media root\nAND\nreferenced original file does not exist\nAND\nno corresponding current managed file exists<\/code><\/pre>\n\n\n\n
Before making the change, the complete legacy-record directory was copied to a temporary recovery location.<\/p>\n\n\n\n
The invalid records were then moved into a quarantine directory, leaving the three valid records untouched.<\/p>\n\n\n\n
The result was:<\/p>\n\n\n\n
Moved orphaned  : 37\nKept valid      : 3\nNeed attention  : 0<\/code><\/pre>\n\n\n\n
No active database rows or current attachments were deleted.<\/p>\n\n\n\n
\n\n\n\n
An Interrupted Upgrade Created a Schema-Version Mismatch<\/h2>\n\n\n\n
The next complication was more interesting.<\/p>\n\n\n\n
During the failed update, OpenClaw had already migrated its shared SQLite database from:<\/p>\n\n\n\n
schema 15<\/code><\/pre>\n\n\n\n
to:<\/p>\n\n\n\n
schema 16<\/code><\/pre>\n\n\n\n
But because Doctor later failed, the updater restored the installed OpenClaw package to version 2026.9.1<\/code>.<\/p>\n\n\n\n
The result was:<\/p>\n\n\n\n
Database schema: 16\nInstalled OpenClaw supports: 15<\/code><\/pre>\n\n\n\n
OpenClaw correctly refused to start against a database newer than the binary understood:<\/p>\n\n\n\n
This OpenClaw build cannot open your existing data.\n\nOpenClaw state database uses newer schema version 16;\nthis build supports 15.<\/code><\/pre>\n\n\n\n
At this point, repeatedly running:<\/p>\n\n\n\n
openclaw update<\/code><\/pre>\n\n\n\n
was no longer useful because the old binary could not even safely open the current state database.<\/p>\n\n\n\n
The correct recovery direction was forward, not backward.<\/p>\n\n\n\n
\n\n\n\n
Manually Installing the Compatible OpenClaw Version<\/h2>\n\n\n\n
The compatible release was installed directly with npm while the Gateway remained stopped.<\/p>\n\n\n\n
Conceptually:<\/p>\n\n\n\n
npm install -g openclaw@2026.9.3<\/code><\/pre>\n\n\n\n
After installation:<\/p>\n\n\n\n
openclaw --version<\/code><\/pre>\n\n\n\n
reported:<\/p>\n\n\n\n
OpenClaw 2026.9.3<\/code><\/pre>\n\n\n\n
This restored binary\/database compatibility.<\/p>\n\n\n\n
The next step was:<\/p>\n\n\n\n
openclaw doctor --fix<\/code><\/pre>\n\n\n\n
Doctor successfully migrated the three valid managed outgoing image records:<\/p>\n\n\n\n
Migrated 3 managed outgoing image record(s) \u2192 shared SQLite state\nRemoved legacy managed outgoing image JSON after SQLite verification<\/code><\/pre>\n\n\n\n
That confirmed that the earlier image-state problem had been resolved correctly.<\/p>\n\n\n\n
\n\n\n\n
A Second Migration Blocker: Skill Workshop<\/h2>\n\n\n\n
Doctor then encountered another legacy-state problem:<\/p>\n\n\n\n
Failed to migrate Skill Workshop proposal:\nproposal draft hash does not match proposal metadata<\/code><\/pre>\n\n\n\n
An old Skill Workshop proposal contained:<\/p>\n\n\n\n
PROPOSAL.md\nproposal.json\nrollback.json<\/code><\/pre>\n\n\n\n
but the stored proposal metadata did not match the current draft hash.<\/p>\n\n\n\n
Because the proposal could not be verified safely, Doctor refused to continue migrating state.<\/p>\n\n\n\n
The affected legacy proposal was removed from the active proposal-discovery location and placed into a recovery directory instead of editing its metadata or fabricating a matching hash.<\/p>\n\n\n\n
Afterward, Doctor was run again.<\/p>\n\n\n\n
This time it completed:<\/p>\n\n\n\n
Doctor complete.<\/code><\/pre>\n\n\n\n
The temporary recovery copy was later deleted after the installation had been verified healthy and the old proposal was no longer required.<\/p>\n\n\n\n
\n\n\n\n
The Important Codex Repair<\/h2>\n\n\n\n
During the successful Doctor run, one line was especially relevant to the original problem:<\/p>\n\n\n\n
Removed 1 orphaned Codex app-server session binding(s)<\/code><\/pre>\n\n\n\n
This was the strongest direct indication that persistent Codex session state had contributed to the original:<\/p>\n\n\n\n
Codex session became active in another runner<\/code><\/pre>\n\n\n\n
failure.<\/p>\n\n\n\n
A reboot alone had not solved the problem because the bad relationship was not merely an in-memory process lock. It existed in persistent OpenClaw state.<\/p>\n\n\n\n
Doctor repaired that state.<\/p>\n\n\n\n
\n\n\n\n
Model Migration: gpt-5.4-mini<\/code> to gpt-5.6-luna<\/code><\/h2>\n\n\n\n
There was also a second independent issue visible in the old Gateway logs.<\/p>\n\n\n\n
Requests using the former default model produced:<\/p>\n\n\n\n
The 'gpt-5.4-mini' model is not supported when using Codex with a ChatGPT account.<\/code><\/pre>\n\n\n\n
The old primary model had been:<\/p>\n\n\n\n
openai\/gpt-5.4-mini<\/code><\/pre>\n\n\n\n
OpenClaw 2026.9.3 automatically migrated the default to:<\/p>\n\n\n\n
openai\/gpt-5.6-luna<\/code><\/pre>\n\n\n\n
with a fallback such as:<\/p>\n\n\n\n
deepseek\/deepseek-v4-flash<\/code><\/pre>\n\n\n\n
Doctor explicitly reported the migration:<\/p>\n\n\n\n
Replaced retired agents.defaults.model.primary\n\"openai\/gpt-5.4-mini\"\nwith\n\"openai\/gpt-5.6-luna\".<\/code><\/pre>\n\n\n\n
Residual active references to gpt-5.4-mini<\/code> were subsequently removed.<\/p>\n\n\n\n
The resulting model status was approximately:<\/p>\n\n\n\n
Default:\n  openai\/gpt-5.6-luna\n\nFallback:\n  deepseek\/deepseek-v4-flash\n\nOpenAI runtime:\n  Codex\n  authentication usable\n  no route issues<\/code><\/pre>\n\n\n\n
A separate openai\/gpt-5.4<\/code> alias remained in configuration, but it was not the retired gpt-5.4-mini<\/code> model and was not the active default.<\/p>\n\n\n\n
\n\n\n\n
Plugin Version Alignment<\/h2>\n\n\n\n
Doctor initially warned that several official plugins were still associated with the previous OpenClaw generation.<\/p>\n\n\n\n
After repair and restart, the important enabled plugins were verified as:<\/p>\n\n\n\n
Codex             2026.9.3\nDeepSeek          2026.9.3\nDuckDuckGo        2026.9.3\nNextcloud Talk    2026.9.3<\/code><\/pre>\n\n\n\n
There were no remaining 2026.9.1<\/code> versions among those active plugins.<\/p>\n\n\n\n
This mattered because running a newer core with stale managed plugin generations could otherwise introduce confusing compatibility failures.<\/p>\n\n\n\n
\n\n\n\n
Fixing the systemd Gateway Service<\/h2>\n\n\n\n
The upgraded OpenClaw binary also detected that its systemd unit was outdated.<\/p>\n\n\n\n
The old service contained:<\/p>\n\n\n\n
KillMode=control-group<\/code><\/pre>\n\n\n\n
OpenClaw 2026.9.3 required:<\/p>\n\n\n\n
KillMode=mixed<\/code><\/pre>\n\n\n\n
The reason is operationally important.<\/p>\n\n\n\n
With KillMode=mixed<\/code>, OpenClaw can first allow active work to drain cleanly before systemd performs final child-process cleanup.<\/p>\n\n\n\n
The service was regenerated using OpenClaw’s supported installer:<\/p>\n\n\n\n
openclaw gateway install --force<\/code><\/pre>\n\n\n\n
Afterward:<\/p>\n\n\n\n
systemctl --user show openclaw-gateway.service -p KillMode<\/code><\/pre>\n\n\n\n
returned:<\/p>\n\n\n\n
KillMode=mixed<\/code><\/pre>\n\n\n\n
An existing memory-limit drop-in was preserved:<\/p>\n\n\n\n
[Service]\nMemoryHigh=5G\nMemoryMax=6G<\/code><\/pre>\n\n\n\n
This demonstrated why operator-owned systemd drop-ins are preferable to manually editing generated service units.<\/p>\n\n\n\n
\n\n\n\n
Why Gateway Restart Initially Appeared to Fail<\/h2>\n\n\n\n
Starting or restarting the Gateway sometimes produced:<\/p>\n\n\n\n
Gateway start timed out after 45s waiting for \/healthz and \/readyz<\/code><\/pre>\n\n\n\n
This looked alarming but was not actually a failed start.<\/p>\n\n\n\n
Logs showed approximately:<\/p>\n\n\n\n
systemd service started\n...\nGateway loading configuration\n...\ndatabase opened\n...\nHTTP server listening\n...\nGateway ready<\/code><\/pre>\n\n\n\n
The host simply needed roughly one minute to become fully ready, while the CLI waited only 45 seconds.<\/p>\n\n\n\n
The actual runtime state afterward was:<\/p>\n\n\n\n
Runtime: running\nConnectivity probe: ok<\/code><\/pre>\n\n\n\n
So the timeout was a readiness-wait timeout, not a service crash.<\/p>\n\n\n\n
\n\n\n\n
Controlled Restart and Drain Behavior<\/h2>\n\n\n\n
Once KillMode=mixed<\/code> was active, a later Gateway restart entered a controlled drain:<\/p>\n\n\n\n
pendingReplies=1\nembeddedRuns=1<\/code><\/pre>\n\n\n\n
During this phase, temporary HTTP 503 responses and transport loss were expected.<\/p>\n\n\n\n
Eventually the client reported:<\/p>\n\n\n\n
gateway reconnected after transport loss<\/code><\/pre>\n\n\n\n
A subsequent inspection showed a new Gateway PID, zero restart-loop count, and healthy runtime state.<\/p>\n\n\n\n
This confirmed that the managed restart mechanism was functioning correctly.<\/p>\n\n\n\n
\n\n\n\n
Database Verification<\/h2>\n\n\n\n
After all migrations and restarts, the shared SQLite database was checked directly in read-only mode.<\/p>\n\n\n\n
Schema:<\/p>\n\n\n\n
PRAGMA user_version;<\/code><\/pre>\n\n\n\n
Result:<\/p>\n\n\n\n
16<\/code><\/pre>\n\n\n\n
Integrity:<\/p>\n\n\n\n
PRAGMA integrity_check;<\/code><\/pre>\n\n\n\n
Result:<\/p>\n\n\n\n
ok<\/code><\/pre>\n\n\n\n
This was an important final check because the recovery had crossed a database migration boundary.<\/p>\n\n\n\n
No attempt was made to manually change SQLite schema-version metadata.<\/p>\n\n\n\n
\n\n\n\n
Verifying the Original Error Did Not Return<\/h2>\n\n\n\n
The most useful log check was not to search the entire historical log indiscriminately.<\/p>\n\n\n\n
Instead, logs were scoped to the startup time of the new Gateway process.<\/p>\n\n\n\n
Only events after the repaired Gateway started were examined for:<\/p>\n\n\n\n
Codex session became active in another runner\nmodel_not_found\norphaned binding\nschema errors\ndatabase failures\nplugin failures\nfatal\npanic\ncrash<\/code><\/pre>\n\n\n\n
No new occurrences of the original Codex runner error appeared.<\/p>\n\n\n\n
The new process showed normal activity such as:<\/p>\n\n\n\n
sessions.list\nchat.history\nmodels.authStatus\ntalk.catalog<\/code><\/pre>\n\n\n\n
with successful responses.<\/p>\n\n\n\n
The old active in another runner<\/code> entries remained in historical logs, but they belonged to the previous Gateway process and were therefore not evidence of a current fault.<\/p>\n\n\n\n
This distinction between historical errors<\/strong> and current errors<\/strong> is essential during long-running troubleshooting sessions.<\/p>\n\n\n\n
\n\n\n\n
Backup Hygiene<\/h2>\n\n\n\n
The repair also exposed an important operational issue: automatically generated backup names do not always match a desired archival convention.<\/p>\n\n\n\n
The preferred long-term convention was:<\/p>\n\n\n\n
original-filename.YYYYMMDD-HHMMSS<\/code><\/pre>\n\n\n\n
Examples:<\/p>\n\n\n\n
openclaw.json.20260909-123456\nopenclaw-gateway.service.20260909-123456<\/code><\/pre>\n\n\n\n
Names such as:<\/p>\n\n\n\n
*.bak\n*.old\n*.orig\n*.save<\/code><\/pre>\n\n\n\n
were not considered valid long-term archive names.<\/p>\n\n\n\n
When OpenClaw generated a .bak<\/code> file, it was compared against an existing timestamped backup using SHA-256.<\/p>\n\n\n\n
If both files were byte-for-byte identical, the non-standard duplicate could be removed safely while retaining the timestamped archive.<\/p>\n\n\n\n
Temporary diagnostic and recovery data created during troubleshooting were also removed after successful recovery, including temporary migration copies and support\/triage artifacts that were no longer needed.<\/p>\n\n\n\n
Historical timestamped backups were preserved.<\/p>\n\n\n\n
\n\n\n\n
Final State<\/h2>\n\n\n\n
The repaired installation ended in approximately the following state:<\/p>\n\n\n\n
OpenClaw:          2026.9.3\nNode.js:           24.x\nSQLite schema:     16\nSQLite integrity:  ok\n\nGateway:\n  active\n  running\n  connectivity probe: ok\n\nsystemd:\n  KillMode=mixed\n  MemoryHigh=5G\n  MemoryMax=6G\n\nDefault model:\n  openai\/gpt-5.6-luna\n\nFallback:\n  deepseek\/deepseek-v4-flash\n\nKey plugins:\n  Codex             2026.9.3\n  DeepSeek          2026.9.3\n  DuckDuckGo        2026.9.3\n  Nextcloud Talk    2026.9.3\n\nCodex:\n  orphaned app-server binding repaired\n  no new \"active in another runner\" errors observed<\/code><\/pre>\n\n\n\n
\n\n\n\n
Lessons From the Incident<\/h2>\n\n\n\n
The main lesson was that several superficially unrelated errors were actually consequences of persistent state accumulated across OpenClaw versions<\/strong>.<\/p>\n\n\n\n
The original Codex error was not solved by rebooting because at least part of the problem existed in persisted application state rather than merely in running processes.<\/p>\n\n\n\n
The upgrade then exposed old media metadata and a malformed Skill Workshop proposal because the newer Doctor performed stricter migrations.<\/p>\n\n\n\n
The failed migration temporarily left the database newer than the installed binary, making a forward manual package installation necessary.<\/p>\n\n\n\n
Once the correct binary was installed, the migration blockers were isolated rather than destructively removed, Doctor completed, the orphaned Codex binding was repaired, retired model configuration was migrated, and the systemd service definition was brought in line with the current release.<\/p>\n\n\n\n
The broader operational principle is simple:<\/p>\n\n\n\n
\n
Repair the smallest proven fault, preserve recoverable state, distinguish historical errors from current ones, and stop troubleshooting once the service is demonstrably healthy.<\/strong><\/p>\n<\/blockquote>\n\n\n\n
A working system does not become healthier merely because more commands are run against it.<\/p>\n","protected":false},"excerpt":{"rendered":"
A seemingly small OpenClaw error can occasionally expose several layers of state that have accumulated over time: stale Codex session bindings, retired model references, legacy media metadata, database schema migrations, old plugin generations, and outdated systemd service definitions. This article documents one such recovery on an Ubuntu host. All hostnames, …<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[88],"tags":[337,264],"class_list":["post-6201","post","type-post","status-publish","format-standard","hentry","category-4brz75t44cdxjqe","tag-openclaw","tag-troubleshooting"],"_links":{"self":[{"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/posts\/6201","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/comments?post=6201"}],"version-history":[{"count":1,"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/posts\/6201\/revisions"}],"predecessor-version":[{"id":6202,"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/posts\/6201\/revisions\/6202"}],"wp:attachment":[{"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/media?parent=6201"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/categories?post=6201"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.eternalsphere.net\/index.php\/wp-json\/wp\/v2\/tags?post=6201"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}