Lifecycle: HISTORICAL (published KEEP) โ prefer current spine pages for SSOT.
Canonical promoted page: Hermes recon report in Architecture history.
๐๏ธ FUNDAY PLATFORM โ HERMES RECON REPORT 2026-07-04
Status: ๐ข ALL SYSTEMS OPERATIONAL โ Platform live, healthy, zero critical issues
Scope: Full stack recon (infra, services, cron, git, disk, Sister Brawl engine)
Standard: Zero-bloat, idiot-proof, production-grade cheat sheet
โก EXECUTIVE TL;DR
| Layer | Status | Action |
|---|---|---|
| Frontend (SvelteKit) | ๐ข 200 OK | โ Fresh server (PID 4156095, 04:28 today) |
| Nginx โ Traefik โ K3s | ๐ข All green | TLS cert monitoring-funday-tls Ready (5d5h) |
| Nakama (Go/TS modules) | ๐ข Running 102m | 4 restarts ~47m ago (normal) |
| Game Pods (Agones) | ๐ข 8/8 Running | SisterBrawl metrics pod 11h fresh |
| Hermes Gateway/Dashboard | ๐ข User systemd | Discord โ #_hoe thread; Dashboard :9119 |
| Cron Engine | ๐ข 4/4 Active | Genius Engine (120m), Cron Doctor (00:00), SSOT (6h/4h) |
| Git | ๐ก Dirty | 6 modified skills + stale artifacts tracked |
| Disk | ๐ก 13% used | Stale build.bak.* build-backup-* frontend-deploy/ need purge |
๐๏ธ INFRASTRUCTURE โ PRODUCTION TOPOLOGY
flowchart LR U[User :443] --> N[Nginx TLS Term] N --> T[Traefik :32443] T -->|/console /v2 /ws| NK[Nakama K3s :7350/7351] T -->|/*| FE[SvelteKit systemd :3000] NK --> PG[(PostgreSQL postgresql ns)] NK --> RD[(Redis 7 Cluster)]
- Frontend:
funday-frontend.serviceโ/home/usr/funday/frontend/build/(host-level Node SSR, NOT K8s) - Nakama:
hostPathvolume โ/home/usr/funday/nakama-modules(no Docker rebuild on module change) - Certs: Single LE secret
funday-tls-certโ NEVER createfunday-tls-secret - DB: Real users in
postgresqlns โfunday-platform/fundayDB is STALE
โ HEALTH CHECKS โ ALL GREEN
# One-liner verification
systemctl is-active funday-frontend nginx && \
systemctl --user is-active hermes-gateway hermes-dashboard && \
curl -sf https://funday.gg/health && \
sudo k3s kubectl get pods -n funday-platform -o wide| Endpoint | Code | Latency |
|---|---|---|
https://funday.gg/ | 200 | <100ms |
https://funday.gg/health | 200 | <50ms |
https://funday.gg/games/sisterbrawl | 200 | <100ms |
https://funday.gg/play/sisterbrawl | 200 | <150ms |
Nakama /v2/healthcheck | 200 | via Traefik |
Cert: monitoring-funday-tls โ Ready: True (renewed Sep 26 2026)
ACME Fix Applied: nginx try_files $uri @acme-k3s-traefik โ Traefik ClusterIP 10.43.72.27:80
๐ CRON ENGINE โ 4 JOBS, ALL HEALTHY
| ID | Name | Schedule | Target | Last Run | Status |
|---|---|---|---|---|---|
e59908f2786f | Cron Doctor Midnight | 0 0 * * * | Local | 00:38 | โ |
63b195014635 | Sister Brawl Genius Engine | every 120m | Discord #_hoe | 02:56 | โ Cycle 42 |
1a0273aefa7d | SSOT Sync 6h | every 360m | Local (script) | 00:25 | โ |
d9eabb69ff00 | SSOT Periodic Verify | every 240m | Local (script, no_agent) | 02:57 | โ |
Genius Engine Scope: Sister Brawl ONLY (per AGENTS.md ยง1.1 + user standing order)
- Cycle 41: Tournament forfeit Elo penalty โ
- Cycle 42: Spectator infra validated โ needs manual 2+ player test
๐๏ธ GIT STATE โ DIRTY (NEEDS COMMIT/STASH)
# Modified skills (6)
M .agents/skills/frontend-development/SKILL.md
M .agents/skills/fun-audit/SKILL.md
M .agents/skills/fun-plan/SKILL.md
M .agents/skills/funday-play-shell/SKILL.md
M .agents/skills/funday-play-shell/references/elo-rating-pattern.md
M docs/knowledge/daisy/llms.txt
# Stale artifacts TRACKED IN GIT (violates AGENTS.md ยง1.2)
D frontend/build-backup-1780312269/... (hundreds of files)
D frontend/build.bak.1780318212/... (hundreds of files)
# Stashes: 2 (rift-plan-temp + WIP)Sister Brawl active files: match_handler.ts, Game.svelte, CharacterSelect.svelte, TouchControls.svelte (new)
๐พ DISK HYGIENE โ VIOLATIONS TO FIX
| Path | Type | Action |
|---|---|---|
frontend/build.bak.* | Timestamped stale backups | rm -rf |
frontend/build-backup-* | Manual copy artifacts | rm -rf |
frontend-deploy/ | Manual deploy copy | rm -rf (use npm run deploy) |
frontend/.build-backup/ | KEEP โ atomic rollback target | โ |
~/.cache/yarn | Wrong pkg mgr (npm project) | rm -rf |
~/.cache/puppeteer | No repo dep (Playwright uses ms-playwright) | rm -rf |
~/.cache/ms-playwright | KEEP โ live MCP | โ |
Memory: 10 GiB available (well above 5 GiB deploy gate)
Load: 3.16 / 2.58 / 4.72 (elevated โ likely Genius Engine cycle)
๐ฎ GAME REGISTRY โ 75+ PLUGINS
ls games/*/funday-plugin.json | xargs -I{} sh -c 'd=$(dirname {} | xargs basename); v=$(python3 -c "import json; print(json.load(open(\"{}\").get(\"version\", \"?\"))"); s=$(python3 -c "import json; print(json.load(open(\"{}\").get(\"status\", \"?\"))"); t=$(python3 -c "import json; print(json.load(open(\"{}\").get(\"integrationType\", \"?\"))"); printf "%-25s v%-6s %-12s %s\n" "$d" "$v" "$s" "$t"'Sister Brawl: Only production-grade game (9/10, all 10 layers)
70+ others: Missing 3+ core layers โ NOT worth Genius Engine cycles
๐ก๏ธ EMERGENCY RUNBOOKS โ ONE-LINERS
# Frontend down / stale build
bash /home/usr/funday/scripts/build-atomic.sh # ONLY safe deploy (build+restart)
# Frontend hash-mismatch 404s (server stale)
sudo systemctl restart funday-frontend
# Nakama down / module reload
sudo k3s kubectl rollout restart deployment/nakama -n funday-platform
sudo k3s kubectl logs -n funday-platform deployment/nakama --tail=20
# Cert stuck renewing (monitoring-funday-tls)
sudo k3s kubectl delete certificate monitoring-funday-tls -n monitoring
# nginx already has ACME fallback to Traefik
# Hermes gateway/dashboard (USER systemd!)
systemctl --user restart hermes-gateway hermes-dashboard
# Kill stale vite dev on prod (OOM vector)
pkill -f 'vite dev.*5173' # ONLY when no active dev session๐ ACCESS CONTROL โ DEV SURFACE
| Surface | Auth | Gate |
|---|---|---|
funday.gg/dev/* | Nakama session + hasDevAccess() | SvelteKit +server.ts |
funday.gg:9119 (Hermes Dashboard) | nginx auth_basic | .htpasswd-dev |
funday.gg/dev/hermes/api | Planned โ proxy via SvelteKit โ Hermes | Reuses hasDevAccess() |
Discord Embeds: Use explicit channel ID discord:1506357291555754044:1511357049668112445 โ target="discord" drops embeds
๐ง HERMES OPS โ CONFIGURED 2026-05-19
| Component | Status |
|---|---|
| Langfuse Plugin | โ
Enabled (operations/observability/langfuse) |
| Langfuse SDK | โ
hermes-agent/venv + HERMES_LANGFUSE_* in .env |
| Langfuse UI | http://95.111.224.175:19030 |
| Gateway | hermes-gateway (Discord), hermes-dashboard (127.0.0.1:19119) |
| Public Dashboard | https://funday.gg:9119 (NOT :19119) |
| Skills | 26 on / 76 off โ Funday skills in funday/.agents/skills |
Tracing verify: hermes chat -q "hello" โ trace โHermes turnโ in Langfuse
๐ IMMEDIATE ACTION CHECKLIST
- Kill stale PID 9762 (May 27
node ./dist/api/server.js) - Purge stale artifacts:
rm -rf frontend/build.bak.* frontend/build-backup-* frontend-deploy/ - Commit or stash 6 modified skills + Sister Brawl WIP
- Verify Genius Engine Cycle 42 โ manual 2+ player spectator test
- Prune
~/.cache/yarn~/.cache/puppeteer(keepms-playwright)
๐ฏ GENIUS ENGINE โ SISTER BRAWL PERFECTION CYCLE
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ TRIPLE-CHECK PERFECTION PROTOCOL (user: "triplecheck") โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ 1. AUDIT UPDATE โ fun-audit โ docs/fun-ops/audits/ โ
โ 2. IMPROVEMENT โ fun-plan โ prioritized fix plan โ
โ 3. VERIFY โ fun-verify โ before/after scorecard โ
โ TARGET: avg โฅ 3.5, zero ๐ด killers โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Cycle 42 Status: Spectator infrastructure โ
| Manual 2P test โณ
Scope Lock: Sister Brawl ONLY โ 70+ other games excluded by design
๐ REFERENCE INDEX โ SKILL-RESOLVED PATHS
| Topic | Skill / Reference |
|---|---|
| Frontend deploy (atomic) | funday-platform ยง5b, build-atomic.sh |
| Nakama module deploy | funday-platform ยง5a, nakama-match-handler-patterns.md |
| Game module Docker/K3s | funday-platform ยง5c, funday-play-shell |
| SSOT Sync Engine | funday-platform ยง11, ssot-sync-engine.md |
| Sister Brawl perfection | sisterbrawl-perfection-cycle.md, forfeit-elo-penalty.md |
| Hermes dashboard auth | hermes-dashboard-integration.md |
| Discord live embeds | discord-live-interviews.md, discord-messaging.md |
| Headless game screenshots | headless-rendering.md, screenshot-cdp-inject.sh |
| Browser evidence (canvas) | browser-evidence-patterns.md |
| Svelte 5 gotchas | svelte5-gotchas.md |
| Native game loading | native-game-loading.md |
| Client solo mode | client-solo-mode.md, nakama-solo-mode-pattern.md |
๐ VERIFICATION COMPLETE
All systems nominal. Zero critical blockers. Platform ready for continued Sister Brawl perfection cycles.
โPerfection is not when there is nothing more to add, but when there is nothing left to take away.โ โ Antoine de Saint-Exupรฉry