Skip to content

Latest commit

 

History

History
184 lines (134 loc) · 5.56 KB

PROJECT_GUIDE.md

File metadata and controls

184 lines (134 loc) · 5.56 KB

ASA QA Harness Project Guide

Project Goal

Build an authorized QA harness for ARK: Survival Ascended on a private server to find, reproduce, and report inventory duplication defects. The harness is for controlled testing, inventory consistency checks, observability, and anomaly reporting.

This project should not document public-server abuse, evasion, or exploit procedures.

Current Phase

Phases 1-3 are implemented:

  • ManualInventoryObserver loads inventory snapshots from JSON fixtures.
  • LogInventoryObserver reads simulated [ASA_QA] inventory observations from text logs.
  • EpisodeRecorder persists observations and episode results.
  • run_episode orchestrates generic observers.
  • run_dry_episode runs fixture-backed dry episodes.
  • run_log_episode runs log-backed episodes.
  • compare_inventory_totals checks inventory conservation.
  • Pytest coverage exists for models, observers, recorder, invariants, and runners.

ASA QA Inventory Probe v0.1 Status

The DevKit Blueprint to [ASA_QA] logs to LogInventoryObserver to inventory invariants circuit is validated with reproducible fixtures. The Python repo owns the log contract, parser tests, and invariant checks; it does not implement the visual Ark DevKit Blueprint.

See ASA_QA_INVENTORY_PROBE_V0_1.md for the v0.1 Blueprint variables, functions, key bindings, supported resource, log examples, and troubleshooting notes.

Hardening v0.1 covers missing snapshots, invalid actions, debug noise, multiple actions, and multiple snapshots.

SQLite index v0.1 allows local querying of episodes, actions, checks, and anomalies without replacing JSON or JSONL episode files.

rebuild-index regenerates SQLite from data/episodes without repeating log-run.

The SQLite index normalizes legacy episodes to avoid NULL/None counts in queries.

export-episode v0.1 creates evidence bundles for individual episodes.

guided-run v0.1 guides manual DevKit tests and executes log-run at the end.

DevKit operation recipe v0.1 documents G/H snapshots and U/I/O move_stack presets for guided manual QA episodes.

Architecture Boundaries

  • runner: orchestrates episodes and maps outcomes to EpisodeResult.
  • observers: collect state and return validated Observation objects.
  • recorder: persists observations.jsonl and result.json.
  • invariants: detect consistency violations from collected state.
  • adapters: future external integrations such as server APIs or admin tooling.
  • fuzzer: future work for generated test scenarios.
  • ML: future prioritization only, not current decision-making or automation.

Data Contracts

InventoryState

{
  "box_a": {
    "metal": 100
  },
  "box_b": {
    "metal": 0
  },
  "player": {
    "metal": 0
  }
}

Validation rules:

  • The top level is an object.
  • Each location name is a non-empty string.
  • Each location maps to an object of item counts.
  • Each item name is a non-empty string.
  • Each count is an integer greater than or equal to 0.

Observation

{
  "episode_id": "ep_000001",
  "timestamp": "2026-05-12T20:00:00Z",
  "observation_type": "inventory_state",
  "phase": "before",
  "state": {
    "box_a": {
      "metal": 100
    },
    "box_b": {
      "metal": 0
    },
    "player": {
      "metal": 0
    }
  }
}

phase is currently before or after. observation_type is currently inventory_state.

EpisodeResult

Fields:

  • episode_id
  • seed
  • test_type
  • started_at
  • ended_at
  • status
  • anomalies
  • error_message
  • evidence
  • observation_count
  • evidence_paths

EpisodeStatus

Allowed values:

  • PASS
  • FAIL_BOT
  • FAIL_OBSERVER
  • ANOMALY
  • NEEDS_REVIEW
  • CRASH
  • TIMEOUT

Observer Contract

Observers implement:

class InventoryObserver(Protocol):
    def observe(self, episode_id: str, phase: str) -> Observation:
        ...

The runner calls one observer for before and one observer for after. Observers should raise ObserverError when state cannot be collected or validated.

Log Format

LogInventoryObserver scans text logs for [ASA_QA] as an in-line marker. Logs may include engine or category prefixes before the marker, such as Unreal LogBlueprintUserMessages lines. Lines without [ASA_QA] are ignored.

Each [ASA_QA] line contains JSON after the marker:

[ASA_QA] {"episode_id": "ep_000001", "timestamp": "2026-05-12T20:00:00Z", "observation_type": "inventory_state", "phase": "before", "state": {"box_a": {"metal": 100}, "box_b": {"metal": 0}, "player": {"metal": 0}}}
LogBlueprintUserMessages: [BP_ASA_QA_Probe_C_1] [ASA_QA] {"episode_id": "ep_000001", "timestamp": "2026-05-12T20:00:00Z", "observation_type": "inventory_state", "phase": "before", "state": {"box_a": {"metal": 100}, "box_b": {"metal": 0}, "player": {"metal": 0}}}

If multiple matching observations exist for the same episode_id and phase, the last matching log line is used.

How To Run

Run the test suite:

uv run --with pytest pytest

Run the basic fixture-backed example:

uv run python examples/basic_episode.py

Current Limitations

  • No ArkDevKit Blueprint implementation in this Python repo.
  • No OCR.
  • No bot input.
  • No RCON or Nitrado adapter.
  • No fuzzing.
  • No ML.

Future Path

  • Keep the private ASA QA Inventory Probe contract stable as the Blueprint evolves.
  • Continue reading [ASA_QA] logs with LogInventoryObserver.
  • Later add authorized action automation.
  • Later add fuzzing for generated inventory scenarios.
  • Later add ML-assisted prioritization for which scenarios to review first.