From 627b47406128ff6ae2227c606ad662cf1586969f Mon Sep 17 00:00:00 2001 From: Harzva <49864959+Harzva@users.noreply.github.com> Date: Fri, 7 Aug 2026 06:24:31 +0800 Subject: [PATCH] feat: activate verified MobileCore Omni runtime --- docs/mobilecore-dual-app-qa.md | 58 +++- mobile_agent/lib/screens/home_screen.dart | 89 +++++- .../services/mobilecore_adaptive_policy.dart | 18 ++ .../lib/services/tuima_provider_service.dart | 160 +++++++++- .../mobilecore_adaptive_policy_test.dart | 67 +++++ .../services/tuima_provider_service_test.dart | 275 ++++++++++++++++++ 6 files changed, 658 insertions(+), 9 deletions(-) diff --git a/docs/mobilecore-dual-app-qa.md b/docs/mobilecore-dual-app-qa.md index ea1f47b..78564d0 100644 --- a/docs/mobilecore-dual-app-qa.md +++ b/docs/mobilecore-dual-app-qa.md @@ -16,7 +16,11 @@ MobileCode is the control center. It owns model routing, user consent, Phone Use - memory/storage preflight and typed failure code; - installed models, device recommendations, and decode/latency metrics; - model load, unload, and switch controls using public `model_id` and optional - public `projector_id` only. + public `projector_id` only; +- typed verified-Omni status plus a dedicated load control that refuses partial + or unverified pairs and rechecks the active `/health` runtime, artifact + verification and pinned digests, and requested image/audio capability after + loading. The v2 client reads health before and after the models/metrics/recommendations bundle. If MobileCore changes the active runtime during that window, MobileCode @@ -26,7 +30,20 @@ IDs containing path separators or control characters are rejected before a request is sent, and a switch is successful only when `/health` reports the exact requested public model ID. -The TuiMa control sheet exposes that state inside MobileCode. Image and audio entry points are capability-gated. Selected media is held only in memory, sent only to the local MobileCore endpoint, removed after the request, and excluded from saved chat turns and ActionEvidence. A local-only media request fails closed instead of falling back to cloud inference. +The TuiMa control sheet exposes that state inside MobileCode. When MobileCore +reports a complete verified Omni pair that is not active, the sheet offers an +explicit local activation control. Image and audio entry points remain hidden +until the loaded runtime's `/health` response advertises the corresponding +capability. If an already-selected attachment survives an external model +change, MobileCode may reactivate that verified pair, then repeats the runtime +capability check before sending any media. Selected media is held only in +memory, sent only to the local MobileCore endpoint, removed after the request, +and excluded from saved chat turns and ActionEvidence. A local-only media +request fails closed instead of falling back to cloud inference. + +This control path proves protocol enforcement and fail-closed selection; it is +not evidence that the large Qwen2.5-Omni artifacts have run successfully on a +physical phone. That remains a separate gate below. Adaptive routing currently applies these rules: @@ -191,6 +208,31 @@ request. Logcat contained no MobileCode crash, ANR, OOM, or SIGABRT marker. The local APK is Android Debug-signed and is therefore QA evidence, not the production-signed GitHub Release asset. +### Verified Omni activation control refresh + +On 2026-08-07, a clean `pureDebug` MobileCode v0.1.77 (`67`) APK containing +the typed Omni status and dedicated verified-pair activation path was built, +installed, and cold-launched on the Android 16 ARM64 emulator. The APK SHA-256 +was `3833b98f95c113454658d471e9824bb66d2f83c9117c03c48d192a86a47a3c11`. +Cold launch completed in 2,694 ms, the app remained resumed, and filtered logcat +contained no MobileCode fatal exception, ANR, OOM, or SIGABRT marker. + +MobileCore 0.1.4-rc4 exposed protocol v2 and loaded the real local +`qwen2.5-0.5b-instruct-q4_k_m` runtime by public model ID. After a visible +service restart and control-sheet retry, MobileCode displayed the exact active +model, `llama.cpp`, CPU backend, `Q4_K_M`, 462 MB peak runtime memory, ready +preflight, three installed models, and explicit `image no ยท audio no` +capabilities. The stopped-service interval was surfaced as the typed +`service_unavailable` state instead of stale controls. + +The same emulator lacked both pinned Qwen2.5-Omni artifacts, and its preflight +reported insufficient memory and storage for that pair. MobileCode therefore +did not render the verified-Omni activation control and did not expose image or +audio attachment entry points. This proves real dual-app capability gating and +the unverified/insufficient-resource refusal path; the successful activation +path remains contract-tested until a capable physical device with both verified +artifacts is available. + ### Withdrawn v0.1.76 Android artifact The official APK described below was withdrawn on 2026-08-07 after post-build @@ -354,13 +396,15 @@ Raw screenshots and sanitized logcat remain under the ignored `.qa-artifacts/` d ## Verification -- The complete MobileCode Flutter suite passed 571 tests after the Client v2, +- The complete MobileCode Flutter suite passed 577 tests after the Client v2, adaptive-routing, pressure-switch, proactive-offline, one-task cloud approval, protocol-handshake, cancellation, and bounded-output follow-ups. -- The focused MobileCore client/adaptive-policy/approval suite passed 37 tests +- The focused MobileCore client/adaptive-policy/approval suite passed 43 tests covering coherent runtime snapshots, exact switch confirmation, public projector IDs, - path-like ID rejection, projector metadata, and image-capability parsing. + path-like ID rejection, projector metadata, image-capability parsing, + verified-Omni activation, partial-pair rejection, post-load pinned-digest + matching, and audio capability confirmation. - The focused adaptive-policy coverage includes privacy/offline fail-closed routing, cloud-consent gating, constrained context/model choice, one-task approval expiry semantics, and multimodal capability retention under @@ -373,6 +417,10 @@ Raw screenshots and sanitized logcat remain under the ignored `.qa-artifacts/` d - A clean local Android arm64 `pureRelease` build passed and its manifest version was verified as `0.1.76+66`; this local build is validation evidence only and is not the stable-signed GitHub Release asset. +- A later `pureDebug` v0.1.77 (`67`) APK with verified-Omni activation control + also built, installed, cold-launched, and rendered the real MobileCore control + state on the Android 16 ARM64 emulator. It is debug-signed QA evidence, not a + release asset. - The real emulator UI displayed the one-task cloud approval card before a complex cloud request. Declining routed to MobileCore, and the subsequent local timeout remained fail-closed instead of opening the cloud provider. diff --git a/mobile_agent/lib/screens/home_screen.dart b/mobile_agent/lib/screens/home_screen.dart index d3d87df..24d9ddb 100644 --- a/mobile_agent/lib/screens/home_screen.dart +++ b/mobile_agent/lib/screens/home_screen.dart @@ -14065,6 +14065,9 @@ class _ChatPanelState extends State<_ChatPanel> { Future<_MobileCoreControlData> _loadMobileCoreControlData() async { final snapshot = await _tuimaProviderService.runtimeSnapshot(); + final omniStatus = await _tuimaProviderService + .omniStatus() + .catchError((_) => const MobileCoreOmniStatus()); if (mounted) setState(() => _tuimaHealth = snapshot.health); if (snapshot.health.state == TuimaConnectionState.unavailable) { throw MobileCoreProviderException( @@ -14078,6 +14081,7 @@ class _ChatPanelState extends State<_ChatPanel> { models: snapshot.models, metrics: snapshot.metrics, recommendations: snapshot.recommendations, + omniStatus: omniStatus, ); } @@ -14087,6 +14091,7 @@ class _ChatPanelState extends State<_ChatPanel> { required DateTime startedAt, required bool success, Object? error, + Map safeMetadata = const {}, }) async { final safeFailure = error is MobileCoreProviderException ? error.code @@ -14111,6 +14116,7 @@ class _ChatPanelState extends State<_ChatPanel> { 'inferenceLocation': 'on_device', 'operation': operation, 'modelId': modelId, + ...safeMetadata, 'redactionApplied': true, }, )); @@ -14182,6 +14188,41 @@ class _ChatPanelState extends State<_ChatPanel> { } } + Future _activateVerifiedOmni({ + MobileCoreAttachmentKind? requiredCapability, + int contextLength = 4096, + }) async { + final startedAt = DateTime.now(); + try { + final result = await _tuimaProviderService.loadVerifiedOmni( + requiredCapability: requiredCapability, + contextLength: contextLength, + ); + if (mounted) setState(() => _tuimaHealth = result.health); + await _recordMobileCoreControl( + operation: requiredCapability == null + ? 'activate_verified_omni' + : 'activate_verified_omni_for_${requiredCapability.name}', + modelId: result.status.modelId, + startedAt: startedAt, + success: true, + safeMetadata: result.status.evidenceMetadata, + ); + return result.health; + } on Object catch (error) { + await _recordMobileCoreControl( + operation: requiredCapability == null + ? 'activate_verified_omni' + : 'activate_verified_omni_for_${requiredCapability.name}', + modelId: 'verified-omni-pair', + startedAt: startedAt, + success: false, + error: error, + ); + rethrow; + } + } + Future _openMobileCoreControlSheet() async { var loader = _loadMobileCoreControlData(); await showModalBottomSheet( @@ -14303,6 +14344,26 @@ class _ChatPanelState extends State<_ChatPanel> { label: const Text('Unload active model'), ), ], + if (data.omniStatus.loadable && + !(data.health.canInfer && + data.health.mainArtifact.verified && + data.health.projectorArtifact.verified && + data.health.runtime.contains('libmtmd'))) ...[ + const SizedBox(height: 10), + Text( + 'A verified local Omni pair is installed. Media controls remain hidden until the runtime capability check passes.', + style: + const TextStyle(color: _muted, fontSize: 12), + ), + const SizedBox(height: 8), + FilledButton.icon( + onPressed: () => unawaited(run(() async { + await _activateVerifiedOmni(); + })), + icon: const Icon(Icons.auto_awesome_outlined), + label: const Text('Activate verified Omni'), + ), + ], ], ), ), @@ -14422,9 +14483,33 @@ class _ChatPanelState extends State<_ChatPanel> { measuredDecodeTokensPerSecond: measuredSpeedFor(health), ); var currentHealth = health; + if (attachment != null && + !currentHealth.capabilities.supports(attachment.kind)) { + final omniStatus = await _tuimaProviderService + .omniStatus() + .catchError((_) => const MobileCoreOmniStatus()); + if (MobileCoreAdaptivePolicy.shouldActivateVerifiedOmni( + health: currentHealth, + omniStatus: omniStatus, + attachmentKind: attachment.kind, + )) { + currentHealth = await _activateVerifiedOmni( + requiredCapability: attachment.kind, + contextLength: decision.contextLength, + ); + decision = MobileCoreAdaptivePolicy.decideForTask( + health: currentHealth, + recommendations: recommendations, + telemetry: telemetry, + task: taskSignals, + attachmentKind: attachment.kind, + measuredDecodeTokensPerSecond: measuredSpeedFor(currentHealth), + ); + } + } if (MobileCoreAdaptivePolicy.shouldSwitchToRecommendedModel( decision: decision, - activeModelId: health.activeModel, + activeModelId: currentHealth.activeModel, attachmentKind: attachment?.kind, )) { currentHealth = await _tuimaProviderService.loadModel( @@ -21093,12 +21178,14 @@ class _MobileCoreControlData { this.models = const [], this.metrics = const MobileCoreMetrics(), this.recommendations = const MobileCoreRecommendations(), + this.omniStatus = const MobileCoreOmniStatus(), }); final TuimaHealth health; final List models; final MobileCoreMetrics metrics; final MobileCoreRecommendations recommendations; + final MobileCoreOmniStatus omniStatus; } class _MobileCoreRuntimeDecision { diff --git a/mobile_agent/lib/services/mobilecore_adaptive_policy.dart b/mobile_agent/lib/services/mobilecore_adaptive_policy.dart index 2f6d274..764ce9b 100644 --- a/mobile_agent/lib/services/mobilecore_adaptive_policy.dart +++ b/mobile_agent/lib/services/mobilecore_adaptive_policy.dart @@ -243,6 +243,24 @@ class MobileCoreAdaptivePolicy { recommended != activeModelId; } + /// A selected local attachment may reactivate the pinned Omni runtime when + /// another model became active after the attachment was chosen. The status + /// only proves artifact readiness; the client must still verify the active + /// `/health` capability after loading before sending media bytes. + static bool shouldActivateVerifiedOmni({ + required TuimaHealth health, + required MobileCoreOmniStatus omniStatus, + required MobileCoreAttachmentKind attachmentKind, + }) { + final pinnedOmniActive = health.canInfer && + health.mainArtifact.verified && + health.projectorArtifact.verified && + health.runtime.contains('libmtmd'); + return omniStatus.loadable && + !pinnedOmniActive && + !health.capabilities.supports(attachmentKind); + } + static MobileCorePolicyDecision decide({ required TuimaHealth health, required MobileCoreRecommendations recommendations, diff --git a/mobile_agent/lib/services/tuima_provider_service.dart b/mobile_agent/lib/services/tuima_provider_service.dart index c2f09d5..856237a 100644 --- a/mobile_agent/lib/services/tuima_provider_service.dart +++ b/mobile_agent/lib/services/tuima_provider_service.dart @@ -154,6 +154,78 @@ class MobileCoreArtifactHealth { } } +class MobileCoreOmniStatus { + const MobileCoreOmniStatus({ + this.modelId = '', + this.revision = '', + this.phase = 'unknown', + this.pairVerified = false, + this.mainArtifact = const MobileCoreArtifactHealth(), + this.projectorArtifact = const MobileCoreArtifactHealth(), + this.preflightPassed, + this.preflightFailureCode, + }); + + final String modelId; + final String revision; + final String phase; + final bool pairVerified; + final MobileCoreArtifactHealth mainArtifact; + final MobileCoreArtifactHealth projectorArtifact; + final bool? preflightPassed; + final String? preflightFailureCode; + + bool get loadable => + modelId.isNotEmpty && + pairVerified && + mainArtifact.present && + mainArtifact.verified && + projectorArtifact.present && + projectorArtifact.verified; + + Map get evidenceMetadata => { + 'modelId': modelId, + 'revision': revision, + 'phase': phase, + 'pairVerified': pairVerified, + 'mainInstalled': mainArtifact.present, + 'mainVerified': mainArtifact.verified, + 'projectorInstalled': projectorArtifact.present, + 'projectorVerified': projectorArtifact.verified, + 'preflightPassed': preflightPassed, + if (preflightFailureCode != null) + 'preflightFailureCode': preflightFailureCode, + 'redaction': 'artifact_paths_and_payloads_omitted', + }; + + factory MobileCoreOmniStatus.fromJson(Object? value) { + final map = _stringMap(value); + final artifacts = _stringMap(map['artifacts']); + final preflight = _stringMap(map['preflight']); + return MobileCoreOmniStatus( + modelId: map['model_id']?.toString().trim() ?? '', + revision: map['revision']?.toString().trim() ?? '', + phase: map['phase']?.toString().trim() ?? 'unknown', + pairVerified: map['pair_verified'] == true, + mainArtifact: MobileCoreArtifactHealth.fromJson(artifacts['main']), + projectorArtifact: MobileCoreArtifactHealth.fromJson(artifacts['mmproj']), + preflightPassed: + preflight.containsKey('passed') ? preflight['passed'] == true : null, + preflightFailureCode: _nullableString(preflight['failure_code']), + ); + } +} + +class MobileCoreOmniLoadResult { + const MobileCoreOmniLoadResult({ + required this.status, + required this.health, + }); + + final MobileCoreOmniStatus status; + final TuimaHealth health; +} + class MobileCoreResourceHealth { const MobileCoreResourceHealth({ this.availableBytes = 0, @@ -577,6 +649,7 @@ class MobileCoreClient { Uri? modelLoadUri, Uri? modelUnloadUri, Uri? omniStatusUri, + Uri? omniLoadUri, Uri? inferenceCancelUri, }) : _client = client ?? HttpClient(), healthUri = healthUri ?? Uri.parse('http://127.0.0.1:8080/health'), @@ -592,6 +665,8 @@ class MobileCoreClient { Uri.parse('http://127.0.0.1:8080/mobilecore/model/unload'), omniStatusUri = omniStatusUri ?? Uri.parse('http://127.0.0.1:8080/mobilecore/omni/status'), + omniLoadUri = omniLoadUri ?? + Uri.parse('http://127.0.0.1:8080/mobilecore/omni/load'), inferenceCancelUri = inferenceCancelUri ?? Uri.parse('http://127.0.0.1:8080/mobilecore/inference/cancel'); @@ -604,6 +679,7 @@ class MobileCoreClient { final Uri modelLoadUri; final Uri modelUnloadUri; final Uri omniStatusUri; + final Uri omniLoadUri; final Uri inferenceCancelUri; Future probe({ @@ -697,10 +773,74 @@ class MobileCoreClient { ); } - Future> omniStatus({ + Future omniStatus({ Duration timeout = const Duration(seconds: 5), - }) => - _getJson(omniStatusUri, timeout: timeout); + }) async => + MobileCoreOmniStatus.fromJson( + await _getJson(omniStatusUri, timeout: timeout), + ); + + Future loadVerifiedOmni({ + MobileCoreAttachmentKind? requiredCapability, + int contextLength = 4096, + int threads = 4, + Duration timeout = const Duration(minutes: 2), + }) async { + await _requireCompatibleHealth(timeout: const Duration(seconds: 5)); + final status = await omniStatus(timeout: const Duration(seconds: 5)); + if (!status.loadable) { + throw const MobileCoreProviderException( + code: 'omni_pair_not_verified', + message: + 'MobileCore does not have a complete verified local Omni pair.', + ); + } + _validatedPublicArtifactId( + status.modelId, + code: 'invalid_omni_model_id', + label: 'Omni model', + ); + await _postJson( + omniLoadUri, + { + 'context_length': contextLength.clamp(128, 32768), + 'threads': threads.clamp(1, 16), + 'gpu_layers': 0, + }, + timeout: timeout, + ); + final health = await probe(timeout: const Duration(seconds: 5)); + final pinnedPairActive = health.canInfer && + health.mainArtifact.verified && + health.projectorArtifact.verified && + _matchesPinnedArtifact( + expected: status.mainArtifact, + active: health.mainArtifact, + ) && + _matchesPinnedArtifact( + expected: status.projectorArtifact, + active: health.projectorArtifact, + ) && + health.runtime.contains('libmtmd'); + final anyMediaCapability = + health.capabilities.imageInput || health.capabilities.audioInput; + if (!pinnedPairActive || !anyMediaCapability) { + throw const MobileCoreProviderException( + code: 'omni_runtime_not_ready', + message: + 'MobileCore loaded the pair but did not confirm a verified multimodal runtime.', + ); + } + if (requiredCapability != null && + !health.capabilities.supports(requiredCapability)) { + throw MobileCoreProviderException( + code: 'unsupported_modality', + message: + 'The verified local Omni runtime does not support ${requiredCapability.name} input.', + ); + } + return MobileCoreOmniLoadResult(status: status, health: health); + } Future cancelInference({ Duration timeout = const Duration(seconds: 3), @@ -1182,6 +1322,19 @@ class MobileCoreClient { return normalized; } + static bool _matchesPinnedArtifact({ + required MobileCoreArtifactHealth expected, + required MobileCoreArtifactHealth active, + }) { + final expectedDigest = expected.expectedDigest.trim().toLowerCase(); + if (expectedDigest.isEmpty) return true; + final activeDigest = active.expectedDigest.trim().toLowerCase(); + final expectedAlgorithm = expected.digestAlgorithm.trim().toLowerCase(); + final activeAlgorithm = active.digestAlgorithm.trim().toLowerCase(); + return activeDigest == expectedDigest && + (expectedAlgorithm.isEmpty || activeAlgorithm == expectedAlgorithm); + } + static bool _isConsistentRuntimeSnapshot({ required TuimaHealth before, required TuimaHealth after, @@ -1301,6 +1454,7 @@ class TuimaProviderService extends MobileCoreClient { super.modelLoadUri, super.modelUnloadUri, super.omniStatusUri, + super.omniLoadUri, super.inferenceCancelUri, }); diff --git a/mobile_agent/test/services/mobilecore_adaptive_policy_test.dart b/mobile_agent/test/services/mobilecore_adaptive_policy_test.dart index 5d97119..5ae966d 100644 --- a/mobile_agent/test/services/mobilecore_adaptive_policy_test.dart +++ b/mobile_agent/test/services/mobilecore_adaptive_policy_test.dart @@ -277,6 +277,73 @@ void main() { ); }); + test('verified Omni activation is selected only for a missing media ability', + () { + const verified = MobileCoreOmniStatus( + modelId: 'verified-omni', + pairVerified: true, + mainArtifact: MobileCoreArtifactHealth(present: true, verified: true), + projectorArtifact: + MobileCoreArtifactHealth(present: true, verified: true), + ); + + expect( + MobileCoreAdaptivePolicy.shouldActivateVerifiedOmni( + health: _health(image: false), + omniStatus: verified, + attachmentKind: MobileCoreAttachmentKind.image, + ), + isTrue, + ); + expect( + MobileCoreAdaptivePolicy.shouldActivateVerifiedOmni( + health: _health(image: true), + omniStatus: verified, + attachmentKind: MobileCoreAttachmentKind.image, + ), + isFalse, + ); + expect( + MobileCoreAdaptivePolicy.shouldActivateVerifiedOmni( + health: TuimaHealth( + state: TuimaConnectionState.modelReady, + version: '0.1.4-rc4', + backend: 'cpu', + runtime: 'llama.cpp/libmtmd', + activeModel: 'verified-omni', + capabilities: const MobileCoreCapabilities(textInput: true), + mainArtifact: + const MobileCoreArtifactHealth(present: true, verified: true), + projectorArtifact: + const MobileCoreArtifactHealth(present: true, verified: true), + ), + omniStatus: verified, + attachmentKind: MobileCoreAttachmentKind.audio, + ), + isFalse, + ); + }); + + test('partial Omni artifacts can never trigger automatic media activation', + () { + const partial = MobileCoreOmniStatus( + modelId: 'partial-omni', + pairVerified: false, + mainArtifact: MobileCoreArtifactHealth(present: true, verified: true), + projectorArtifact: + MobileCoreArtifactHealth(present: true, verified: false), + ); + + expect( + MobileCoreAdaptivePolicy.shouldActivateVerifiedOmni( + health: _health(image: false), + omniStatus: partial, + attachmentKind: MobileCoreAttachmentKind.image, + ), + isFalse, + ); + }); + test('complex work needs explicit approval before cloud routing', () { final pending = MobileCoreAdaptivePolicy.decide( health: _health(), diff --git a/mobile_agent/test/services/tuima_provider_service_test.dart b/mobile_agent/test/services/tuima_provider_service_test.dart index e6e3346..d8a2bb3 100644 --- a/mobile_agent/test/services/tuima_provider_service_test.dart +++ b/mobile_agent/test/services/tuima_provider_service_test.dart @@ -89,6 +89,7 @@ void main() { modelLoadUri: root.resolve('/mobilecore/model/load'), modelUnloadUri: root.resolve('/mobilecore/model/unload'), omniStatusUri: root.resolve('/mobilecore/omni/status'), + omniLoadUri: root.resolve('/mobilecore/omni/load'), inferenceCancelUri: root.resolve('/mobilecore/inference/cancel'), ); }); @@ -549,6 +550,280 @@ void main() { expect(unloadedHealth.canInfer, isFalse); }); + test('activates only a verified Omni pair and rechecks audio capability', + () async { + var loaded = false; + var loadRequests = 0; + server.listen((request) async { + request.response.headers.contentType = ContentType.json; + switch (request.uri.path) { + case '/health': + request.response.write(jsonEncode({ + 'status': 'ok', + 'service': 'mobilecore', + 'protocol': _mobileCoreProtocolV2, + 'version': '0.1.4-rc4', + 'backend': 'cpu', + 'runtime': loaded ? 'llama.cpp/libmtmd' : 'llama.cpp', + 'active_model': loaded ? 'Qwen2.5-Omni-3B-Q4_K_M' : null, + 'model_loaded': loaded, + 'capabilities': { + 'text_input': loaded, + 'image_input': loaded, + 'audio_input': loaded, + 'text_output': loaded, + }, + 'artifacts': loaded + ? { + 'main': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'main-pin', + }, + 'mmproj': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'projector-pin', + }, + } + : const {}, + })); + break; + case '/mobilecore/omni/status': + request.response.write(jsonEncode({ + 'model_id': 'qwen2.5-omni-3b-verified', + 'revision': 'fixed-public-revision', + 'phase': 'verified', + 'pair_verified': true, + 'artifacts': { + 'main': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'main-pin', + }, + 'mmproj': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'projector-pin', + }, + }, + 'preflight': {'passed': true}, + })); + break; + case '/mobilecore/omni/load': + loadRequests += 1; + expect(request.headers.value(HttpHeaders.authorizationHeader), + 'Bearer local'); + final body = jsonDecode(await utf8.decoder.bind(request).join()) + as Map; + expect(body['context_length'], 2048); + expect(body['threads'], 4); + expect(body['gpu_layers'], 0); + expect(body.containsKey('path'), isFalse); + loaded = true; + request.response.write('{"loaded":true}'); + break; + } + await request.response.close(); + }); + + final result = await service.loadVerifiedOmni( + requiredCapability: MobileCoreAttachmentKind.audio, + contextLength: 2048, + ); + + expect(loadRequests, 1); + expect(result.status.loadable, isTrue); + expect(result.status.preflightPassed, isTrue); + expect(result.health.capabilities.audioInput, isTrue); + expect(result.health.mainArtifact.verified, isTrue); + expect(result.health.projectorArtifact.verified, isTrue); + expect(result.status.evidenceMetadata.toString(), + isNot(contains('/private/'))); + expect( + result.status.evidenceMetadata.toString(), isNot(contains('.gguf'))); + }); + + test('rejects an unverified Omni pair before the load route', () async { + var loadRequests = 0; + server.listen((request) async { + request.response.headers.contentType = ContentType.json; + if (request.uri.path == '/health') { + request.response.write(jsonEncode({ + ..._healthPayload(), + 'active_model': null, + 'model_loaded': false, + })); + } else if (request.uri.path == '/mobilecore/omni/status') { + request.response.write(jsonEncode({ + 'model_id': 'qwen2.5-omni-3b-verified', + 'phase': 'installed', + 'pair_verified': false, + 'artifacts': { + 'main': {'installed': true, 'verified': true}, + 'mmproj': {'installed': true, 'verified': false}, + }, + })); + } else if (request.uri.path == '/mobilecore/omni/load') { + loadRequests += 1; + request.response.write('{"loaded":true}'); + } + await request.response.close(); + }); + + await expectLater( + service.loadVerifiedOmni( + requiredCapability: MobileCoreAttachmentKind.image, + ), + throwsA(isA().having( + (error) => error.code, + 'code', + 'omni_pair_not_verified', + )), + ); + expect(loadRequests, 0); + }); + + test('rejects audio when the loaded Omni runtime reports image only', + () async { + var loaded = false; + server.listen((request) async { + request.response.headers.contentType = ContentType.json; + if (request.uri.path == '/health') { + request.response.write(jsonEncode({ + 'status': 'ok', + 'service': 'mobilecore', + 'protocol': _mobileCoreProtocolV2, + 'version': '0.1.4-rc4', + 'backend': 'cpu', + 'runtime': loaded ? 'llama.cpp/libmtmd' : 'llama.cpp', + 'active_model': loaded ? 'Qwen2.5-Omni-3B-Q4_K_M' : null, + 'model_loaded': loaded, + 'capabilities': { + 'text_input': loaded, + 'image_input': loaded, + 'audio_input': false, + 'text_output': loaded, + }, + 'artifacts': loaded + ? { + 'main': {'installed': true, 'verified': true}, + 'mmproj': {'installed': true, 'verified': true}, + } + : const {}, + })); + } else if (request.uri.path == '/mobilecore/omni/status') { + request.response.write(jsonEncode({ + 'model_id': 'qwen2.5-omni-3b-verified', + 'phase': 'verified', + 'pair_verified': true, + 'artifacts': { + 'main': {'installed': true, 'verified': true}, + 'mmproj': {'installed': true, 'verified': true}, + }, + })); + } else if (request.uri.path == '/mobilecore/omni/load') { + await utf8.decoder.bind(request).join(); + loaded = true; + request.response.write('{"loaded":true}'); + } + await request.response.close(); + }); + + await expectLater( + service.loadVerifiedOmni( + requiredCapability: MobileCoreAttachmentKind.audio, + ), + throwsA(isA().having( + (error) => error.code, + 'code', + 'unsupported_modality', + )), + ); + }); + + test('rejects a loaded Omni runtime whose artifact digest changed', + () async { + var loaded = false; + server.listen((request) async { + request.response.headers.contentType = ContentType.json; + if (request.uri.path == '/health') { + request.response.write(jsonEncode({ + 'status': 'ok', + 'service': 'mobilecore', + 'protocol': _mobileCoreProtocolV2, + 'version': '0.1.4-rc4', + 'backend': 'cpu', + 'runtime': loaded ? 'llama.cpp/libmtmd' : 'llama.cpp', + 'active_model': loaded ? 'Qwen2.5-Omni-3B-Q4_K_M' : null, + 'model_loaded': loaded, + 'capabilities': { + 'text_input': loaded, + 'image_input': loaded, + 'audio_input': loaded, + 'text_output': loaded, + }, + 'artifacts': loaded + ? { + 'main': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'different-main', + }, + 'mmproj': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'projector-pin', + }, + } + : const {}, + })); + } else if (request.uri.path == '/mobilecore/omni/status') { + request.response.write(jsonEncode({ + 'model_id': 'qwen2.5-omni-3b-verified', + 'phase': 'verified', + 'pair_verified': true, + 'artifacts': { + 'main': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'main-pin', + }, + 'mmproj': { + 'installed': true, + 'verified': true, + 'digest_algorithm': 'sha256', + 'digest': 'projector-pin', + }, + }, + })); + } else if (request.uri.path == '/mobilecore/omni/load') { + await utf8.decoder.bind(request).join(); + loaded = true; + request.response.write('{"loaded":true}'); + } + await request.response.close(); + }); + + await expectLater( + service.loadVerifiedOmni( + requiredCapability: MobileCoreAttachmentKind.image, + ), + throwsA(isA().having( + (error) => error.code, + 'code', + 'omni_runtime_not_ready', + )), + ); + }); + test('rejects path-like lifecycle ids before contacting MobileCore', () async { var requests = 0;