Skip to content

examples: iOS live streaming ASR demo - #35

Closed
Kashif-E wants to merge 2 commits into
mudler:masterfrom
Kashif-E:ios-demo
Closed

examples: iOS live streaming ASR demo#35
Kashif-E wants to merge 2 commits into
mudler:masterfrom
Kashif-E:ios-demo

Conversation

@Kashif-E

@Kashif-E Kashif-E commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Adds examples/ios/, a SwiftUI demo that does live, on-device streaming ASR with parakeet.cpp: tap the mic, talk, watch the transcript stream in. Waveform on top, start/stop button in the middle, live transcript at the bottom, with a language picker.

It links libparakeet statically (Metal, embedded shader library) and drives the streaming C-API (parakeet_capi_stream_*) over mic audio captured with AVAudioEngine and resampled to 16 kHz mono.

Layout

  • project.yml — XcodeGen spec (source of truth; .xcodeproj is generated/gitignored)
  • scripts/build_xcframework.sh — cross-compiles libparakeet + ggml for device + simulator into Parakeet.xcframework
  • ParakeetDemo/Sources/ — SwiftUI app, AVAudioEngine capture, and a thin Swift wrapper over the streaming C-API

Verified

  • xcframework builds for ios-arm64 + ios-arm64_x86_64-simulator
  • App builds clean and runs on device (A16), transcribing live with a streaming model (nemotron-3.5-asr-streaming-0.6b)

Caveats

  • No CI coverage — the iOS build needs full Xcode + a streaming GGUF, so it's manual-verify / docs only.
  • Requires a streaming model; offline-only GGUFs fail stream_begin.
  • The model is bundled as a resource for the demo (gitignored); for production you'd download on first launch rather than ship a ~0.5 GB bundle.

Happy to gate this behind discussion if an in-tree iOS example isn't wanted.

Copilot AI review requested due to automatic review settings June 23, 2026 19:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new examples/ios/ SwiftUI demo app that performs live, on-device streaming ASR using parakeet.cpp’s streaming C-API, along with a script to build a static Parakeet.xcframework and an XcodeGen project spec to generate the Xcode project.

Changes:

  • Introduces an iOS SwiftUI demo app that captures mic audio with AVAudioEngine, resamples to 16 kHz mono, and feeds parakeet_capi_stream_* for streaming transcription.
  • Adds a build script to produce vendor/Parakeet.xcframework (device + simulator) and vendor the C-API header for Swift bridging.
  • Adds XcodeGen project.yml, README build instructions, and gitignores for generated/bundled artifacts.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
examples/ios/scripts/build_xcframework.sh Builds a static xcframework (device + simulator) and vendors parakeet_capi.h for Swift.
examples/ios/README.md Documents setup/build steps and demo constraints for the iOS example.
examples/ios/project.yml XcodeGen spec defining the app target, framework dependency, and build settings.
examples/ios/ParakeetDemo/Sources/TranscriberModel.swift Main UI model: permission, model/session lifecycle, feeding audio into streaming decode, publishing transcript/waveform state.
examples/ios/ParakeetDemo/Sources/ParakeetSession.swift Thin Swift wrapper over the streaming C-API with serialized access and memory management.
examples/ios/ParakeetDemo/Sources/ParakeetDemoApp.swift SwiftUI app entry point.
examples/ios/ParakeetDemo/Sources/Parakeet-Bridging-Header.h Exposes parakeet_capi.h to Swift via bridging header.
examples/ios/ParakeetDemo/Sources/ContentView.swift SwiftUI view for waveform, start/stop, language picker, and streaming transcript display.
examples/ios/ParakeetDemo/Sources/AudioStreamer.swift Captures mic audio and resamples to 16 kHz mono float blocks with an RMS level for visualization.
examples/ios/.gitignore Ignores generated Xcode project, built artifacts, and bundled GGUF model resource.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread examples/ios/README.md Outdated
Comment thread examples/ios/ParakeetDemo/Sources/ParakeetSession.swift
@mudler

mudler commented Jun 24, 2026

Copy link
Copy Markdown
Owner

It's a genuinely nice addition and the streaming integration looks well done.

But my main hesitation is maintenance. Neither I nor @richiejp have an Apple/macOS setup to build, run, or test an iOS target, so once this lands in the tree we'd be carrying code we can't actually exercise. That means we couldn't catch regressions, validate changes against new Xcode/SwiftUI/AVAudioEngine versions, or confidently review future updates to it, which isn't fair to either the project or to users who'd assume it's supported.

So rather than merging it into the core repo as-is, I'd love for this to live as a community-maintained example. If someone with an Apple toolchain is willing to step up and own it, keep it building, respond to issues, and update it as the C-API evolves, I'm very happy to support that, whether that's here with a clear "community-maintained, untested by core" note, or as a linked external/companion repo.

Really appreciate the effort, I just want to be upfront that we can't commit to maintaining something we have no way to test.

@Kashif-E

Kashif-E commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Makes complete sense, I'd rather you not carry something you have no way to test. I'll keep the iOS demo as a personal example rather than ask Core to maintain it. Happy to have it linked from the README as "community-maintained, untested by core" whenever useful. Thanks for the thoughtful review and for the library!

@Kashif-E Kashif-E closed this Jun 24, 2026
@mudler

mudler commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Makes complete sense, I'd rather you not carry something you have no way to test. I'll keep the iOS demo as a personal example rather than ask Core to maintain it. Happy to have it linked from the README as "community-maintained, untested by core" whenever useful. Thanks for the thoughtful review and for the library!

I'd be happy to link to in the README and have an apposite section for community projects, feel free to open up a PR once up, and I'll review it 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants