💡 On-device voice transcription for offline zero-tap inspections

[github-actions[bot]]

Summary

Integrate an on-device speech-to-text engine (e.g., Whisper.cpp or expo-speech-recognition with on-device models) so the zero-tap voice inspection workflow functions without cellular connectivity. The current architecture routes all voice processing through Gemini STT via Vertex AI API — a cloud dependency that breaks Broodly's core differentiator in the exact rural/field conditions where beekeepers operate. A hybrid approach (on-device transcription + cloud NLU when connected) preserves offline resilience while leveraging cloud intelligence for structured data extraction.

Market Signal

HiveSense already ships on-device Whisper AI for offline voice transcription with BLE sensor integration. HIVESOUND offers multi-colony continuous voice sessions (directly competing with Broodly's claimed core differentiator). BeeKeeperVoice and APiLOG both offer voice-driven inspection logging with AI-powered data extraction. Voice-first beekeeping is now table stakes — at least 4 competitors ship it — and offline voice processing is the new competitive frontier. The broader agritech UX trend in 2026 confirms voice-enabled, hands-free field interfaces as a baseline expectation.

User Signal

PRD Technical Constraints explicitly require "Field workflows must tolerate low-connectivity conditions" and "Logging must be fast and multimodal, with voice-first entry." The architecture's cloud-only STT (Gemini via Vertex AI API) creates a direct contradiction with these requirements. Beeyards are frequently in rural areas with limited or no cellular coverage — the exact conditions where Broodly's zero-tap promise matters most.

Technical Opportunity

Expo supports expo-speech-recognition for on-device STT. Whisper.cpp has React Native bindings (whisper.rn). The architecture already plans local media staging (audio captured locally, uploaded when connected) — adding local transcription is a natural extension of this pattern. On-device models handle raw transcription while cloud Gemini handles higher-order NLU (entity extraction, structured observation mapping, recommendation triggering) when connectivity returns. The async worker service architecture (Cloud Run triggered by Pub/Sub) already supports this deferred-processing pattern.

Assessment

Dimension	Score	Rationale
Feasibility	med	On-device STT libraries exist for React Native but Expo managed workflow compatibility needs validation. Model size (40-150MB) requires progressive download strategy.
Impact	high	Directly enables Broodly's core differentiator (zero-tap) in the environments where it matters most. Without this, the product promise is broken for a significant portion of the target market.
Urgency	high	HiveSense already ships this capability. Every month of delay is a month where a competitor offers what Broodly promises but can't deliver. Architecture decisions made now (cloud-only STT) will be expensive to retrofit.

Adversarial Review

Strongest objection: On-device Whisper models add 40-150MB to app size, React Native/Expo support for on-device ML is still maturing, and maintaining two STT paths (on-device + cloud) increases system complexity and testing surface.

Rebuttal: HiveSense already ships on-device Whisper successfully in a React Native app, proving the approach is production-viable. Model download can be optional and progressive (download on first use, not at install). The alternative — a "zero-tap" app that requires cellular connectivity to function — is a product-breaking contradiction that no amount of marketing can paper over. Beeyards ARE the low-connectivity environment. The dual-path complexity is manageable because the on-device path only needs to produce raw transcription text; all structured interpretation happens in the existing cloud pipeline.

Suggested Next Step

Spike: evaluate expo-speech-recognition on-device mode and whisper.rn library for Expo managed workflow compatibility. Measure model size options (tiny/base/small), transcription accuracy on beekeeping-specific vocabulary, and cold-start latency on mid-range iOS/Android devices.

---

Generated by weekly feature ideation workflow — 2026-06-26

[github-actions[bot]]

Weekly Update

What Changed

Significant platform and competitive shifts since this idea was created:

• Apple iOS 26 SpeechAnalyzer: Apple has replaced SFSpeechRecognizer with a new SpeechAnalyzer API — fully on-device, Swift concurrency-native, no network required. This is a major upgrade for field use and directly enables offline zero-tap inspections on iOS.
• Moonshine model (245M params): 6x smaller than Whisper Large v3 with matching accuracy. Built for streaming with minimal token revision. A strong candidate for cross-platform on-device deployment.
• Competitive crowding: At least 7 beekeeping apps now offer voice/hands-free features: HiveSense (offline Whisper), APiLOG (auto-transcription), HiveMasterPro (hands-free + offline sync), AI BeeKeeper (NFC + voice), Apiary Book (voice notes), BeeKing (voice + NFC), HIVESOUND (dictation). However, none offer zero-tap continuous multi-hive voice navigation — Broodly's differentiator holds.
• React Native ecosystem: expo-speech-recognition (jamsch) and expo-speech-transcriber are available as cross-platform wrappers. Both support on-device/offline modes.
• Google Chirp 3: Built-in denoiser, speaker diarization, auto language detection — relevant for noisy field environments.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→ (iOS 26 SpeechAnalyzer makes iOS path easier; cross-platform via expo-speech-recognition)
Impact	high	high	→ (zero-tap multi-hive remains unique despite 7+ competitors adding basic voice)
Urgency	high	high	↑ (competitors are rapidly adding voice features — execution speed matters to maintain lead)

Recommendation

Advance. The competitive window for zero-tap differentiation is narrowing as more apps add basic voice. iOS 26 SpeechAnalyzer should be the primary on-device path for Apple users. The expo-speech-recognition library enables cross-platform development. Priority: implement the continuous multi-hive voice session (FR19c) before competitors catch up to basic voice and start attempting session-level flows.

Sources:

• iOS SpeechAnalyzer Guide
• Moonshine vs Whisper comparison
• expo-speech-recognition
• Best Beekeeping Apps 2026 — HiveSense

[prashantweb3task-collab]

I think offline transcription is a must-have for this type of application. If the core use case is field inspections in areas with poor connectivity, relying entirely on cloud-based speech recognition creates a weak point in the user experience.

A hybrid approach makes a lot of sense use on-device transcription for speed and reliability, then sync with the cloud later for more advanced AI processing when a connection is available. That gives users the best of both worlds.

I'd also recommend benchmarking against some of the Best Voice to Text App solutions on the market to compare transcription accuracy, startup time, and performance in noisy outdoor environments. Those insights could help fine-tune the implementation and improve the overall experience.

Overall, this seems like a practical feature that aligns well with the product's offline-first vision. Looking forward to seeing how it evolves.

[github-actions[bot]]

Weekly Update — 2026-07-17

What Changed

Direct competitor has shipped this capability. HiveSense (iOS, $3.99/month hobbyist tier) now includes on-device Whisper-based voice transcription for offline inspection notes. This validates the market demand but also raises the urgency — on-device voice is no longer speculative, it's shipping product.

React Native ExecuTorch has matured significantly. The react-native-executorch library now provides a useWhisper hook for on-device speech-to-text via ExecuTorch, meaning Broodly can implement this on its existing Expo + React Native stack without leaving the framework. The library also offers useImageClassification, useObjectDetection, and useLLM hooks — a single integration path for multiple on-device AI capabilities.

On-device inference benchmarks are production-ready. Sub-20ms inference latency on Snapdragon 8 Elite devices, with models in the 1-5 GB range reaching practical quality thresholds. This means on-device Whisper can run real-time on mid-range and flagship phones from 2024 onward.

Sources: HiveSense 2026 Review, React Native ExecuTorch Docs, On-Device AI 2026 Benchmarks

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	high	↑ react-native-executorch provides turnkey useWhisper hook
Impact	high	high	→ still core to zero-tap differentiator
Urgency	med	high	↑ HiveSense has shipped this; competitive window narrowing

Recommendation

Advance. The technology is ready (react-native-executorch + Whisper) and a direct competitor has validated market demand. Recommend prioritizing on-device Whisper integration as part of the voice pipeline (Epic 11) to ensure the zero-tap inspection flow works offline — this is now a competitive necessity rather than a nice-to-have.

[github-actions[bot]]

Weekly Update — 2026-07-31

What Changed

New React Native / Expo voice libraries expand implementation options. Since the last update:

1. expo-speech-transcriber (new library): On-device speech transcription using Apple's Speech framework (iOS) and Android's SpeechRecognizer API. Works offline, privacy-focused. Key advantage: uses native platform APIs instead of bundling a separate model — no 142MB download required, but accuracy may be lower than Whisper.

2. rn-whisper-stt (new library): Offline speech-to-text for React Native using TFLite-optimized Whisper. Claims "fast, private, and efficient transcription without internet." This is a TensorFlow Lite approach complementing the existing whisper.rn (C++ port) option.

3. Picovoice's 2026 Complete Guide: Published a comprehensive React Native speech recognition guide covering on-device voice system architecture, model management, and performance optimization — useful reference for implementation planning.

Competitive landscape update: HiveSense's on-device Whisper approach (142MB one-time download) is confirmed working in production with iOS users. Their implementation validates the approach but also reveals a user friction point: requiring Wi-Fi for the initial model download. Broodly could differentiate by either (a) using native platform APIs (expo-speech-transcriber — smaller, no download, but less accurate) or (b) pre-bundling a smaller Whisper model in the app binary.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→ (More library options reduce risk; both native API and Whisper paths are viable)
Impact	high	high	→ (Core differentiator remains; HiveSense validates market demand)
Urgency	high	high	→ (HiveSense has shipped; Broodly's zero-tap orchestration layer is the differentiation, not transcription alone)

Recommendation

Advance — but shift framing. On-device voice transcription alone is no longer a differentiator (HiveSense has it). Broodly's advantage is the full zero-tap inspection orchestration: voice-driven hive navigation, observation capture, and photo triggers in a continuous multi-hive session. The transcription engine is infrastructure for that orchestration, not the feature itself. Implementation decision: evaluate expo-speech-transcriber (native APIs, no download) vs. rn-whisper-stt (higher accuracy, model download required) with a side-by-side accuracy benchmark on beekeeping-specific terminology.

[github-actions[bot]]

Weekly Update

What Changed

rn-whisper-stt now available for React Native. A production-ready React Native library providing offline speech-to-text using TFLite-optimized Whisper models is now available (github.com/israr002/rn-whisper-stt). Requires a one-time ~142MB model download. All processing happens on-device — no audio leaves the phone. This is the most direct technical path for offline voice transcription in Broodly's Expo + React Native stack.

Picovoice Rhino for voice command/intent recognition. Picovoice offers on-device intent recognition specifically designed for voice command parsing — available for React Native. This could power the "next hive", "move to Hive 4", "done with this hive" navigation commands without internet. Separate from transcription (which captures observations), intent recognition handles structured voice commands. Picovoice Cheetah provides streaming speech-to-text as an alternative to Whisper.

Apple Core AI Framework (WWDC 2026). Apple opened its on-device inference engine to third-party developers with a free 3B parameter model, Foundation Models Framework with guided generation, streaming, and tool calling. While this is a native Swift API (not directly callable from React Native), Expo modules could bridge to it for iOS-specific voice processing. This creates an iOS-only fast path option alongside the cross-platform Whisper approach.

Expo SDK 53+ audio improvements. expo-audio is now the stable, recommended audio library (replacing expo-av which will not be maintained from SDK 54). The New Architecture (JSI) enables synchronous native module calls — critical for real-time voice processing and camera frame analysis. 74.6% of Expo projects adopted New Architecture by April 2025.

Competitor validation: HiveSense ships offline Whisper voice. HiveSense launched with on-device Whisper AI for voice-to-text transcription, requiring a one-time ~142MB model download. This validates the technical approach and demonstrates user demand. However, HiveSense implements voice as note-taking (dictation), not the zero-tap continuous session flow that Broodly targets.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	high	↑
Impact	high	high	→
Urgency	high	high	→

Feasibility upgraded: the React Native libraries, Expo audio infrastructure, and production deployment by HiveSense collectively prove this is now technically viable with available tooling. The recommended architecture: rn-whisper-stt for observation transcription + Picovoice Rhino for structured navigation commands, with expo-audio (SDK 53+) for capture.

Recommendation

Advance — technical blockers are resolved. The hybrid architecture (Whisper for transcription + Picovoice for intent recognition) should be prototyped. Key decision: use rn-whisper-stt (cross-platform) vs. Apple Core AI (iOS-only but potentially higher quality). HiveSense's deployment validates user acceptance of the ~142MB model download tradeoff.

---

Generated by weekly feature ideation scan — 2026-08-31

[github-actions[bot]]

Weekly Update

What Changed

Critical competitive signal: HiveSense has shipped on-device voice transcription. HiveSense, a new beekeeping app identified in this week's market scan, ships on-device voice transcription via Whisper that "runs on-device" for "offline dictation." Their positioning — "every feature works without internet, including voice-to-text processing" — directly targets Broodly's zero-tap differentiator. They're priced at $3.99-$8.99/month with voice features available at all tiers.

HiveSense's competitive positioning explicitly calls out this gap. In their competitive comparison, HiveSense identifies that HiveTracks lacks "rock-solid offline field use or hands-free voice entry" and that Apiary Book lacks "deep offline reliability, voice entry, or direct BLE sensor reads." They are positioning voice+offline as their primary differentiator — the same positioning Broodly claims.

Broader market validation. Tunen, an agriculture voice assistant, demonstrates that domain-specific voice recognition (understanding agricultural terminology) significantly improves field usability. Inspectly360 offers voice-to-text in multiple languages for agriculture field audits. Voice-first field inspection is no longer experimental — it's becoming table stakes.

Technical path update: Gemini Nano vs. Whisper for on-device STT. See update on Discussion #382 for details on Gemini Nano as a potential on-device inference runtime. For STT specifically, options now include: (1) Whisper via ExecuTorch (what HiveSense uses), (2) Gemini Nano multimodal via AICore (Android) + Core ML (iOS), (3) Apple Speech Recognition framework (iOS-only, free). Each has different accuracy/latency/platform trade-offs.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	med	→ (multiple viable technical paths now validated in production)
Impact	high	high	→
Urgency	med	high	↑ (HiveSense is shipping; Broodly's core differentiator is at risk of being table stakes)

Recommendation

Advance with high priority. On-device voice transcription has moved from "nice to have for offline resilience" to "competitive necessity." HiveSense's market entry validates demand and proves feasibility, but their implementation lacks Broodly's AI-powered decision support layer — they capture what beekeepers say but don't guide what they should observe or recommend next actions. Broodly's response should be: match on-device transcription capability, then differentiate on the intelligence layer (confidence-scored recommendations, adaptive observation prompts, contextual guidance). Suggested next step: benchmark Whisper (via ExecuTorch), Gemini Nano STT, and Apple Speech Recognition on beekeeping-specific vocabulary (hive terms, observation language, action commands) to select the optimal runtime.

[github-actions[bot]]

Weekly Update

What Changed

Apple SpeechAnalyzer (iOS 26) — game-changer for on-device STT:
Apple shipped SpeechAnalyzer in iOS 26 (announced WWDC 2025), benchmarking at 2.2x faster than Whisper Large V3 on long-form conversational speech (9to5Mac, Picovoice, Blake Crosley benchmarks, June 2025). Key characteristics that directly benefit Broodly's zero-tap flow:

• Long-form audio: Trained for sustained transcription over minutes or hours, not just short queries — purpose-built for continuous inspection sessions
• Fully on-device: No network calls, fast and private
• System-managed: Automatic model updates via OS updates, no model download overhead
• Low battery: System-optimized inference path

Google ML Kit GenAI Speech Recognition:
Google shipped the ML Kit GenAI Speech Recognition API using on-device Gemini Nano. Available on Pixel 10+ devices. Provides equivalent on-device STT capability for Android users.

New competitor validation — HiveSense:
HiveSense (tryhivesense.com) launched with on-device Whisper for voice-to-text, validating market demand for offline voice in beekeeping. Free for up to 15 hives, $5.99/mo Pro. However, HiveSense treats voice as dictation-to-form, NOT as a continuous hands-free workflow — Broodly's zero-tap multi-hive flow with voice-commanded navigation remains unique.

React Native ecosystem:

• whisper.rn (React Native binding of whisper.cpp) is stable for cross-platform fallback
• React Native ExecuTorch v0.3+ ships Whisper for STT and Kokoro for TTS
• rn-whisper-stt offers a 1.2MB TFLite Whisper model for ultra-lightweight offline STT

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→ (now with THREE platform options instead of one)
Impact	high	high	→
Urgency	high	high	↑ (HiveSense competitor validates market; SpeechAnalyzer removes iOS technical risk)

Recommendation

Advance. The recommended implementation strategy is now a three-tier platform-native approach:

1. Apple SpeechAnalyzer on iOS 26+ as primary — fastest, lowest battery, automatic model updates, purpose-built for long-form inspection transcription
2. Google ML Kit GenAI Speech on Android — equivalent on-device capability via Gemini Nano
3. whisper.rn or ExecuTorch Whisper as cross-platform fallback for older OS versions

This replaces the previous Whisper-only approach. The platform abstraction layer (src/platform/) is already designed for this pattern. Each platform gets its best-in-class engine with graceful fallback. Concrete next step: benchmark SpeechAnalyzer vs whisper.rn on a representative beekeeping voice corpus (outdoor ambient noise, domain-specific vocabulary like "Varroa," "super," "brood pattern") to validate accuracy claims for this specific use case.

[github-actions[bot]]

Weekly Update

What Changed

Critical competitive development: HiveSense, a new beekeeping app, has shipped on-device voice transcription via Whisper (~181MB bundled model) with a free tier supporting 15 hives. This directly implements the capability proposed in this discussion. HiveSense's approach:

• Whisper model runs locally, no internet required
• One-tap voice/photo/text note capture from the hive screen
• Transcription includes confidence indication ("tells you honestly when it wasn't sure")
• Free tier includes offline voice (Pro at $5.99/mo)

ApiNote 4.5 (September 2026) also added an AI Voice Assistant for hands-free inspection record keeping.

Technology maturation: Multiple React Native / Expo solutions now exist:

• react-native-executorch — Meta-backed, supports Whisper natively
• whisper.rn — dedicated Whisper wrapper, ~150MB model
• expo-speech-transcriber — uses platform-native speech recognition (iOS 17+ / Android 13+ offline)
• @jamsch/expo-speech-recognition — Expo-compatible speech recognition

Platform-native speech recognition (no model download) is available offline on iOS 17+ and Android 13+ using built-in language models.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→ (multiple proven approaches now available)
Impact	high	high	→ (still critical for zero-tap promise)
Urgency	med	high	↑ (HiveSense shipping this in free tier; ApiNote added voice AI)

Recommendation

Advance immediately. On-device voice transcription has moved from "planned differentiator" to "competitive catch-up." HiveSense's free tier offering means Broodly cannot treat this as a premium feature. Recommend evaluating platform-native speech recognition first (zero download, iOS 17+/Android 13+ offline) as the fastest path, with ExecuTorch/Whisper as a quality upgrade for users who want higher accuracy. See also new discussion #582 for leveraging Apple Intelligence and Gemini Nano as an alternative approach.