WebML WG - TPAC 2024 agenda

[anssiko]

Important

This is a work in progress draft agenda for the WebML WG meeting at TPAC 2024. Agenda proposals and suggestions welcome via comments. We expect to announce a more baked-in agenda in August. 🌱

Registration

• Registration is now open until 13 September 2024: https://www.w3.org/register/tpac2024
• All registered WebML WG/CG participants and invited guests welcome.

Logistics

• 🏨 Venue: Hilton Anaheim in-person & remote
• 🗓️ Meeting invite: 23 September 2024, 09:00–18:00 Pacific Daylight Time
• ⏰ Schedule: 23 September 2024, 09:00–18:00 PDT local time (see your local time)
  • ⏰ Start: 9:00 local / 16:00 UTC
  • ⏰ Morning break: 10:30 - 11:00 local / 17:30 - 18:00 UTC
  • ⏰ Lunch: 12:30 - 13:30 local / 19:30 - 20:30 UTC
  • ⏰ Afternoon break: 14:30 - 15:00 local / 21:30 - 22:00 UTC
  • ⏰ End: 18:00 local / 00:00 UTC
• 📞 Zoom details: see 🗓️ Meeting invite > Joining Instructions
• ✍️ IRC: https://irc.w3.org/?channels=#webmachinelearning or irc://irc.w3.org:6667/#webmachinelearning
• ✍️ Minutes:
  • https://www.w3.org/2024/09/23-webmachinelearning-minutes.html
• 💬 Slack: https://w3ccommunity.slack.com/#webmachinelearning (sign-up instructions)

Related meetings

Note

Any related meetings of interest to the WebML WG/CG participants to be listed here.

• Check out the breakout schedule for 1-hour sessions on new or existing topics

Agenda

Warning

This is a Living Agenda, please expect updates. We'll do the usual agenda bashing at the beginning of the meeting for any last-minute additions subject to available agenda time.

⏰ Start meeting 9:00 local / 16:00 UTC

👋 Welcome

• Intros
• Agenda bashing

📘 Charter orientation

• WG and CG interplay
• Current scopes, future aspirations
• Deliverables: WebNN, Ethical Principles, tentative Model Loader
• Coordination with other groups

😇 Ethics

• 📍9:30-10:00 In-browser explainability libraries and viz tools (@xiaohk, OpenAI / Georgia Tech) - slides

⏳ Spec orientation

• Triage pass through open issues: breaking changes, priorities, next steps for the issue - see triage guidance (All)

⌛ Break 10:30 local / 17:30 UTC

⏰ Resume meeting 11:00 local / 18:00 UTC

✨ New features

• A refreshed analysis of popular models, operator & data type gaps (@fdwr) - slides
• Quantization and dequantization (QDQ), a bag of multiple issues (@inexorabletash)
• Platform capability detection (@philloooo) #463 #755 opSupportLimits()
• Future-proof device selection abstractions (@mwyrzykowski) - slides

🍴 Lunch 12:30 local / 19:30 UTC

⏰ Resume meeting 13:30 local / 20:30 UTC

📢 Customer feedback & collaborations

• 📍13:30-14:00 Universal Large-language Model
  Deployment with ML Compilation (@tqchen) - slides
• Transformers.js WebNN backend (@xenova) - Initial WebNN Support huggingface/transformers.js#890, slides
• ONNX Runtime Web & WebNN EP
• Google Chrome Feedback revisited #453
• Other standards positions #763

🤝 Interop and cross-group coordination

• Interop issues across different backends (@huningxin)
• Core operator set #573
• MLTensor (was MLBuffer) #754 (@a-sully) and #760 (@bbernhar)
• MLConstantOperand #668 (@reillyeon)

☕ Afternoon break 16:00 - 16:30 local / 23:00 - 23:30 UTC

⏰ Resume WG meeting 16:30 local / 23:30 UTC

🔮 Implementation plans and trials

• Next step for implementations, Origin Trial or equivalent and align with framework developer feedback

🛤️ Advancement on the W3C Rec Track

• Wide review status, close on TAG review feedback
• W3C “living standards” expectations (@dontcallmedom)

🌱 Incubations

• Custom ops (@huningxin) - slides
• Built-in APIs for translation and prompting (@domenic) - slides
• Model management (@mmccool)

🫶 Wrap up

• Synthesis of key outcomes, next steps

⏰ End meeting 18:00 local / 01:00 UTC

• 🍴 Group dinner

[domenic]

Hi! Over on the Chrome team, we'd love to discuss some of our work on built-in AI APIs with the group. (Currently published: translation and prompting.)

We know this doesn't currently fit in the group's charter, but I think the group previously had a brief discussion about how it might fit in the future. And, we suspect a lot of the right people will be in the room.

We're also happy to join one of the telecons before TPAC if that would be helpful.

[inexorabletash]

It might be productive to spend some cycles doing a triage pass through open issues:

• Breaking? Would acting on this issue impact code such as webnn-samples or ONNX Runtime Web?
• Priority? Is this a blocker for milestones such as an "origin trial" of the API?
• Next steps? What action is required, especially for older/broader issues?

This could be all consuming so definitely time-box (overall and per-issue), but real-time discussion could help make progress.

[inexorabletash]

Potential topic: Quantization / Dequantization (QDQ)

Including issues such as:

• Add QuantizeLinear and DequantizeLinear for mixed precision webnn#93
• WebNN should support int8 quantized models webnn#128
• WebNN should support NPU and QDQ operations webnn#623
• And specifically past discussions linked from WebNN should support int8 quantized models webnn#128 (comment)

And topics including:

• Are there key decisions to make, e.g. between operand descriptor encapsulate scale/zp vs. dedicated Q/DQ ops that frameworks introduce (and impls may optimize out) vs. variants of ops specialized for Q data?
• Quantization data types beyond int8 (int4, int2)
• Is support a requirement for v1?

[inexorabletash]

Potential topic: Should we make the spec a "living standard" from the get-go? TPAC might be a good place to identify concerns and walk through this with W3C experts.

[anssiko]

The group discussed F2F agenda on our call today. We reviewed proposals and solicited input on additional F2F topics.

F2F topics proposed in this issue (thanks @inexorabletash @domenic) received support:

• built-in APIs for translation and prompting
• triage pass through open issues at F2F
• quantization and dequantization (QDQ)
• W3C “living standards” topic

@huningxin proposed:

• interop issues across different backends
• next step for implementation, Origin Trial or equivalent and align with framework developer feedback
• incubation for custom ops

@mwyrzykowski proposed:

• MLBuffer and WebGPU interop
• (See also Add MLTensor explainer webnn#754)

@mmccool proposed:

• incubation for model management AI Model Management w3c/tpac2024-breakouts#15

@anssik proposed:

• platform capability detection (opSupportLimits())
• future-proof device selection abstractions
• "customer-centric session" to hear from people working on frameworks that integrate with WebNN API: ONNX Runtime Web & WebNN EP, demo of Transformers.js v3 with WebNN integration
• ML ethics: WebSHAP tool author @xiaohk from OpenAI / Georgia Tech to present on in-browser explainability libraries and viz tools, ~30 mins for presentation + Q&A, prefers AM PDT timeslot
• Wide review status: close on any remaining TAG review feedback (resource contention considerations ...)
• Core operator set webnn#573 revisited

@domenic, the group is happy to hear more from built-in AI APIs for translation and prompting. We'd like to invite you to attend our TPAC meeting to discuss these proposals with us. Please let us know, including any time preference you may have for 23 Sep.

[domenic]

@domenic, the group is happy to hear more from built-in AI APIs for translation and prompting. We'd like to invite you to attend our TPAC meeting to discuss these proposals with us. Please let us know, including any time preference you may have for 23 Sep.

Awesome! Coincidentally, we just finished up publishing the rest of our explainers. In total we have:

• WICG/translation-api: translator and language detector APIs
• explainers-by-googlers/writing-assistance-apis: summarizer, writer, and rewriter APIs. We hope to move these to WICG soon: Writing Assistance APIs WICG/proposals#163
• explainers-by-googlers/prompt-api: the prompt API, which is extra-experimental and is not yet proposed to moving to WICG as its ultimate destiny is unclear.

We hope to present on all of them together, under the heading of "built-in AI APIs".

I will be available any time on September 23.

[anssiko]

@domenic great to hear, we'll come up with a more baked in agenda this month and will slot you in.

Adding to the pile of proposals: for our possible ML ethics / transparency & explainability discussion, adding Visual Blocks visual programming framework. I believe @xenova added a few new nodes to this framework for familiar vision, NLP, other tasks. While not purpose-built to solve ethical issues, this tool kind of serves that purpose too while being educational also. Very demo worthy, see "Example" links at https://huggingface.co/hf-vb for client-only nodes and blog for hybrid server + client pipeline example.

[anssiko]

Adding to the proposed F2F topics:

• a refreshed analysis of popular models, operator & data type gaps by @fdwr et al., includes also proposed IDL amendments: Support for transformers webnn#375 (comment)

[anssiko]

More exciting agenda updates:

• @tqchen is joining us for an update on WebLLM and other projects that build upon the Machine Learning Compilation solution. The group received an introduction from TQ earlier this year. Possible synergies with the proposed incubation for custom ops from @huningxin et al. TQ's availability 12-2 PM PDT (3-5 PM EDT). ~20-30 min slot.

• I'm also pleased to confirm @xiaohk from OpenAI / Georgia Tech will join us to present on in-browser explainability libraries and viz tools. This contributes to our ethics deliverable. Jay's preference AM PDT timeslot.

[mmccool]

Regarding AI Model Management, while there is a breakout later in the week I'd like 10-15m in the "main call" to summarize the issues and alternatives and gather input.

[anssiko]

The proposed WebML TPAC 2024 agenda topics have been assigned timeslots, please take a look: #25 (comment)

I've assigned the following timeslots to our guest speakers:

📍9:30-10:00 In-browser explainability libraries and viz tools (@xiaohk, OpenAI / Georgia Tech)
📍13:30-14:00 WebLLM + MLC (@tqchen)

For other topics, we have some wiggle room to move things around.

You should have received a meeting invite that links back to this agenda issue. Zoom joining instructions will appear in the meeting invite link around 19 Sep. You can also manually export the invite as needed.

Thank you all for your contributions and proposals. I am looking forward to meeting you all for a day of discussions on these exciting and diverse topics.

[reillyeon]

I'd like to add the following issue to the agenda:

• Do we need an MLConstantOperand? webnn#668

[anssiko]

@reillyeon thanks for the proposal, added to our set of interop issues.

[bbernhar]

I'd also like to add webmachinelearning/webnn#760

[anssiko]

Added Tranformers.js update materials from @xenova and webmachinelearning/webnn#760 from @bbernhar.