⬅️ Large Language Marmalade 🧭 Reflections on the Rise of Modern Science ➡️

A Spotlight on A.I. on Devices

November 5, 2023 by Jay Cuthrell

Share and discuss on LinkedIn or HN

Origins

This week we take a look at A.I. running on devices nearest the data. Fudge Sunday inspiration for this week comes from a recent comment at M.G. Siegler’s 500ish.com:

My working theory is that “the cloud” will have some advantages and then there will be the “on device” advantages heavily marketed by the same Apple which has been angling ever closer to making supercomputers that fit in your lap and in your hand. So, even if you are “doing email” there will be abilities for user opt-in beneficial machine learning (A.I.) on your activities that provide recommendations, improvements, and surfacing knowledge of the past that are exclusively “on device” capabilities. Said another way, Spotlight is ripe for being put in… the spotlight.

Source: Apple’s Literal “Scary Fast” Event on 500ish by M.G. Siegler

In related inspiration, if you aren’t already reading Matt Baker’s newsletter or following his updates on LinkedIn, here are a few recent gems:

[…] it will run in the cloud, in the datacenter, at the edge, and on your workstations and notebooks
Source: LinkedIn

[…] the author massively optimized the model adaptation process. Performing fine-tuning on a 7 billion parameter LLM on a SINGLE rather small GPU with only 16GB of memory.
Source: LinkedIn

Devices Doing Deep Data Duties

Okay, sounds great, right? More powerful mobile devices means more ways to see A.I. appear in our pockets!

Oh wait. What about my device battery life?

🤖 + 📱 = 🪫 😭

Device battery life is a reasonable concern. However, there are also a growing number of examples where groups of research and development minded folks are assuming devices will be doing more data intenstive operations in the near future while simultaneously contemplating (avoiding?) battery anxiety in addition to overall power requirements.

Expect a growing list of A.I. initatives focused on determinstic power, weight, cooling, and geometry. I shoud make that an acronym and tag past issues accordingly. DPWCG perhaps?

Also, mark your calendars. As we approach May 2024, analyst coverage of on device A.I. will reach a natural one (1) year looking back moment where much content and digital ink will be spilled.

After all, there is that whole peak of expectations thing to consider, right? 🤓