Apple Machine Learning Research

Most AI stories in 2025 still orbit the cloud: giant models, branded “copilots,” and oceans of user data flowing off your devices. On the Mac, the direction is more subtle — and arguably more interesting. With macOS Sequoia and Apple Intelligence, Apple is turning the Mac into a *device-first AI machine*: intelligence built into the operating system, models that run increasingly on your own hardware, and developer tools that treat AI as part of normal computing, not a separate destination. --- ## macOS Sequoia + Apple Intelligence: AI as Part of the Interface Apple’s latest desktop release, *macOS Sequoia*, looks like a classic productivity update — iPhone Mirroring, a smarter Safari, a dedicated Passwords app. But it’s also the main delivery vehicle for *Apple Intelligence*, Apple’s new system-wide AI layer. Official overviews: - Apple Intelligence: https://www.apple.com/apple-intelligence/ - macOS Sequoia announcement: https://www.apple.com/newsroom/2024/06/macos-sequoia-takes-productivit… On macOS, Apple Intelligence shows up as small, targeted upgrades: Apple’s machine-learning hub for developers lays out that strategy: - Machine Learning & AI on Apple platforms: https://developer.apple.com/machine-learning/ Key pieces that sit naturally on macOS: - *Core ML* – runs optimized ML models on Apple silicon and Intel Macs, from image recognition to language models: …

Apple has an ML research group. They do a mixture of obviously-Apple things, other applications, generally useful optimizations, and basic research. https://machinelearning.apple.com/

I didn't see this, thank you! They have a follow-up as well: https://machinelearning.apple.com/research/ferret-ui-2

This is close to what the apple paper [1] also found on constraint satisfaction problems. As an example, on towers of hanoi, past a frontier, accuracy collapses. Even when the algorithm steps are laid out precisely, they cannot be followed. Perhaps, LLMs should be trained on turing machine specs and be given a tape lol. Constraint satisfaction and combinatorics are where the search space is exponential, and the techniques are not formalized (not enough data in training set), and remain hard for machines as seen in the Problem 6 of IMO which could not be solved by LLMs. I suspect, there is this aspect of human intelligence which is not yet captured in LLMs. [1] - https://machinelearning.apple.com/research/illusion-of-thinking

In the follow-up, they go all the way to 3.8 billion parameters: https://machinelearning.apple.com/research/starflow

The on-device models used for Apple Intelligence (writing tools, notification and email/message summaries, etc.) are around ~3B parameters. I mean, they could be better (to put it nicely), but there is a legitimate use-case for them and I'd love to see more work in this space. https://machinelearning.apple.com/research/introducing-apple-foundatio… https://arxiv.org/abs/2407.21075

Quantum computers don't and won't meaningfully exist for a while, and once they do exist, they still won't be able to crack it. Quantum computers aren't this magical "the end is nigh" gotcha to everything and unless you're that deep into the subject, the bigger question you've got to ask yourself is why is a magic future technology so important to you that you just had to post your comment? Anyway, back to the subject at hand; here's Apple on that subject: > We use BFV parameters that achieve post-quantum 128-bit security, meaning they provide strong security against both classical and potential future quantum attacks https://machinelearning.apple.com/research/homomorphic-encryption https://security.apple.com/blog/imessage-pq3/

NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that. LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :) https://machinelearning.apple.com/research/gsm-symbolic https://arxiv.org/pdf/2410.05229 Paper came out in October, I don't think many have fully absorbed the implications. It's hard to take any of the claims of "LLMs can do reasoning!" seriously, once you understand that simply changing what names are used in a 8th grade math word problem can have dramatic impact on the accuracy.

The article seems to be based on misinterpreted information: > Apple’s model has extremely low latency (0.6 milliseconds to first token), outperforms similar sized Phi and Gemini models from Microsoft and Google. From [1] it is 0.6 millisecond per prompt token so unless the prompt is one token the latency would be higher. > Apple’s server-side models running in the Private Cloud Compute are apparently quite near the GPT-4o in terms of quality The benchmarks from [1] never mentioned GPT-4o, the best model they compared to for the server model is GPT-4-0125. If their model almost matches GPT-4o they wouldn't need to integrate with ChatGPT. [1]: https://machinelearning.apple.com/research/introducing-apple-foundatio…

> they have to be employing serious researchers, but they’ll never get to publish on volition. That’s not true. I wouldn’t know how free they are to publish but they do publish stuf. See https://machinelearning.apple.com/

There's a lot of information here about chips which are mostly built for training neural networks. It's worth noting there are very widely deployed chips primarily built for inference (running the network) especially on mobile phones. Depending on the device and manufacturer sometimes this is implemented as part of the CPU itself, but functionally it's the same idea. The Apple Neural Engine is a good example of this. This is separate to the GPU which is also on the CPU. Further information is here: https://machinelearning.apple.com/research/neural-engine-transformers The Google Tensor CPU used in the pixel has a similar coprocessor called the EdgeTPU.

https://machinelearning.apple.com/research/recognizing-people-photos > Photos uses a number of machine learning algorithms, running privately on-device, to help curate and organize images, Live Photos, and videos. An algorithm foundational to this goal recognizes people from their visual appearance. Photos can also learn from identity information to build a private, on-device knowledge graph that identifies interesting patterns in a user’s library, such as important groups of people, frequent places, past trips, events, the last time a user took an image of a certain person, and more. The knowledge graph powers the beloved Memories feature in Photos, which creates engaging video vignettes centered around different themes in a user’s library.

Interesting. But how about the Apple Neural Engine (ANE)? I've always wondered if the ANE is ML worthy, maybe it's really only with inference or who knows, even training somehow. I've seen Apple's marketeers bragging about it [1], with even code examples, but ifaik no useful libraries nor reliable measurements and community interest exist in the wild for doing ANE ML on Macs. 1. https://machinelearning.apple.com/research/neural-engine-transformers Edit: just found this: https://github.com/ggerganov/whisper.cpp/pull/566

They have something for ML: https://machinelearning.apple.com

Siri is not where it needs to be because Apple refuses to mine user data to enrich it. They also are very hesitant to allow researchers to publish their breakthroughs which makes recruitment very hard. Although this is changing https://machinelearning.apple.com/

Remember when Apple created a “Machine Learning journal”? Well, it seems like they’ve stopped publishing to it and now have gone back to introducing stuff at presentations, if at all: https://machinelearning.apple.com/

Apple also has a blog where they discuss (some of) their machine learning results[1]. 1: https://machinelearning.apple.com/

Actually they fixed that 2 years ago: https://www.theverge.com/platform/amp/2016/12/6/13858354/apple-publish… https://machinelearning.apple.com/

Apple's epsilon reset problem is real, but it's worth pointing out that they use additional heuristics based on hashing that plausibly add another layer of privacy [1]. Plausibly, not provably, but it's a bit more than just resetting epsilon. I believe Google and Microsoft use similar tweaked forms of differential privacy. In particular, note that all of these companies -- again, going off public papers -- use the "local" variant of differential privacy, which requires less trust on the user's part. The question of "lifetime" differential privacy, for a single user across different computations and datasets, is still fairly open as far as I know. [1] https://machinelearning.apple.com/docs/learning-with-privacy-at-scale/…

I am wondering what their baseline is. They call it "Current Best Non-WaveNet". Quite frankly, Apple's most recent deep learning-based speech synthesis sounds superior, but there aren't enough samples to for a proper comparison: https://machinelearning.apple.com/2017/08/06/siri-voices.html

It does: https://machinelearning.apple.com/feed.xml It's "just" not referenced in the source. I get that social networks are the tool of choice to prioritise articles from the major news outlets–if The Atlantic publishes something spectacular, it will find me. But how on earth am I supposed to follow something like this, when I have to guess randomly to even find a feed? Are they expecting me to bookmark it and check for news once a week? There's no Twitter account, newsletter signup, Facebook link... nothing.

Craig Federighi SVP of Software at Apple “The keyboard now leverages a transformer language model, which is state of the art for word prediction, making autocorrect more accurate than ever.”[1] "New features available with macOS Sonoma" doc states "Autocorrect fixes mistakes for you more accurately than ever before by leveraging a new transformer language model in English, French, and Spanish keyboards." [2] "Introducing Apple’s On-Device and Server Foundation Models" blog post states "Apple Intelligence is comprised of multiple highly-capable generative models that...have been fine-tuned for user experiences such as writing and refining text.. In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model."[3] "A look at Apple’s new Transformer-powered predictive text model" blog post[4] is the kind of deep dive you'd have to do to find out what's going on under the hood in your spelling/grammar checker. TL;DR AppleSpell uses GPT-2 which proves my point. [1] https://eu.usatoday.com/story/tech/2023/06/06/wwdc-2023-apple-ios17-au… [2] https://www.apple.com/mideast/macos/sonoma/pdf/en/macOS_All_New_Featur… [3] https://machinelearning.apple.com/research/introducing-apple-foundatio… [4] https://jackcook.com/2023/09/08/predictive-text.html

Y'all did such a good job with this. It captivated HN and was the top post for the entire day, and will probably last for much of tomorrow. If you don't know already, you need to leverage this. HN is one of the biggest channels of engineers and venture capitalists on the internet. It's almost pure signal (minus some grumpy engineer grumblings - we're a grouchy lot sometimes). Post your contract info here. You might get business inquiries. If you've got any special software or process in what you do, there might be "venture scale" business opportunities that come your way. Certainly clients, but potentially much more. (I'd certainly like to get in touch!) -- I believe volumetric is going to be huge for creative work in the coming years. Gaussian splats are a huge improvement over point clouds and NeRFs in terms of accessibility and rendering, but the field has so many potential ways to evolve. I was always in love with Intel's "volume", but it was impractical [1, 2] and got shut down. During the pandemic, to get over my lack of in-person filmmaking, I wrote Unreal Engine shaders to combine the output of several Kinect point clouds [3] to build my own lightweight version. The VGA resolution was a pain and I was faced with fpga solutions for higher real time resolution, or going 100% offline. Today, World Labs and Apple are doing exciting work with image-to-Gaussian models [4, 5], and World Labs created the fantastic Spark library [6] for viewing them. Most recently,…

"The apple study" is being overblown too, but here it is: https://machinelearning.apple.com/research/illusion-of-thinking The crux is that beyond a bit of complexity the whole house of cards comes tumbling down. This is trivially obvious to any user of LLMs who has trained themselves to use LLMs (or LRMs in this case) to get better results ... the usual "But you're prompting it wrong" answer to any LLM skepticism. Well, that's definitely true! But it's also true that these aren't magical intelligent subservient omniscient creatures, because that would imply that they would learn how to work with you. And before you say "moving goalpost" remember, this is essentially what the world thinks they are being sold. It can be both breathless hysteria and an amazing piece of revolutionary and useful technology at the same time. The training set argument is just a fundamental misunderstanding, yes, but you should think about the contrapositive - can an LLM do well on things that are _inside_ its training set? This paper does use examples that are present all over the internet including solutions. Things children can learn to do well. Figure 5 is a good figure to show the collapse in the face of complexity. We've all seen that when tearing through a codebase or trying to "remember" old information.

Let’s not forget that every round trip with the LLM costs latency (and extra input tokens). We now have parallel tool calls which sometimes works in some models[1]. But it’s great because now a model can say “write these 3 files then read these 2 files” before the time-to-first token latency is incurred once more (not to mention input token cost). I think LLMs will indirectly move towards being fuzzy VMs that output tokens much like VM instructions so they can prepare multiple conditional branches of tool calling, load/unload useful subprograms, etc. It might not be expressed exactly like that, but I think given how LLMs today are very poor at reusing things in their context window, we will naturally add features that take us in this direction. Also see frameworks like CodeAct[2] etc. [1] This can be converted to a single tool call with many arguments instead, which you’ll see providers do in their internal tools, but it’s just messier. [2] https://machinelearning.apple.com/research/codeact

My speculation is that the small on-device models are simply not useful enough for practical purposes. The on-device transformer models are described as having 3B parameters[0]. Their own figures from 6 months ago show that humans prefer other models' output 25-40% of the time, which aligns with Gurman's reporting. I don't know how well Apple's A-series chips can handle transformers, but if you play with their on-device diffusion models (Image Playground, Genmoji), you can watch your battery drop before your eyes. And the output of these models is also embarrassingly behind the state of the art. If Apple can make great foundation models for Private Cloud Compute, that's great. But then what's the point of buying a new iPhone for on-device inference capabilities? Secondly, I speculate that allowing a server-side model to query the personal context database is going to be hard to do performantly, requiring several round trips and uploading fragments of the data to the cloud for processing. 0: https://machinelearning.apple.com/research/introducing-apple-foundatio…

> I'd also like to ask a different question: if there's no reason to ever worry about this feature, then why is there even an option to turn it off in the first place? I mean for one, because of people like you that are concerned about it. Apple wants you to have the choice if you are against this feature. It's silly to try to use that as some sort of proof that the feature isn't safe. My iPhone has a button to disable the flash in the camera app. Does that imply that somehow using the camera flash is dangerous and Apple is trying to hide the truth from us all? Obviously not, it simply means that sometimes you may not want to use the flash. They likely chose to make it opt-out because their research shows that this is truly completely private, including being secure against future post-quantum attacks. > If an adversary is able to intercept encrypted communications, they can store it in hopes of decrypting it in the future in the event that a feasible attack against the cryptosystem emerges. I don't know how likely this is to happen against homomorphic encryption schemes, but the answer is not zero. Also, if you're going to wildly speculate like this it is at least (IMO) worth reading the research press release since it does answer many of the questions you've posed here[0]. > it's pretty upsetting that it's becoming incredibly hard to the point of being nearly impractical to get modern devices to behave this way and not just fling data around all over the place…

It seems obvious to me that LLMs wouldn't be able to find examples of every single problem posed to them in training data. There wouldn't be enough examples for the factual look up needed in an information retrieval style search. I can believe that they're doing some form of extrapolation to create novel solutions to posed problems. It's interesting that this paper doesn't contradict the conclusions of the Apple LLM paper[0], where prompts were corrupted to force the LLM into making errors. I can also believe that LLMs can only make small deviations from existing example solutions in creation of these novel solutions. I hate that we're using the term "reasoning" for this solution generation process. It's a term coined by LLM companies to evoke an almost emotional response on how we talk about this technology. However, it does appear that we are capable of instructing machines to follow a series of steps using natural language, with some degree of ambiguity. That in of itself is a huge stride forward. [0] https://machinelearning.apple.com/research/gsm-symbolic

erm... https://machinelearning.apple.com/

Conceptually, to the best of my understanding, nothing too serious; perhaps the inefficiency of processing a larger input than necessary? Practically, a few things: If you want to have your cake & eat it too, they recommend Enumerated Shapes[1] in their coremltools docs, where CoreML precompiles up to 128 (!) variants of input shapes, but again this is fairly limiting (1 tok, 2 tok, 3 tok... up to 128 token prompts.. maybe you enforce a minimum, say 80 tokens to account for a system prompt, so up to 200 tokens, but... still pretty short). But this is only compatible with CPU inference, so that reduces its appeal. It seems like its current state was designed for text embedding models, where you normalize input length by chunking (often 128 or 256 tokens) and operate on the chunks — and indeed, that’s the only text-based CoreML model that Apple ships today, a Bert embedding model tuned for Q&A[2], not an LLM. You could used a fixed input length that’s fairly large; I haven’t experimented with it once I grasped the memory requirements, but from what I gather from HuggingFace’s announcement blog post[3], it seems that is what they do with swift-transformers & their CoreML conversions, handling the details for you[4][5]. I haven’t carefully investigated the implementation, but I’m curious to learn more! You can be sure that no one is more aware of all this than Apple — they published "Deploying Transformers on the Apple Neural Engine" in June 2022[6]. I look forward to…