Core ML vs Foundation Models: Which Should You Use?

Source: Dev.to

Source: Dev.to – Core ML vs Foundation Models: Which Should You Use?

iOS 26.3: Core ML vs. Foundation Models

With iOS 26.3 now in the wild, iOS developers have two powerful on‑device AI frameworks to choose from:

• Core ML – Apple’s veteran ML inference engine.
• Foundation Models – the new framework that exposes Apple’s ~3 B‑parameter LLM from iOS 26 onward.

They sound like they do similar things. They don’t.

This article cuts through the confusion, explains what each framework is actually designed for, and gives you a clear decision framework so you can pick the right tool for the job — or know when to combine both.

What Each Framework Actually Is

Before comparing them, let’s be precise about what they are.

Core ML has been available since iOS 11. It’s a general‑purpose inference engine — you bring a trained model (in .mlmodel or .mlpackage format), and Core ML runs it on‑device using the best available hardware (Neural Engine, GPU, or CPU) depending on the task. Core ML itself doesn’t contain any models. You either:

• Train your own (with Create ML),
• Convert from TensorFlow or PyTorch, or
• Download pre‑trained ones.

It supports image classification, object detection, NLP, audio analysis, tabular data prediction, and more.

Foundation Models was introduced in iOS 26. It’s an API that gives you direct access to Apple’s own pre‑trained ~3 B‑parameter large language model — the same one behind Apple Intelligence. You don’t bring a model; the model is already on the device (downloaded as part of enabling Apple Intelligence). The framework specializes in natural language: text generation, summarisation, structured data extraction, and tool calling.

Bottom line: Core ML is a runtime that runs your models. Foundation Models is an API for a specific pre‑built Apple model.

Device Availability: A Critical Difference

This is where the two frameworks diverge most sharply, and it matters enormously for your architecture decisions.

If your feature needs to work for the majority of your users today, Core ML is the safer choice. If you’re building a feature that’s an enhancement for users on newer devices, Foundation Models is a compelling option.

What Each Framework Is Good At

Think of the two frameworks as operating in completely different problem spaces.

Core ML: Structured ML Tasks

Core ML excels when your task is well‑defined, has a clear input/output structure, and can be solved with a trained model:

• Image classification – “What object is in this photo?”
• Object detection – “Where are the faces/products/items in this frame?”
• Pose estimation – “Where are this person’s joints?”
• Audio classification – “Is this the sound of a dog or a car?”
• Text classification / sentiment analysis – “Is this review positive or negative?”
• Tabular prediction – “Based on these health metrics, what category does this fall into?”
• Real‑time video analysis – Frame‑by‑frame inference at high frequency

A key characteristic of Core ML tasks: you can define the exact output schema up front, and the model reliably produces a label, a bounding box, a confidence score, or a numeric prediction. Deterministic, fast, and predictable.

Foundation Models: Language and Reasoning Tasks

Foundation Models is designed for tasks that involve language understanding and generation:

• Text summarisation – turning long content into concise summaries
• Structured extraction – pulling structured data from unstructured text (e.g., extracting a name, date, and location from a messy user note)
• Content classification using natural language – nuanced categorisation with explanation
• Contextual suggestions – generating personalised recommendations based on user context
• Tool‑augmented reasoning – letting the model call into your app’s data to answer user questions

Apple explicitly warns that the on‑device 3 B model is not designed for world‑knowledge Q&A, code generation, or complex maths. It’s optimised for task‑oriented, app‑integrated intelligence.

When to Use Core ML

Use Core ML when:

• You need vision or audio inference. Core ML is the only on‑device option for camera‑based features — real‑time object detection, face analysis, pose estimation, scene classification. Foundation Models cannot process images (no image input support as of iOS 26.3).
• You need it to work on older devices. If your feature must work on an iPhone 12, 13, or 14, Core ML is your only on‑device option. It runs on any device with iOS 11+.
• You have a specific, narrow ML task. A model trained to classify 10 types of skin lesions, or to detect a specific product in a frame, will outperform a general 3 B LLM on that narrow task — and at a fraction of the memory and compute cost.
• You need deterministic, repeatable outputs. Core ML models return the same output for the same input, every time. Foundation Models, being a generative LLM, produces varied responses.
• You’re doing real‑time inference. Core ML can process frames from a camera feed at 30+ fps. Foundation Models is not suited for frame‑by‑frame tasks.

When to Use Foundation Models

Use Foundation Models when:

• Your task is fundamentally a language task. Summarising a document, extracting key facts from a user’s note, generating a personalised caption — these are natural fits.
• You want structured output from unstructured text. The @Generable macro lets you define the exact JSON schema you expect, and the model will attempt to produce output that conforms to it.
• You need on‑device privacy‑first LLM capabilities. Because the model lives on the device, no user data leaves the phone, satisfying strict privacy requirements.
• You’re targeting the newest hardware. If your app’s primary audience has iPhone 15 Pro/Max or later, you can leverage the full power of Apple Intelligence without worrying about fallback paths.
• You want to augment existing Core ML pipelines. For example, you might run a Core ML vision model to detect objects, then feed the detected labels into a Foundation Models prompt to generate a natural‑language description of the scene.

Combining Both Frameworks

In many real‑world apps, the best solution is a hybrid:

1. Run a Core ML vision model to extract concrete entities (e.g., “cat”, “red ball”).
2. Pass those entities to a Foundation Models prompt to generate a user‑friendly narrative (“A cat is playing with a red ball on the carpet”).

This pattern lets you keep the heavy‑lifting image work on Core ML (fast, deterministic) while still delivering the rich, language‑centric experience that users love.

Quick Decision Checklist

Final Thoughts

Core ML and Foundation Models are not competitors; they’re complementary tools in Apple’s on‑device AI toolbox.

• Core ML = your model, any modality, works everywhere.
• Foundation Models = Apple’s pre‑built LLM, language‑first, works on the newest hardware.

Pick the one (or both) that aligns with your data modality, device‑coverage requirements, and the nature of the problem you’re solving. When you do, you’ll deliver fast, private, and intelligent experiences that feel native to iOS.

# Decision Guide for Using Apple Intelligence

## When to Choose **Core ML** vs. **Foundation Models**

| Scenario | Recommended Framework |
|----------|-----------------------|
| Vision / audio inference | **Core ML** |
| Need to support iPhone 12‑14 or older | **Core ML** |
| Deterministic, low‑latency output | **Core ML** |
| Real‑time frame‑by‑frame processing | **Core ML** |
| Pure language tasks (summarisation, extraction, generation) | **Foundation Models** |
| Structured JSON output from free‑form text | **Foundation Models** |
| Tool‑calling / app‑integrated reasoning | **Foundation Models** |
| Targeting only the latest Apple‑Intelligence‑enabled devices | **Foundation Models** |

Both frameworks can be combined: use **Core ML** for heavy‑weight vision/audio preprocessing, then feed the results into **Foundation Models** for natural‑language reasoning or summarisation. The key is to match the problem space to the right tool.

---

### TL;DR Decision Matrix  

- **You need tool‑augmented reasoning.**  
  The model can call your app’s APIs (via the tool‑calling interface) to fetch up‑to‑date data, perform calculations, or query a database, then incorporate the result into its response.  

- **You’re targeting the newest devices and can rely on Apple Intelligence being enabled.**  
  If your user base is primarily iPhone 15 Pro/Max or later, you can safely assume the model is present.  

- **You accept non‑deterministic, generative results.**  
  For creative text generation, nuanced classification, or contextual suggestions, variability can be a feature, not a bug.  

---

### Summary  

- **Core ML** → Best for on‑device, deterministic, low‑latency tasks (vision, audio, real‑time processing).  
- **Foundation Models** → Ideal for pure language tasks, structured output, tool‑calling, and when you can depend on the latest Apple‑Intelligence‑enabled hardware.  

Combine them as needed to leverage the strengths of each framework.

## Extracting Type‑Safe Swift Structs Directly from Free‑Form Input  
*No JSON parsing, no regex, no post‑processing.*

### Why Use Natural‑Language Reasoning with Tool Calling?  
Foundation models can decide **when** to call into your app’s data, fetch it, and incorporate it into a response. Core ML models can’t reason about **when** or **whether** to request more context.

### Zero Model Maintenance  

- **Core ML** – you own the model, retrain it, update it, and handle drift over time.  
- **Foundation Models** – Apple maintains the base model; you receive improvements automatically with OS updates.

### Speed of Integration  

Building a Core ML‑powered feature involves:

1. Selecting an architecture  
2. Gathering data  
3. Training the model  
4. Converting it for on‑device use  
5. Integrating it into your app  

With foundation models, you can achieve the same result **in an afternoon**.

## The Combination Pattern  

The two frameworks aren’t competitors; they’re complementary layers. The most powerful apps combine them.

**Example:** *SwingVision* (a tennis/pickleball coaching app) uses Core ML to analyze video frames and extract structured data about a user’s movement and technique. That structured output is then fed as context into a Foundation Model session to generate natural‑language coaching feedback.

### How It Works  

| Layer                | Responsibility                                                            |
|----------------------|-----------------------------------------------------------------------------|
| **Core ML**          | Perception – processes images, audio, or sensor data into structured signals |
| **Foundation Models**| Reasoning & language – turns those signals into meaningful, natural‑language insights |

### Use Cases  

- **Fitness app:** Core ML for pose estimation → Foundation Models generate a personalized workout summary.  
- **Cooking app:** Core ML identifies ingredients from a photo → Foundation Models suggest a recipe.

```swift
// Step 1: Core ML classifies the image
let classificationRequest = VNCoreMLRequest(model: mlModel) { request, _ in
    guard let result = request.results?.first as? VNClassificationObservation else { return }

    // Step 2: Feed the structured output into Foundation Models
    Task {
        let session = LanguageModelSession()
        let response = try await session.respond(
            to: "The user photographed: \(result.identifier) with \(Int(result.confidence * 100))% confidence. Suggest what they could make with this."
        )
        print(response.content)
    }
}

Decision Framework

Summary

• Core ML – A mature, flexible inference engine for deterministic machine‑learning tasks that works on all supported Apple devices.
• Foundation Models – A purpose‑built API for language‑model tasks on Apple‑Intelligence‑compatible devices, requiring virtually no setup.

Requirements