Six modalities, one schema
Structured outputs turned extraction from a training problem into a schema-design problem. Six worked examples carry the same trick past text — into images, video, audio, sensor streams, and DNA.
- Structured outputs: the guarantee behind a clean {vendor, amount} extraction
- Vision, video, and audio models mapped straight onto typed schemas
- Sensor time series narrated into labeled segments, not raw numbers
- DNA and protein sequences annotated as the identical shape of problem
“Unstructured data” is doing something dishonest to the word “structure.” A video is a rigid grid of numbers, frame after frame, in an order nobody’s allowed to shuffle. A sensor stream is a rigid sequence of floats, one per tick, in exactly the order the clock produced them. None of that is disordered. What it lacks is a schema — column names, types, something a WHERE clause could use.
Call it what it is: data we hadn’t built a parser for yet.
For twenty years the fix was one parser per modality. OCR plus regex for scanned text. Named-entity recognition pipelines for prose. Bespoke feature extractors for audio, video, sensor readings — each one hand-built, each one only good for the modality it was built for. Extraction never generalized as a discipline, because the parser and the data were always the same kind of thing.
Frontier models change what’s on the other side of the pipe. Hand one a JSON schema, get back values that validate against it — and the input modality stops mattering, because the model reads all of them the same way: as tokens.
The shape underneath all six examples below: signal in, schema in, typed row out. Same three arguments to the same function, six times, aimed at a different kind of input each time. If you remember nothing else, remember the shape — everything below is just the shape wearing a different costume.
The trick, once
OpenAI shipped the current version of this — Structured Outputs, a JSON Schema the API actually enforces instead of just requesting — in August 2024, and every frontier lab now ships its own name for the same idea. What it removed was the part everyone used to write by hand: the parser that turned a wall of prose into a dict, plus the regex and the exception handling around it for when the wall of prose didn’t cooperate. You describe the shape you want. The API enforces it.
from pydantic import BaseModel
from openai import OpenAI
class Invoice(BaseModel):
vendor: str
amount: float
due_date: str
client = OpenAI()
result = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[{"role": "user", "content": email_text}],
response_format=Invoice,
)
invoice = result.choices[0].message.parsed
# Invoice(vendor='Acme Logistics', amount=4820.0, due_date='2026-08-14')
That’s the entire trick, on the most boring input there is: a paragraph. Everything below is the same trick, aimed at things that don’t have paragraphs.
Six inputs, one function call, one output shape. The rest of this piece just walks the six, in order of how far they are from a paragraph.
Images: a shelf photo as a database write
Extraction was never about vision. It was always about the shape you asked for.
OCR used to be its own pipeline stage — a model dedicated to reading characters, followed by regex to find the fields you actually wanted, followed by validation to catch what the regex missed. A vision-capable frontier model collapses that into one call: point it at a photo, hand it a schema, get a row back.
import base64
from pydantic import BaseModel
from openai import OpenAI
class ShelfScan(BaseModel):
sku: str
price: float
facing_count: int
planogram_compliant: bool
def to_data_url(path: str) -> str:
with open(path, "rb") as f:
encoded = base64.b64encode(f.read()).decode()
return f"data:image/jpeg;base64,{encoded}"
client = OpenAI()
result = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Read the shelf tag and stock count for the SKU in focus."},
{"type": "image_url", "image_url": {"url": to_data_url("shelf_aisle_7.jpg")}},
],
}],
response_format=ShelfScan,
)
scan = result.choices[0].message.parsed
# ShelfScan(sku='SKU-88213', price=4.29, facing_count=6, planogram_compliant=False)
The interesting failure mode isn’t blur or bad lighting. It’s the model confidently returning a plausible SKU that isn’t the one on the shelf. Structured extraction inherits the model’s hallucination problem wholesale. The schema doesn’t fix that. It just makes the wrong answer typed correctly.
Video: a timeline, not a caption
A timeline is a schema that happens to have a clock built into it.
Video understanding, in production, isn’t a one-line caption — “a man walks into a room.” It’s an array: a sequence of typed events at whatever frame rate you choose, each one timestamped, because a timestamp is what a database wants back.
from pydantic import BaseModel
from openai import OpenAI
class VideoEvent(BaseModel):
t_start: float
t_end: float
event: str
actors: list[str]
confidence: float
class VideoTimeline(BaseModel):
events: list[VideoEvent]
client = OpenAI()
video_file = client.files.create(file=open("match_highlights.mp4", "rb"), purpose="user_data")
result = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "List every goal and offside flag as timestamped events."},
{"type": "file", "file": {"file_id": video_file.id}},
],
}],
response_format=VideoTimeline,
)
timeline = result.choices[0].message.parsed
for event in timeline.events:
print(f"{event.t_start:>6.1f}s {event.event:<14} {event.actors}")
# 12.4s goal ['team_a_10']
# 41.0s offside_flag ['team_b_4']
The schema is the interesting design decision here, not the model call. Ask for “events” and you get a sparse, useful timeline. Ask for “everything happening at each second” and you get a firehose nothing downstream wants. Extraction quality is bottlenecked by how well you specify the shape, not by whether the model can “see.”
Audio, non-speech: sound as telemetry
No transcript, no words, still a typed row — extraction was never really about language.
Most people hear “audio model” and think transcription. That’s the boring half. The more useful case for structured extraction has no words in it at all — a factory floor, a running engine, a compressor cycling. An audio-native model classifies the acoustic event straight into a schema, without passing through a sentence on the way.
import base64
from datetime import datetime, timezone
from pydantic import BaseModel
from openai import OpenAI
class AcousticEvent(BaseModel):
machine_id: str
sound_signature: str
anomaly_score: float
def classify_clip(client: OpenAI, machine_id: str, audio_path: str) -> AcousticEvent:
with open(audio_path, "rb") as f:
audio_b64 = base64.b64encode(f.read()).decode()
result = client.beta.chat.completions.parse(
model="gpt-4o-audio-preview",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": f"Classify the acoustic signature for {machine_id}."},
{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
],
}],
response_format=AcousticEvent,
)
return result.choices[0].message.parsed
client = OpenAI()
event = classify_clip(client, "line-3-motor-b", "line3_motor_b_09-14.wav")
reading = {**event.model_dump(), "timestamp": datetime.now(timezone.utc).isoformat()}
# {'machine_id': 'line-3-motor-b', 'sound_signature': 'bearing_whine',
# 'anomaly_score': 0.81, 'timestamp': '2026-07-25T09:14:02+00:00'}
Notice what’s ours and what’s the model’s: machine_id, sound_signature, and anomaly_score come back from the classifier. timestamp doesn’t — the model has no idea what time it is, so the pipeline stamps that on afterward. Extraction only covers what’s actually in the signal.
This is the modality where “extraction” and “classification” visibly become the same operation. There’s no text to extract from. The model is mapping a waveform directly onto typed fields — which is a more honest description of what text extraction was always doing too.
Sensor time series: narrating a chart
A label is just a number that agreed to be readable.
A multivariate sensor stream doesn’t fit in a prompt as raw numbers, so you serialize it — turn the readings into a compact textual form — and ask for labeled segments back instead of raw values.
from pydantic import BaseModel
from openai import OpenAI
class Segment(BaseModel):
segment_start: str
segment_end: str
label: str
severity: str
class SegmentReport(BaseModel):
segments: list[Segment]
def narrate_readings(client: OpenAI, readings: list[tuple[str, float, float]]) -> SegmentReport:
serialized = "\n".join(f"{t}: temp={temp:.1f} vib={vib:.2f}" for t, temp, vib in readings)
result = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[{
"role": "user",
"content": f"Label anomalous segments in this sensor log:\n{serialized}",
}],
response_format=SegmentReport,
)
return result.choices[0].message.parsed
client = OpenAI()
report = narrate_readings(client, hourly_readings)
for seg in report.segments:
print(f"{seg.segment_start}-{seg.segment_end} {seg.label:<8} severity={seg.severity}")
# 08:41-08:47 spike severity=high
The model isn’t doing signal processing. It’s doing labeling on a serialized sequence. That’s not a replacement for a real anomaly detector trained on the actual signal — it’s a legitimate, cheap first pass. A way to get a labeled dataset before you’ve built the thing that’s supposed to replace it.
DNA and protein sequences: the same shape, a different field
Same shape, different lab coat — extraction doesn’t know what field it’s in.
“Sequence data” doesn’t mean “things that happen over time.” A DNA strand is a sequence with no time axis at all, and it’s exactly as amenable to the trick. Sequence-native models — and general frontier models prompted over annotated sequence text — return the same kind of typed row: a motif, a position, a confidence.
from pydantic import BaseModel
from openai import OpenAI
class Annotation(BaseModel):
motif: str
position: int
domain: str
confidence: float
class SequenceReport(BaseModel):
sequence_id: str
annotations: list[Annotation]
def annotate_sequence(client: OpenAI, sequence_id: str, bases: str) -> SequenceReport:
result = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[{
"role": "user",
"content": f"Sequence {sequence_id}: {bases}\nAnnotate known regulatory motifs.",
}],
response_format=SequenceReport,
)
return result.choices[0].message.parsed
client = OpenAI()
report = annotate_sequence(client, "seq-0091", "...GGGCTATATAAAGGGCA...")
for a in report.annotations:
print(f"{a.position:>5} {a.motif:<12} {a.domain} ({a.confidence:.0%})")
# 142 TATA_box promoter_region (88%)
Genomic extraction is the identical shape of problem wearing a lab coat. If this reads uncannily similar to the shelf-photo example, that’s not a coincidence. It’s the point of the whole article.
Why now, not five years ago
Two things had to happen, and only one of them was about model capability.
Structured outputs constrained generation so the API enforces the contract, instead of you parsing a hopeful string and catching exceptions when it lied. That’s plumbing, not intelligence, and it mattered more than the intelligence did.
Native multimodal ingestion removed the second half — the bespoke feature-extraction pipeline that used to sit between “raw signal” and “thing a model can read.” Once a model takes video or audio as input directly, there’s no separate system to build and maintain per modality.
Put together, extraction stopped being a training problem and became a prompt-and-schema-design problem. That’s a much cheaper problem to have, and a more boring one — which is usually the sign that a problem got actually solved rather than merely made impressive.
Where this breaks
Schema drift, when the input format changes and nobody updates the schema. Hallucinated fields on out-of-distribution input — a shelf photo of a product that doesn’t exist yet, a sound signature the model has never heard. And cost, which scales with tokens, and video or audio at any reasonable frame rate is a lot of tokens.
None of that is a reason not to use it. It’s a reason to treat the schema as the artifact you actually own and test — the same way you used to test a regex you were proud of.
Closing
Before the metaphor, the homework: pick one thing you’re still parsing by hand — a regex over PDFs, an intern squinting at screenshots, a scraper held together with try/except — and write the schema before you write anything else. Watch how much of the pipeline that one pydantic.BaseModel replaces.
“Unstructured” was never about disorder. It was about the absence of a contract. Foundation models didn’t teach the world to make sense. They just agreed, finally, to fill out the form.