Beyond ChatGPT

What else can AI become?

Yuntian Deng & Bing Yan

Vision 2026 · October 14 · Brantford

This morning…

My request to ChatGPT
Actual request: For my upcoming talk in brantfort find the exact location and time
From ChatGPT’s answer
“3:00–4:15 p.m.”

I wrote zero code for these slides

My requests to ChatGPT
Two actual requests in the conversation: ‘Ok now strategize how to add them to slides’ and ‘Ok now build’.
The result
The finished deck: Beyond ChatGPT, What else can AI become?, by Yuntian Deng and Bing Yan.
Yuntian Deng

Yuntian Deng

Assistant Professor
University of Waterloo

I study new ways to build AI.

yuntiandeng.com
yuntian@uwaterloo.ca

AI can generate a video

“A gorgeously rendered papercraft world of a coral reef, rife with colorful fish and sea creatures.”
Sora-generated paper coral reef with colorful fish

OpenAI · Sora

What can AI do
beyond ChatGPT?

What makes AI powerful?

Demonstrations

→

Training

→

A new skill

This desktop is generated by AI

NeuralOS generated desktop

How is it done?

Break it into one small prediction at a time.

Earlier screens
Earlier desktop frameAnother earlier desktop frame
Current screen
NeuralOS desktop with Home selected

Double-click Home

→NeuralOS→
Next screen
Generated file manager open

Then take the next action and repeat.

How does it learn?

Prediction
Real earlier-stage prediction: a distorted file-manager windowReal later-stage prediction: the file-manager window is clearer

After more training

Recorded answer
Recorded target: the real file-manager window

How far off is the prediction?

Training error

ErrorMore training →

Adjust. Repeat.

We can invent the demonstrations

The original desktop without DoomThe same desktop with a Doom icon pasted into the top right corner
Start with a desktopPaste in a Doom icon
A real VizDoom gameplay frame spliced into the training recording
Splice in gameplay

Doom was never installed.

The model learns to simulate Doom

A Doom scene generated by NeuralOS

Where does it fail?

1

Fine-grained interactions

Getting a click or
typed text exactly right.

2

Persistent memory

Remembering what changed
after a window closes.

A small interaction can go wrong

NeuralOS generates an unrelated, garbled page after the input wikipedia.orgRequested: wikipedia.org

Where it fails: persistent memory

A file-manager folder being created and revisited

Can AI do
math we find hard?

Could you give just the answer?

12345678 × 87654321

No calculator. No written steps.

AI struggled too.

Teach AI to work without written steps

12×34
48+360
408

First, AI learns
with all the steps.

Remove a step.
Train again.

Remove more steps.
Train again.

Until only the answer remains.

Now it can give just the answer

12345678 × 87654321

Actual model demo: the reference product and the model prediction both read 1082152022374638, with every predicted digit correct.

huggingface.co/spaces/yuntian-deng/gpt2-multiplication

Can AI turn English
into programs?

One AI model creates another

Code generation

English→Python code

Program-as-Weights

English→A small AI model

Input → Output

A real program: find the verbs

Program description
Actual specification: Given a sentence, return all the verbs in it.
Program input
Actual input: Maya opened the door and smiled.
→
Program output
Actual output: ["opened", "smiled"]

programasweights.com/hub/779da82d4923b7d93dd6

Easy to describe. Hard to write every rule.

I watch birds.

watch is a verb

My watch broke.

broke is the verb

Teach it with demonstrations

Find the verbs

“Maya opened the door.”

opened

Is it urgent?

“Call now. The room is flooding.”

urgent

Many tasks. Inputs. Correct outputs.

The model learns to build the program

Task description

“Find the verbs.”

→
AI builder

Creates a program

→
Try an example

Does it return
the right verbs?

Compare with the demonstration.Improve the builder. Repeat.

An AI helper built from small programs

A question follows a path through specialized helper programs

Each block
has one job.

yuntiandeng.com/#ask

Direct a character in English

The Gangnam character following an English direction
“Now on one foot.”

programasweights.com/gangnam

Can AI keep up with a game?

My GPT-6 Astra helicopter mission attempt

My GPT-6 Astra attempt

One bomb planted.
Then a crash.

Small AI skills + fast control code

Recorded autonomous walking, van entry and helicopter bomb pickup

Where do small AI programs fail?

Count the r’s in this exact string.

Small program returned4
Correct count5

A counting tool gets it right: text.count("r") 5

programasweights.com/hub/ff9f42e546987d7c7c39

The reality of the Turing test

Karpathy’s meme: an Ex Machina interview becomes a series of unusual AI tests, beginning with counting the r’s in strawberry.

Andrej Karpathy · December 1, 2024 ↗

What can AI do
in chemistry?

Bing Yan

Bing Yan

Computer Science PhD candidate, NYU
PhD in Chemistry, MIT

I bring AI and chemistry together.

bingyan.me
bing.yan@nyu.edu

How could AI help us make a medicine?

Ibuprofen

Illustrative ball-and-stick model of ibuprofen, with hydrogen atoms omitted

First, what is a chemical reaction?

BeforeAfter
Before: two H2 molecules and one O2 molecule. After: two water molecules, preserving four hydrogen and two oxygen atoms.

Can AI find what chemists already know?

“How has making
ibuprofen improved?”

Papers→AskChem→Original source

askchem.org

Six reaction steps became three

Earlier route
6

reaction steps

→
Improved industrial route
3

reaction steps

Source: US EPA

Is this conclusion justified?

“Half as many reaction steps
means half the production time.”

A Yes

B We need timing data

Can AI help explain how a reaction happens?

Possibility 1

A⟶C

Possibility 2

A→BTemporary molecule→C

When should we look for B?

At the start
AAAAAA
→
During the reaction
ABBACB
→
At the end
CCCCCC

Can AI work backward from a signal?

Training examples

Known reaction steps

→

Simulated signals

Using the model

Measurements
+ conditions

→

Possible hidden steps

huggingface.co/spaces/bing-yan/spark

Can it infer the wrong reaction?

Known simulated reaction

Step 1+Step 2

SPARK’s top proposal

Step 1+Step 2+Extra step

Can a molecule change without a reaction?

Illustrative butane molecule in an extended conformation→Illustrative butane molecule in a bent conformation

A Different molecule

B Different shape

Same bonds. Different shape.

Two simplified four-carbon skeletons of butane retain the same three connections but have different conformations

Can AI predict a molecule’s shape?

Molecule
+ surroundings

Atoms and bonds supplied

→

EVA-Flow

→

Possible
3D shapes

Does the prediction match?

Dataset reference

AI prediction

In a liquid

EVA-Flow Table 4: dataset reference conformation in the solvation settingEVA-Flow Table 4: predicted conformation in the solvation setting

Near a
protein

EVA-Flow Table 4: dataset reference conformation in the docking settingEVA-Flow Table 4: predicted conformation in the docking setting

What still needs research?

Will it work on
chemistry it hasn’t seen?

Known examples→New chemistry→Experimental check

What do people
actually do with AI?

We collected
4.8M conversations

Where did the conversations come from?

Our public AI chat interface on Hugging Face, shown in Yuntian’s screenshot

Free AI access

Users agreed to share conversations for public research.

huggingface.co/spaces/yuntian-deng/ChatGPT

People joined from around the world

Historical WildChat map with activity across many parts of the world

Search the questions people ask

A search
WildVisualizer filters: Keyword homework and Country United States
One real request
Write a policy and outline best practices associated with homework for middle school learners

wildvisualizer.com

You can investigate this too

wildvisualizer.com

3.2M conversations in the open, non-toxic subset

huggingface.co/datasets/yuntian-deng/WildChat-4.8M

New demonstrations.
New possibilities.

What are you
curious about?

Yuntian Dengyuntiandeng.comyuntian@uwaterloo.ca

Bing Yanbingyan.mebing.yan@nyu.edu

yuntiandeng.com/talks/beyond-chatgpt/

An early model confused hover and click

Cursor near the Close button without clicking
Hover only
Historical model incorrectly closes the window
Window closes
← Questions

The missing example: hover without clicking

Window remaining open while the cursor hovers near Close
← Questions

A measurement gives us a clue

Cyclic voltammogram of SPARK’s bundled RuHex experimental case: measured current versus applied voltage, scan rate 0.10 volts per second
← Questions

Matching answers can still be wrong

ethanol

IUPAC name
=

CCO

SMILES

Same molecule. Different text.

← Questions