Get live statistics and analysis of Jim Fan's profile on X / Twitter

Grok just passed my sanity check

If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all

The famed Stanford Smallville is officially open-source! 25 AI agents inhabit a digital Westworld, unaware that they are living in a simulation. They go to work, gossip, organize socials, make new friends, and even fall in love. Each has unique personality and backstory.…

What if we set GPT-4 free in Minecraft? ⛏️ I’m excited to announce Voyager, the first lifelong learning agent that plays Minecraft purely in-context. Voyager continuously improves itself by writing, refining, committing, and retrieving *code* from a skill library. GPT-4 unlocks…

We are living in a timeline where a non-US company is keeping the original mission of OpenAI alive - truly open, frontier research that empowers all. It makes no sense. The most entertaining outcome is the most likely. DeepSeek-R1 not only open-sources a barrage of models but also spills all the training secrets. They are perhaps the first OSS project that shows major, sustained growth of an RL flywheel. Impact can be done by "ASI achieved internally" or mythical names like "Project Strawberry". Impact can also be done by simply dumping the raw algorithms and matplotlib learning curves. I'm reading the paper: > Purely driven by RL, no SFT at all ("cold start"). Reminiscent of AlphaZero - master Go, Shogi, and Chess from scratch, without imitating human grandmaster moves first. This is the most significant takeaway from the paper. > Use groundtruth rewards computed by hardcoded rules. Avoid any learned reward models that RL can easily hack against. > Thinking time of the model steadily increases as training proceeds - this is not pre-programmed, but an emergent property! > Emergence of self-reflection and exploration behaviors. > GRPO instead of PPO: it removes the critic net from PPO and uses the average reward of multiple samples instead. Simple method to reduce memory use. Note that GRPO was also invented by DeepSeek in Feb 2024 ... what a cracked team.

Minecraft has been achieved internally Yes this is Sora's hallucination of Minecraft. It can't resist the urge to make the sky look less pixelated 😅

AI Twitter is flooded with low-quality stuff recently. No, GPT is not “dethroned”. And thin wrapper apps are not “insane”. At all. I feel obligated to surface some quality posts I bookmarked. Every one of them should've been promoted 10x, but ¯\_(ツ)_/¯ In no particular order:

I was very late to own a Tesla but among the earliest to try out FSD v14. It's perhaps the first time I experience an AI that passes the Physical Turing Test: after a long day at work, you press a button, lay back, and couldn't tell if a neural net or a human drove you home. Despite knowing exactly how robot learning works, I still find it magical watching the steering wheel turn by itself. First it feels surreal, next it becomes routine. Then, like the smartphone, taking it away actively hurts. This is how humanity gets rewired and glued to god-like technologies.

10x engineer is a myth. 100x AI-powered engineer is more real than ever. As OpenAI winds down Codex, Microsoft announces GitHub Copilot X. I think it's almost as exciting as GPT-4 itself: - Copilot Chat: any piece of text database will be "chattable", and codebase is no different. Don't read your code, talk to it. - Copilot for Pull Request: improves *human collaboration*. Now GPT will accelerate not just a single dev, but entire OSS communities. - Copilot CLI: bash is so unintuitive and awkward sometimes. No more bash, just English. - Copilot doc: thanks to GPT-4's much longer context (32K tokens), you can fit entire docs in one go. No need to memorize any doc - simply retrieve from the prompt. It's a bit annoying that there is a separate waitlist for each item ... I'll link them in 🧵:

We’ve seen a gazillion startups using OpenAI APIs to do “co-pilot for X”. What’s next? Enter *physical* co-pilot! Here’s a compelling demo: you improvise by playing a “low resolution” piano, and the co-pilot compiles it real-time to Hi-Fi music! It unleashes our inner pianist.🧵

This is a master 4D chess move. WOW. 1. No new corporate structure. MSFT is literally one of the oldest for-profit tech companies out there, with a mature legal structure. Whether it's good for AGI is up for debate. 2. MSFT always wants to own the GPT weights. Now the moment has…

Y'all expecting RTX 5090, cool specs and stuff. But do you fully internalize what Jensen said about graphics? That the new card uses neural nets to generate 90+% of the pixels for your games? Traditional ray-tracing algorithms only render ~10%, kind of a "rough sketch", and then a generative model fills in the rest of fine details. In one forward pass. In real time. AI is the new graphics, ladies and gentlemen.

I don't give a damn about what is or isn't AGI. It doesn't matter. Below is GPT-4's performance on many standardized exams: BAR, LSAT, GRE, AP, etc. The truth is, GPT-4 can apply to Stanford as a student now. AI's reasoning ability is OFF THE CHARTS. Exponential growth is the…

OpenAI Strawberry (o1) is out! We are finally seeing the paradigm of inference-time scaling popularized and deployed in production. As Sutton said in the Bitter Lesson, there're only 2 techniques that scale indefinitely with compute: learning & search. It's time to shift focus to the latter. 1. You don't need a huge model to perform reasoning. Lots of parameters are dedicated to memorizing facts, in order to perform well in benchmarks like trivia QA. It is possible to factor out reasoning from knowledge, i.e. a small "reasoning core" that knows how to call tools like browser and code verifier. Pre-training compute may be decreased. 2. A huge amount of compute is shifted to serving inference instead of pre/post-training. LLMs are text-based simulators. By rolling out many possible strategies and scenarios in the simulator, the model will eventually converge to good solutions. The process is a well-studied problem like AlphaGo's monte carlo tree search (MCTS). 3. OpenAI must have figured out the inference scaling law a long time ago, which academia is just recently discovering. Two papers came out on Arxiv a week apart last month: - Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. Brown et al. finds that DeepSeek-Coder increases from 15.9% with one sample to 56% with 250 samples on SWE-Bench, beating Sonnet-3.5. - Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Snell et al. finds that PaLM 2-S beats a 14x larger model on MATH with test-time search. 4. Productionizing o1 is much harder than nailing the academic benchmarks. For reasoning problems in the wild, how to decide when to stop searching? What's the reward function? Success criterion? When to call tools like code interpreter in the loop? How to factor in the compute cost of those CPU processes? Their research post didn't share much. 5. Strawberry easily becomes a data flywheel. If the answer is correct, the entire search trace becomes a mini dataset of training examples, which contain both positive and negative rewards. This in turn improves the reasoning core for future versions of GPT, similar to how AlphaGo’s value network — used to evaluate quality of each board position — improves as MCTS generates more and more refined training data.

Grok just passed my sanity check

Allegedly shot in Shenzhen. Is this real? Can someone verify? I've seen this company posting very natural humanoid walking gaits a couple months ago. These days, it's hard to tell CGI vs Sora vs real ...

If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all

Font design is about to get a huge upgrade. Traditionally, given the same font, the letter ‘A’ looks identical everywhere. How about a smart & magical font that draws ‘A’ differently based on the semantic meaning of the word? Smells like Stable Diffusion? Deep dive with me: 🧵

Million dollar idea: LLM keyboard. Every time I type on my phone and autocorrect makes a stupid mistake, it screams LLM. This is *literally* next word prediction. We should be typing 10x faster. Input methods need serious upgrades. The LLM doesn’t have to be big and can be…

Minecraft has been achieved internally Yes this is Sora's hallucination of Minecraft. It can't resist the urge to make the sky look less pixelated 😅

I was very late to own a Tesla but among the earliest to try out FSD v14. It's perhaps the first time I experience an AI that passes the Physical Turing Test: after a long day at work, you press a button, lay back, and couldn't tell if a neural net or a human drove you home. Despite knowing exactly how robot learning works, I still find it magical watching the steering wheel turn by itself. First it feels surreal, next it becomes routine. Then, like the smartphone, taking it away actively hurts. This is how humanity gets rewired and glued to god-like technologies.

I don't give a damn about what is or isn't AGI. It doesn't matter. Below is GPT-4's performance on many standardized exams: BAR, LSAT, GRE, AP, etc. The truth is, GPT-4 can apply to Stanford as a student now. AI's reasoning ability is OFF THE CHARTS. Exponential growth is the…

What if we set GPT-4 free in Minecraft? ⛏️ I’m excited to announce Voyager, the first lifelong learning agent that plays Minecraft purely in-context. Voyager continuously improves itself by writing, refining, committing, and retrieving *code* from a skill library. GPT-4 unlocks…

It gives me a lot of comfort knowing that we are the last generation without advanced robots everywhere. Our children will grow up as “robot natives”. They will have humanoids cook Michelin dinner, robot teddy bears tell bedtime stories, and FSD drive them to school. We are the generation of “robot immigrants”, en route to a new world of ubiquitous Physical AI, much like our parents are “digital immigrants”, learning to realign their lives on 6 inches of touch screen. It’s a journey of both inventing sci-fi tech and reinventing ourselves. Everything that moves will be autonomous. Every year from now on will be the Year of Robotics. Here’s to a wild 2025 ahead 🥂

Y'all expecting RTX 5090, cool specs and stuff. But do you fully internalize what Jensen said about graphics? That the new card uses neural nets to generate 90+% of the pixels for your games? Traditional ray-tracing algorithms only render ~10%, kind of a "rough sketch", and then a generative model fills in the rest of fine details. In one forward pass. In real time. AI is the new graphics, ladies and gentlemen.

This is likely the most significant lawsuit in AI history - its outcome would have far-reaching impact on the whole industry. The arguments get fairly philosophical. Quote: "The purpose of copyright law, OpenAI argued, is 'to promote the Progress of Science and useful Arts' by…

The famed Stanford Smallville is officially open-source! 25 AI agents inhabit a digital Westworld, unaware that they are living in a simulation. They go to work, gossip, organize socials, make new friends, and even fall in love. Each has unique personality and backstory.…

Somehow in this epic meltdown, Satya swoops in, wins it all, and wins with grace. I'm floored. OpenAI was invincible until Friday. Now Microsoft will fully own an in-house GPT-4 in ~9 months, leverage its massive distribution power to spin the biggest data flywheel ever, collect…

I see some vocal objections: "Sora is not learning physics, it's just manipulating pixels in 2D". I respectfully disagree with this reductionist view. It's similar to saying "GPT-4 doesn't learn coding, it's just sampling strings". Well, what transformers do is just manipulating