AI Engineering from Scratch

AI Engineering from Scratch — From linear algebra to autonomous agent swarms. learn AI with AI, then ship the tools.

Introduction In a world where AI tools are evolving faster than ever, a learning path that blends deep understanding with practical, shippable artifacts becomes invaluable. This guide presents a comprehensive program designed to take you from foundational math to production-ready AI systems, culminating in autonomous agents and swarms. It’s not just about consuming content; it’s about building, testing, and shipping tools that others can install and use. Each lesson delivers a reusable artifact—prompts, skills, agents, MCP servers—so you don’t finish with just notes. You finish with a toolkit you can deploy in real projects.

Why this course stands apart

• A holistic scope: math, ML, DL, NLP, vision, speech, transformers, LLMs, agents, and swarms — all in one journey.
• A multilingual toolkit: Python, TypeScript, Rust, and Julia accompany the curriculum so you can implement in the language that fits the task.
• Output you can use: beyond theory, you leave with prompts, skills, agents, and MCP servers ready to install.
• A hands-on, AI-native approach: build first, then use frameworks; code and deploy as you go.
• A structured path to real-world impact: 283+ lessons, 20 phases, roughly 320 hours, designed to produce practical, runnable tools.

The Journey: 20 Phases of AI Mastery The program is divided into 20 phases, each with a clear focus and a mix of “build” and “learn” milestones. The journey begins with fundamentals and climbs toward production-grade, autonomous systems and ethical alignment. A quick map of the phases helps orient the path you’ll tread:

• Phase 0 — Setup & Tooling: Prepare your environment for everything that follows. This phase gets your tools, shells, and configurations in order so later lessons can proceed smoothly.
• Phase 1 — Math Foundations: The intuition behind every AI algorithm, translated into code. Foundational topics range from linear algebra to optimization and probabilistic thinking.
• Phase 2 — ML Fundamentals: Classical machine learning approaches that remain essential in production AI today.
• Phase 3 — Deep Learning Core: From perception to backpropagation, you’ll build neural nets from scratch and then connect them to modern frameworks.
• Phase 4 — Computer Vision: Image and video understanding, from pixel processing to vision transformers and open-world perception.
• Phase 5 — NLP Foundations to Advanced: Language as an interface to intelligence, from tokenization to large-scale language modeling ideas.
• Phase 6 — Speech & Audio: Hearing and speaking AI, from spectrograms to TTS and voice agents.
• Phase 7 — Transformers Deep Dive: The architecture that reshaped AI, including attention, encoders/decoders, and scaling considerations.
• Phase 8 — Generative AI: Generative models across modalities—images, video, audio, 3D, and beyond.
• Phase 9 — Reinforcement Learning: Foundations to advanced policy design, RLHF, and multi-agent aspects.
• Phase 10 — LLMs from Scratch: Tokenization, data pipelines, pre-training, distributed training, and practical scaling tricks.
• Phase 11 — LLM Engineering: Production-ready LLM systems, tool use, RAG, and safe deployment patterns.
• Phase 12 — Multimodal AI: Vision-language and beyond—patch tokens, cross-modal integration, and video/multi-sensor streams.
• Phase 13 — Tools & Protocols: Interfaces between AI and the real world, MCPs, tool use, and robust integration patterns.
• Phase 14 — Agent Engineering: The agent loop, memory, planning, and robust execution in real-time.
• Phase 15 — Autonomous Systems: Long-horizon agents, self-improvement ideas, and safety-first design principles.
• Phase 16 — Multi-Agent & Swarms: Coordination, emergence, and collective intelligence in distributed systems.
• Phase 17 — Infrastructure & Production: Production-ready serving, monitoring, scaling, and reliability patterns.
• Phase 18 — Ethics, Safety & Alignment: Building AI that helps humanity, with governance, safety, and fairness baked in.
• Phase 19 — Capstone Projects: End-to-end, ship-ready projects that showcase your integrated capability.

Visual cue for the journey

• Phase badges and milestones punctuate the path, signaling progress and the kind of work you’ll be doing in each phase. These badges serve as visual anchors for the learning journey and reflect the Build/Learn rhythm that characterizes the curriculum.

The Toolkit: Outputs that Ship A standout feature of this program is its commitment to deliverable tooling. Every lesson is designed to produce something you can actually use, not just a theoretical takeaway:

• Prompts: Ready-to-paste templates you can deploy in any AI assistant for expert-level help.
• Skills: Installable into Claude Code, Cursor, or any agent you use—ready to enhance your tooling.
• Agents: Autonomous workers you can deploy to perform tasks, demonstrations, and experiments.
• MCP Servers: MCP (Model Context Protocol) servers that plug into MCP-compatible AI apps, enabling modular context-aware deployments.

All of this aggregates into a folder structure you can explore and extend:

• outputs/
• prompts/
• skills/
• agents/
• mcp-servers/ And you can install them with SkillKit, turning your in-course artifacts into practical tools for your team or project.

How Each Lesson Works: A six-step rhythm Each lesson follows a consistent six-step pattern to maximize learning, intuition, and hands-on capability:

• Motto: One-line core idea that sticks.
• Problem: A concrete scenario where not knowing this hurts.
• Concept: Intuition, diagrams, and framing—no code yet.
• Build It: Implement from scratch in pure Python (no frameworks) to internalize the mechanics.
• Use It: Apply the concept with real tooling (PyTorch, sklearn, etc.) to see how it feels in practice.
• Ship It: The lesson output—a prompt, skill, agent, or MCP server you can deploy.

This build-then-use approach is the keystone: you understand the framework by building it yourself first, then you gain fluency by applying it in real-world tools.

Getting Started: How to begin There are multiple ways to start, depending on your current level and goals:

• Option A — Just start reading: Pick any completed lesson from the website or expand any phase above.
• Option B — Clone and run locally: A quick-start example is to clone the repository and run a sample phase locally, for instance:
• git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
• cd ai-engineering-from-scratch
• python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py
• Option C — Find your level (recommended): If you already know some ML/DL, use the built-in assessment to map your knowledge to a starting phase. In Claude Code, run:
• /find-your-level This guided assessment gives you a personalized starting phase with hour estimates for the remainder of the journey.

Prerequisites and audience

• Prerequisites: You should be comfortable writing code (in Python or any language) and be eager to understand how AI actually works, not just how to call APIs.
• Who this is for:
• New to programming + AI — start at Phase 0 (Setup) and plan for ~306 hours.
• Know Python, new to ML — start at Phase 1 (Math) with ~270 hours.
• ML practitioner aiming at DL — Phase 3 (Deep Learning) with ~200 hours.
• DL expert aiming for LLMs/agents — Phase 10 (LLMs from Scratch) with ~100 hours.
• Senior engineer focusing on agents — Phase 14 (Agent Engineering) with ~60 hours.

Why This Matters Now: Industry signal and foundational papers

• The industry signal is loud: English is the hottest programming language, and software engineering is being remade by AI tooling. The curriculum is designed to anchor this shift in solid fundamentals and projectable outcomes.
• Foundational papers covered across phases span the most influential works in AI:
• Attention Is All You Need (Phase 7)
• GPT-3 and few-shot learning (Phase 10)
• Denoising Diffusion Probabilistic Models (Phase 8)
• RLHF and InstructGPT lineage (Phase 10)
• Chain-of-Thought prompting (Phase 11)
• MCP: Model Context Protocol (Phase 13)
• And many more, all woven into a practical, build-first, ship-later learning path.

The Philosophy: Learn with AI, not just about AI

• This program emphasizes AI-native learning: you learn with AI copilots, not in isolation. The built-in Claude Code skills, steps to assess understanding, and a path to hands-on production-grade artifacts position you to flourish in teams that rely on AI-enabled tooling.
• You’ll toggle between learning and building, moving from conceptual intuition to tangible, shipped components.

Getting Your Hands Dirty: The practical rhythm

• Each lesson’s six-step process builds a finished artifact you can deploy.
• The outputs you produce are not “final exams”; they’re usable components you can install and run in your own projects or with your team.
• The toolkit you assemble is designed to be shared, extended, and repurposed, enabling collaborations across individuals and organizations.

Contributing and community

• The project welcomes all kinds of contributions: new lessons, fixes, translations, and outputs.
• Resources you may consult include CONTRIBUTING.md, ROADMAP.md, and other project documents to learn how to participate and drive improvements.
• A vibrant community of learners and practitioners helps sustain the evolving ecosystem around AI-native learning and ship-ready AI tooling.

Ethics, safety, and responsible scaling

• The curriculum dedicates phases to ethics, safety, and alignment (Phase 18) and to responsible scaling and production readiness (Phase 16 onward).
• Readers encounter important topics like reward hacking, robust guardrails, and governance patterns to ensure that AI systems are safe, fair, and aligned with human values.

A final note on the banner and imagery

• The banner image at the top—AI Engineering from Scratch—is a visual entry point into a detailed, rigorous, and practical exploration of AI engineering. Throughout the post, imagery and visuals from the input (badge visuals and phase markers) are echoed in the rhythm of the journey, the six-step lesson structure, and the artifact-centric outcomes.

In closing: what you walk away with

• A clear, end-to-end path from math foundations to production-ready autonomous systems.
• A large, practical toolkit of prompts, skills, agents, and MCP servers ready to install.
• The ability to design, implement, and ship AI tools with confidence, guided by hands-on practice and an AI-native learning approach.
• A community-driven curriculum that welcomes contributions, improvements, and collaborations.

If this resonates, take the next step: pick a phase, start a lesson, and build your first artifact. Whether you’re a newcomer eager to learn or an experienced engineer aiming to ship AI systems, this journey provides a structured, kinesthetic approach to mastering AI engineering from scratch and moving toward the frontier of autonomous agents and swarms.

Images in this post

• Banner: AI Engineering from Scratch (assets/banner.svg)
• Phase visuals: representative phase badge imagery (as seen in the input’s phase links) to echo the phase-by-phase progression

Key takeaways

• 283+ lessons across 20 phases
• 320 hours of guided, hands-on learning
• Multilingual tooling: Python, TypeScript, Rust, Julia
• Outputs you can install and use: prompts, skills, agents, MCP servers
• A six-step lesson framework that pairs build-without-frameworks and practical deployment
• A community-driven path with ethics, safety, and production-readiness at its core

Ready to begin? Start with Phase 0 to set up your tooling, or jump into Phase 1 to ground yourself in the math that powers every AI algorithm. The journey from linear algebra to autonomous agent swarms awaits.