Volume C · The TensorCore.si Library

C++

Speed and control: the engine room of AI.

C++ gives you direct control over memory and hardware while still offering modern abstractions. The fast cores of PyTorch, TensorFlow, llama.cpp, TensorRT and most game engines are written in it.

Prologue

Why learn C++?

When Python is too slow, the work drops down to C++. Learning it shows you how computers really work, memory, pointers, caches, and lets you write the code that frameworks and inference engines are built from.

Performance: compiled, zero-overhead abstractions and full control over memory layout.
AI infrastructure: PyTorch's core (LibTorch), ONNX Runtime, TensorRT and llama.cpp are C++.
Runs anywhere: from microcontrollers to supercomputers, and alongside CUDA on GPUs.
Modern features: C++20 and C++23 add modules, ranges, concepts and coroutines.

Prologue

Set up and run your first program

Install a compiler: g++ or clang++ on Linux and macOS (via Xcode Command Line Tools), or Visual Studio on Windows. CMake is the standard build tool for larger projects.

terminal

g++ --version
g++ -std=c++20 -O2 -Wall hello.cpp -o hello
./hello

hello.cpp

#include <iostream>
#include <string>

int main() {
    std::string name = "Ada";
    std::cout << "Hello, AI! Welcome to C++, " << name << ".\n";
    return 0;
}

Tip-O2 turns on optimisations and -Wall shows warnings. Always compile with warnings on; they catch many bugs early.

Chapter 1

Variables, types and constants

Every variable has a fixed type known at compile time. Use auto to let the compiler infer it. const means the value will not change, and constexpr means it is known at compile time.

Integer sizes matter in C++: int is usually 32 bits, long long 64 bits, and std::size_t is used for sizes and indexes.

variables.cpp

#include <iostream>
#include <string>

int main() {
    int layers = 32;
    double lr = 3e-4;
    bool use_gpu = true;
    char grade = 'A';
    std::string model = "llama3.1";
    auto params = 8'030'000'000LL;      // long long, digit separators
    const double pi = 3.14159;
    constexpr int kHeadDim = 128;

    std::cout << model << " has " << layers << " layers\n";
    std::cout << "bytes in FP16: " << params * 2 << "\n";
}

Chapter 2

Control flow

C++ uses braces for blocks. Besides if, switch, while and the classic for, the range-based for loops directly over containers.

control.cpp

#include <iostream>
#include <vector>

int main() {
    std::vector<int> vram = {24, 32, 80, 192};
    for (int gb : vram) {                 // range-based for
        if (gb >= 80)      std::cout << gb << " GB: data center\n";
        else if (gb >= 24) std::cout << gb << " GB: high-end desktop\n";
    }
    for (int i = 0; i < 3; ++i) std::cout << "epoch " << i << "\n";

    int precision = 8;
    switch (precision) {
        case 16: std::cout << "FP16\n"; break;
        case 8:  std::cout << "FP8\n";  break;
        default: std::cout << "other\n";
    }
}

Chapter 3

Functions, references and templates

Pass large objects by const& (a read-only reference) to avoid copying them. Functions can be overloaded by parameter type, and templates let one function work with many types.

Lambdas are small unnamed functions, often passed to algorithms.

functions.cpp

#include <iostream>
#include <string>

double memory_gb(double params_b, double bytes = 2.0) {
    return params_b * bytes;
}

void greet(const std::string& name) {      // no copy
    std::cout << "Hi " << name << "\n";
}

template <typename T>
T square(T x) { return x * x; }

int main() {
    std::cout << memory_gb(70) << " GB\n";   // 140
    greet("Ada");
    std::cout << square(4) << " " << square(1.5) << "\n";
    auto add = [](int a, int b) { return a + b; };
    std::cout << add(2, 3) << "\n";
}

Chapter 4

Containers and algorithms

The standard library gives you std::vector (a growable array, your default choice), std::array, std::unordered_map (hash table), std::map (sorted) and more. The <algorithm> header sorts, searches and transforms them.

containers.cpp

#include <algorithm>
#include <iostream>
#include <numeric>
#include <string>
#include <unordered_map>
#include <vector>

int main() {
    std::vector<double> losses = {0.9, 0.4, 0.7, 0.2};
    losses.push_back(0.1);
    std::sort(losses.begin(), losses.end());
    double sum = std::accumulate(losses.begin(), losses.end(), 0.0);
    std::cout << "min " << losses.front() << ", mean " << sum / losses.size() << "\n";

    std::unordered_map<std::string, int> vram{{"H100", 80}, {"B200", 180}};
    vram["MI300X"] = 192;
    for (const auto& [name, gb] : vram) std::cout << name << ": " << gb << "\n";
}

Chapter 5

Classes, RAII and smart pointers

Classes bundle data and functions. C++'s key idea is RAII: an object acquires a resource in its constructor and releases it in its destructor, so memory, files and GPU buffers are freed automatically.

Avoid raw new and delete. Use std::unique_ptr (one owner) or std::shared_ptr (shared ownership).

classes.cpp

#include <iostream>
#include <memory>
#include <string>

class Model {
public:
    Model(std::string name, double params_b) : name_(std::move(name)), params_b_(params_b) {
        std::cout << "load " << name_ << "\n";
    }
    virtual ~Model() { std::cout << "free " << name_ << "\n"; }   // runs automatically
    virtual std::string reply(const std::string& q) const { return name_ + ": " + q; }
protected:
    std::string name_;
    double params_b_;
};

int main() {
    auto m = std::make_unique<Model>("Llama 3.1 8B", 8);
    std::cout << m->reply("hello") << "\n";
}   // m goes out of scope: memory freed, no leak

Watch outA class with virtual functions should have a virtual destructor, so deleting a derived object through a base-class pointer cleans up correctly.

Chapter 6

Pointers and memory

A pointer stores the address of a value. The stack holds local variables and is freed automatically; the heap holds memory you allocate. Understanding this is essential for performance and for GPU programming, where you copy data between host and device memory.

memory.cpp

#include <iostream>
#include <vector>

int main() {
    int x = 42;
    int* p = &x;            // address of x
    *p = 7;                 // change x through the pointer
    std::cout << x << "\n"; // 7

    std::vector<float> weights(1'000'000, 0.5f);   // heap memory, freed automatically
    float* raw = weights.data();                     // pointer to the first element
    std::cout << raw[10] << " " << sizeof(float) * weights.size() / 1e6 << " MB\n";
}

Chapter 7

Errors: exceptions and optional values

Throw exceptions for unexpected failures and catch them with try/catch. For values that may legitimately be missing, return std::optional. C++23 adds std::expected for returning either a value or an error.

errors.cpp

#include <iostream>
#include <optional>
#include <stdexcept>

double load(double model_gb, double vram_gb) {
    if (model_gb > vram_gb) throw std::runtime_error("model does not fit");
    return vram_gb - model_gb;
}

std::optional<int> find_layer(int id) {
    if (id < 32) return id;
    return std::nullopt;
}

int main() {
    try {
        load(140, 24);
    } catch (const std::exception& e) {
        std::cout << "error: " << e.what() << "\n";
    }
    if (auto l = find_layer(40)) std::cout << *l << "\n";
    else std::cout << "no such layer\n";
}

Chapter 8

Headers, CMake and libraries

Declarations go in header files (.hpp), definitions in source files (.cpp). CMake describes how to build a project and find libraries, and package managers such as vcpkg and Conan download dependencies.

CMakeLists.txt

cmake_minimum_required(VERSION 3.20)
project(ai_demo LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 20)

add_executable(ai_demo main.cpp model.cpp)

# then build with:
#   cmake -B build -DCMAKE_BUILD_TYPE=Release
#   cmake --build build -j

Chapter 9

Files and streams

File streams work like std::cout: std::ofstream writes and std::ifstream reads. std::filesystem handles paths and directories.

files.cpp

#include <filesystem>
#include <fstream>
#include <iostream>
#include <string>

int main() {
    std::ofstream out("log.txt");
    out << "epoch 1 loss 0.42\n";
    out.close();

    std::ifstream in("log.txt");
    std::string line;
    while (std::getline(in, line)) std::cout << line << "\n";

    std::cout << std::filesystem::file_size("log.txt") << " bytes\n";
}

Chapter 10

Threads and parallelism

std::thread runs code in parallel, std::mutex protects shared data, and std::async returns a future you can wait on. For data-parallel loops, OpenMP, C++17 parallel algorithms or CUDA scale much further.

threads.cpp

#include <future>
#include <iostream>
#include <numeric>
#include <vector>

long long sum_range(const std::vector<int>& v, size_t a, size_t b) {
    return std::accumulate(v.begin() + a, v.begin() + b, 0LL);
}

int main() {
    std::vector<int> v(10'000'000, 1);
    size_t half = v.size() / 2;
    auto left = std::async(std::launch::async, sum_range, std::cref(v), 0, half);
    long long right = sum_range(v, half, v.size());
    std::cout << left.get() + right << "\n";   // 10000000
}

TipBuild with -pthread on Linux.

Chapter 11

Testing and tooling

GoogleTest and Catch2 are the popular test frameworks. Sanitizers (-fsanitize=address,undefined) catch memory bugs at run time, clang-format keeps code tidy, and clang-tidy finds risky patterns.

test_memory.cpp

#include <gtest/gtest.h>

double memory_gb(double params_b, double bytes) { return params_b * bytes; }

TEST(Memory, Fp16) {
    EXPECT_DOUBLE_EQ(memory_gb(8, 2), 16.0);
}

Part II · AI with C++

AI with C++

Most AI is designed in Python but runs in C++. Every PyTorch operation dispatches to C++ (and CUDA) kernels, and production inference engines are C++ for speed, low memory and easy deployment without a Python runtime.

You can train and run models directly in C++ with LibTorch, the C++ distribution of PyTorch, using an API that mirrors the Python one.

The essential libraries

LibTorchPyTorch's C++ API for training and inference.

llama.cpp / ggmlRun LLMs on CPUs and GPUs with quantized weights.

ONNX RuntimeFast cross-platform inference for exported models.

TensorRT / TensorRT-LLMNVIDIA's optimiser for GPU inference.

OpenCVComputer vision: images, video, detection.

EigenLinear algebra templates used inside many frameworks.

CUTLASSTemplates for writing fast Tensor Core matrix kernels.

OpenVINOIntel's toolkit for CPU, GPU and NPU inference.

Train a model with LibTorch

The same linear-regression model as in the Python volume, written with PyTorch's C++ API.

train.cpp

#include <torch/torch.h>
#include <iostream>

int main() {
    auto X = torch::linspace(-1, 1, 100).unsqueeze(1);
    auto y = 3 * X + 0.5;

    torch::nn::Linear model(1, 1);
    torch::optim::SGD opt(model->parameters(), /*lr=*/0.1);

    for (int step = 0; step < 200; ++step) {
        opt.zero_grad();
        auto loss = torch::mse_loss(model(X), y);
        loss.backward();
        opt.step();
    }
    std::cout << model->weight.item<float>() << " " << model->bias.item<float>() << "\n";
}

Download LibTorch from pytorch.org and build with CMake using find_package(Torch REQUIRED). To run large language models from C++, study llama.cpp, which powers Ollama and LM Studio.

Part III · Build an AI agent

Build an AI agent in C++

C++ agents are useful when you need a small, fast binary, for example on a robot, in a game or on an edge device. This version uses libcurl for HTTP and the header-only nlohmann/json library.

Your goal→Model plans→Tool call→Your code runs the tool→Result back to model→Final answer

agent.cpp

#include <curl/curl.h>
#include <nlohmann/json.hpp>
#include <iostream>
#include <string>
using json = nlohmann::json;

static size_t write_cb(char* p, size_t s, size_t n, void* out) {
    static_cast<std::string*>(out)->append(p, s * n);
    return s * n;
}

json chat(const json& body) {
    CURL* c = curl_easy_init();
    std::string resp, data = body.dump();
    curl_slist* h = curl_slist_append(nullptr, "Content-Type: application/json");
    curl_easy_setopt(c, CURLOPT_URL, "http://localhost:11434/v1/chat/completions");
    curl_easy_setopt(c, CURLOPT_HTTPHEADER, h);
    curl_easy_setopt(c, CURLOPT_POSTFIELDS, data.c_str());
    curl_easy_setopt(c, CURLOPT_WRITEFUNCTION, write_cb);
    curl_easy_setopt(c, CURLOPT_WRITEDATA, &resp);
    curl_easy_perform(c);
    curl_slist_free_all(h);
    curl_easy_cleanup(c);
    return json::parse(resp);
}

std::string get_weather(const std::string& city) {
    return "It is 24°C and sunny in " + city + ".";
}

int main() {
    json tools = json::parse(R"([{"type":"function","function":{"name":"get_weather",
        "description":"Get the current weather for a city","parameters":{"type":"object",
        "properties":{"city":{"type":"string"}},"required":["city"]}}}])");
    json messages = json::array({{{"role", "user"}, {"content", "Should I take an umbrella in Lahore today?"}}});

    while (true) {
        json msg = chat({{"model", "llama3.1"}, {"messages", messages}, {"tools", tools}})["choices"][0]["message"];
        messages.push_back(msg);
        if (!msg.contains("tool_calls") || msg["tool_calls"].empty()) {
            std::cout << msg["content"].get<std::string>() << "\n";
            break;
        }
        for (auto& call : msg["tool_calls"]) {
            json args = json::parse(call["function"]["arguments"].get<std::string>());
            messages.push_back({{"role", "tool"}, {"tool_call_id", call["id"]},
                                {"content", get_weather(args["city"].get<std::string>())}});
        }
    }
}

Build with g++ -std=c++17 agent.cpp -lcurl -o agent. In production, add error handling for failed requests and a maximum number of loop iterations.

Run it locallyInstall Ollama, run ollama pull llama3.1, and keep it running. The code talks to its OpenAI-compatible API at http://localhost:11434/v1, so the same program also works with vLLM, llama.cpp server or a cloud API by changing the URL, model name and adding an API key.

Part IV · Your path

Projects to build

Matrix libraryA small Matrix class with multiply, transpose and tests.

Text adventureA game with rooms, inventory and save files.

Image filterLoad images and apply blur and edge detection, then speed it up with threads.

ONNX inference appRun an exported image model from C++ with ONNX Runtime.

Mini neural networkImplement forward and backward passes for a tiny MLP from scratch.

Local LLM CLIA command-line chat app built on llama.cpp.

Reference

C++ cheat sheet

Task	Syntax
Print	`std::cout << x << "\n";`
Vector	`std::vector<int> v{1, 2}; v.push_back(3);`
Loop over container	`for (const auto& x : v) {}`
Function by reference	`void f(const std::string& s);`
Lambda	`auto f = [](int x) { return x * 2; };`
Smart pointer	`auto p = std::make_unique<T>(args);`
Map	`std::unordered_map<std::string, int> m;`
Sort	`std::sort(v.begin(), v.end());`
Template	`template <typename T> T f(T x);`
Exception	`try { } catch (const std::exception& e) { }`
Compile	`g++ -std=c++20 -O2 -Wall main.cpp -o app`

Reference

Common mistakes

Raw new/delete: easy to leak or double-free. Use containers and smart pointers.
Out-of-bounds access: v[i] does not check bounds. Use .at(i) while learning, and sanitizers.
Copying big objects by accident: pass by const& and return by value (moves are cheap).
Undefined behaviour: signed overflow, dangling references and uninitialised variables can "work" until they do not. Turn on warnings and sanitizers.
Building in Debug and measuring speed: always benchmark Release builds with -O2 or -O3.

Reference

Where to go next

cppreference — the standard library reference
LearnCpp — free, thorough beginner course
C++ Core Guidelines
PyTorch C++ frontend

On TensorCore.si: What is CUDA?, How a GPU works, Roofline simulator.

Keyboard: [ previous language · ] next language · / search the library

C++

Why learn C++?

Set up and run your first program

Variables, types and constants

Control flow

Functions, references and templates

Containers and algorithms

Classes, RAII and smart pointers

Pointers and memory

Errors: exceptions and optional values

Headers, CMake and libraries

Files and streams

Threads and parallelism

Testing and tooling

AI with C++

The essential libraries

Train a model with LibTorch

Build an AI agent in C++

C++ learning roadmap

Beginner

Intermediate

Advanced

Projects to build

C++ cheat sheet

Common mistakes

Where to go next