dlang-http/README.md

# dlang.http

An HTTP/1.1 server plugin for the **dlang** VM. It gives a dlang script a `http` global with
routing, static file serving, custom error pages, a small template engine (`view`) and
file uploads (`multipart/form-data`).

> [!IMPORTANT]
> **This project was generated with AI** (GitHub Copilot) and has not been reviewed by a
> human. Treat the code and this documentation as a starting point: read it, test it, and
> only use it in production once you understand and trust it. See
> [AI-generated code](#ai-generated-code) at the bottom.

> The dlang language and its VM live in their own repository. This repository only contains
> the HTTP plugin; the `dlang/` folder is an unmodified copy of the upstream VM headers and
> sources, kept here so the plugin can be built as a standalone shared library.

---

## Table of contents

- [Features](#features)
- [Requirements](#requirements)
- [Building](#building)
- [Installing the plugin](#installing-the-plugin)
- [Quick start](#quick-start)
- [API reference](#api-reference)
  - [`http.server()`](#httpserver)
  - [`server.listen(port)`](#serverlistenport)
  - [`server.route(path, methods, handler)`](#serverroutepath-methods-handler)
  - [Route handler arguments](#route-handler-arguments)
  - [`server.static(...)`](#serverstatic)
  - [`server.error(code, handler)`](#servererrorcode-handler)
  - [`server.maxBody(bytes)`](#servermaxbodybytes)
  - [File uploads](#file-uploads)
  - [`http.save` / `http.saveTo`](#httpsave--httpsaveto)
  - [Authentication (`auth.*`)](#authentication-auth)
  - [`view.Render(path, vars)`](#viewrenderpath-vars)
- [Status codes](#status-codes)
- [Request and response flow](#request-and-response-flow)
- [Limitations](#limitations)
- [Project layout](#project-layout)
- [AI-generated code](#ai-generated-code)

---

## Features

| Feature | Summary |
| --- | --- |
| Routing | Exact-path routes with a per-route list of HTTP methods |
| Static files | URL prefix → folder, exact URL → file, MIME type detected by extension |
| Request parsing | Query string, url-encoded bodies, JSON bodies and `multipart/form-data` |
| File uploads | Uploaded files as tables (`filename`, `contentType`, `size`, raw `content`) plus `http.save` / `http.saveTo` helpers to write them to disk |
| Cookies | Parsed `Cookie:` header as a table |
| Authentication | HTTP Basic auth for the paths you choose, with PBKDF2-HMAC-SHA256 hashed passwords in `users.json` |
| Error pages | Custom handlers or bodies per status code, with a built-in HTML fallback |
| Templates | `view.Render("page.html", { ... })` with recursive `@include` and `{{ key }}` interpolation |
| Body limit | Configurable request size limit (`server.maxBody`), oversized uploads get `413` |

---

## Requirements

- **Windows** or **Linux** (the plugin refuses to compile on other platforms).
- A **C++20** compiler: MSVC (Windows) or Clang/GCC (Linux).
- **CMake ≥ 3.10** and **Ninja** for the presets in `CMakePresets.json`.
- A **dlang host** (interpreter/runner) that supports plugins, since this repository builds
  a plugin, not a program.

---

## Building

The presets use Ninja with `cl.exe`, so run the commands from a *Developer Command Prompt*
(or build the folder from VS Code with the CMake Tools extension, which sets up the
compiler environment automatically).

```bash
# configure
cmake --preset x64-debug        # or x64-release, x86-debug, x86-release
                                # linux-debug on Linux, macos-debug on macOS

# build
cmake --build out/build/x64-debug
```

Available presets: `x64-debug`, `x64-release`, `x86-debug`, `x86-release` (Windows),
`linux-debug` (Linux) and `macos-debug` (macOS).

The build output is a shared library:

| Platform | Output |
| --- | --- |
| Windows | `out/build/<preset>/dlang.http/dlang.http.dll` |
| Linux | `out/build/<preset>/dlang.http/dlang.http.so` |

> The shared library compiles `entry.cpp` together with the VM sources in
> `dlang.http/dlang/src/`, so no external dlang SDK is needed. Windows additionally links
> `ws2_32`; Linux links `dl` and `pthread`.

---

## Installing the plugin

Plugins are looked up in a `plugins/` folder relative to the working directory of the dlang
host. The loader appends the platform extension itself (`.dll` on Windows, `.so` on Linux),
so the file must be named `dlang.http.dll` / `dlang.http.so`:

```text
my-app/
├─ plugins/
│  └─ dlang.http.dll      # copied from the build output
└─ app.dl
```

Load it from a script with `require` (the returned string is a status message such as
`"plugin loaded!"`):

```dlang
require("dlang.http");
```

Once loaded, the plugin registers three globals: `http`, `view` and `auth`.

---

## Quick start

`app.dl`:

```dlang
require("dlang.http");

// Basic auth: hash the password into ./users.json and gate /admin/**
auth.addUser("admin", "geheim");
auth.protectPrefix("/admin");

var server = http.server();

// Serve everything in ./public under /assets (e.g. /assets/css/site.css)
server.static("/assets", "./public");

// A plain text route
server.route("/", ["GET"], func(req, params, cookies, files)
    return "Hello world!";
end);

// Query-string parameters arrive in `params`
server.route("/hello", ["GET"], func(req, params, cookies, files)
    return "Hello " + params.name + "!";
end);

// Return a table as `body` and the plugin JSON-encodes it for you
server.route("/api/status", ["GET"], func(req, params, cookies, files)
    return {
        body: {
            ok: true,
            name: "dlang.http"
        }
    };
end);

// Uploads: `files` holds the multipart/form-data file fields
server.route("/upload", ["POST"], func(req, params, cookies, files)
    if (#files > 0) then
        var saved = http.saveTo(files.avatar, "./uploads");
        return { body: "Saved to " + saved, contentType: "text/plain" };
    end
    return { body: "No file uploaded", statusCode: 400 };
end);

// Custom error page
server.error(404, func(req)
    return {
        body: "<h1>Not found</h1><p>" + req.path + "</p>",
        contentType: "text/html"
    };
end);

// Blocks and serves requests until the process is stopped
server.listen(8080);
```

> `#files` is the size operator: it returns the number of entries in the table. Because a
> missing table field evaluates to `nil` (and logs a VM warning), checking `#files` first
> keeps handlers free of warnings.

---

## API reference

### `http.server()`

Creates a server table. Every other `server.*` function is called on that table, and the
table itself is not sent to the OS socket until `listen` is called.

### `server.listen(port)`

Binds `0.0.0.0:<port>`, accepts connections in a loop and serves them until the process
ends. Returns `false` when the socket cannot be created, bound or listened on; on success it
never returns.

### `server.route(path, methods, handler)`

Registers a route.

- `path` – the exact request path, e.g. `"/users"` (no wildcards or path parameters; read
  `req.path` if you need to match patterns yourself).
- `methods` – a list of method names, e.g. `["GET", "POST"]`. Comparison is
  case-sensitive, so use uppercase.
- `handler` – a function (see below).

When a path matches but none of its methods do, the server answers `405 Method Not
Allowed`.

### Route handler arguments

Handlers are called with four arguments:

```dlang
func(req, params, cookies, files)
```

| Argument | Contents |
| --- | --- |
| `req` | Table with `method`, `path`, `query` (only when the URL had one), `headers`, `params`, `cookies`, `files`, `body` (only when the request body was not empty) and `user` (only when the request passed an [`auth`](#authentication-auth) gate) |
| `params` | All request parameters merged into one table: query string + body |
| `cookies` | The parsed `Cookie:` header |
| `files` | Uploaded files from a `multipart/form-data` body, keyed by form field name |

`params` is filled from the body depending on the `Content-Type`:

| `Content-Type` | Body handling |
| --- | --- |
| `application/x-www-form-urlencoded` | `a=1&b=2` decoded into strings |
| anything containing `json` | Parsed as a JSON object via the VM's `table.json_decode`, keeping numbers, booleans and nested tables/lists; only the top-level fields are merged |
| `multipart/form-data` | Text fields are merged; file fields go to `files` |
| anything else | Treated as url-encoded form data |

A key that occurs more than once (for example `?tag=a&tag=b`) becomes a list.

`req.headers` holds every request header with its name lower-cased, so
`req.headers.authorization` or `req.headers.content-type` can be read directly; a header
that occurs more than once becomes a list too.

The four-argument signature is backward compatible: older three-parameter handlers keep
working, they simply never see `files`.

**Return values.** A handler returns either a string (sent as the body, `text/html` by
default) or a table:

```dlang
return {
    body: "text or a table (a table is JSON encoded)",
    contentType: "text/plain",
    statusCode: 200
};
```

If a handler throws, the server answers `500` using the configured error page.

### `server.static(...)`

Serves files from disk. Four call shapes are supported:

```dlang
server.static("/assets", "./public");            // URL prefix  -> folder
server.static("/style.css", "./style.css");      // exact URL   -> single file
server.static({ assets: "./public" });           // map of keys -> paths
server.static([["/a.css", "./a.css"]]);          // list of [url, file] pairs
```

Details:

- Table keys must be dlang identifiers (dlang tables have no string keys); a leading `/` is
  added automatically, so the `{ assets: "./public" }` form serves `/assets/...`.
- Folder mounts answer `/` with `index.html`, pick the longest matching prefix and detect the
  MIME type from the file extension.
- Exact URL → file mappings take precedence over folder mounts.
- Static files are only served when no route matches the path.
- Paths are URL-decoded first; anything containing `..` or `\` is rejected, so a request
  cannot escape the mounted folder.

### `server.error(code, handler)`

Overrides the page for a status code. The handler may be a body string, a function, or a
table:

```dlang
server.error(404, "<h1>Not found</h1>");
server.error(500, { body: "Something broke", contentType: "text/plain" });

server.error(404, func(req)
    // req = { method, path, statusCode }
    return { body: "Missing: " + req.path, contentType: "text/html" };
end);
```

Status codes without an override (and handler failures) fall back to a built-in dark-themed
HTML page. A custom body of `""` also falls back to the built-in page.

### `server.maxBody(bytes)`

Changes the maximum request body the server buffers. The default is **16 MiB**. Requests
that announce a larger `Content-Length` are refused with `413 Payload Too Large` before the
body is read:

```dlang
server.maxBody(64 * 1024 * 1024);   // allow 64 MiB uploads
```

### File uploads

An upload (`<form enctype="multipart/form-data">`, or `FormData` in `fetch`/XHR) is split
into its parts. Text fields end up in `params` like any other form post, and every file
field becomes an entry in `files`:

```dlang
func(req, params, cookies, files)
    var avatar = files.avatar;      // one entry per field name
    // avatar.field        -> "avatar"
    // avatar.filename     -> "cat.png"
    // avatar.contentType  -> "image/png"
    // avatar.size         -> 1234
    // avatar.content      -> the raw bytes
end
```

If several files are uploaded under the same field name (for example a multi-select), that
entry becomes a list of file tables. `filename*` (RFC 5987), quoted filenames, CRLF and
LF-only framing are all handled; malformed bodies are ignored instead of raising an error.

The same table is available as `req.files`, so a handler can read `req.files.avatar` if it
prefers not to use the fourth parameter. To store an upload, hand it to
[`http.save` / `http.saveTo`](#httpsave--httpsaveto):

```dlang
var saved = http.saveTo(files.avatar, "./uploads");
```

Raw bodies are available too: `req.body` holds the undecoded request body, which is handy
for uploads sent as `application/octet-stream`.

### `http.save` / `http.saveTo`

Write an upload to disk:

```dlang
http.save(upload, "./uploads/cat.png");   // -> true on success, false on failure
http.saveTo(upload, "./uploads");         // -> the written path (e.g. "./uploads/cat.png"), or "" on failure
```

The helpers live on the `http` table rather than a separate global, because a route
handler's fourth parameter is conventionally called `files` and would shadow a global of
the same name.

- `save` takes an explicit path and creates missing parent directories.
- `saveTo` appends the uploaded `filename` to a folder.
- Both accept a plain string as the first argument, so a raw body can be written as well:
  `http.save(req.body, "./uploads/blob.bin")`.
- The filename is sanitized before use: any directory part (`C:\Users\me\evil.txt`,
  `../../etc/passwd`) is stripped and unsafe characters are replaced, so a client can never
  choose the target directory.

### Authentication (`auth.*`)

Optional HTTP Basic auth. You choose which paths are protected; everything else stays open.

Passwords are **hashed, not encrypted**: the plugin generates a random salt per account and
stores a PBKDF2-HMAC-SHA256 hash of the password, so the file never holds anything that can
be turned back into the original password.

```dlang
require("dlang.http");

auth.addUser("admin", "geheim");     // hashes the password and writes ./users.json
auth.realm("My site");               // the text shown in the browser's login popup
auth.protectPrefix("/admin");        // /admin and everything below need credentials

var server = http.server();

server.route("/admin", ["GET"], func(req, params, cookies, files)
    return "<h1>Welcome back, " + req.user + "</h1>";
end);

server.listen(8080);
```

The first request for a protected path is answered with `401 Unauthorized` plus a
`WWW-Authenticate: Basic realm="..."` header. A browser reacts to that header by showing its
own username/password dialog; once the visitor fills it in, the request is repeated with an
`Authorization: Basic ...` header and the page is served. Inside the handler `req.user` holds
the authenticated name.

The user file lives in the working directory (`users.json` by default) — add it to your
`.gitignore`, since it holds password hashes.

#### The user store

| Call | Description |
| --- | --- |
| `auth.file(path)` | Use another user file; returns the path in use. Default: `users.json`. |
| `auth.reload()` | Re-read the file from disk (after editing it by hand). `true` on success. |
| `auth.addUser(name, password)` | Create the account or replace its password. Writes the file and returns `true`, or `false` when the name/password is rejected or the file cannot be written. |
| `auth.removeUser(name)` | Delete the account. `true` when it existed and the file was written. |
| `auth.hasUser(name)` | `true` when the account exists. |
| `auth.users()` | List of user names. |
| `auth.verify(name, password)` | `true` when the password matches — to check credentials inside your own handler. |
| `auth.login(req)` | The name from the request's `Authorization` header, or `""`. |
| `auth.iterations(count)` | PBKDF2 rounds for new or updated passwords (default `100000`, clamped to `1000`–`1000000`); returns the value in use. |
| `auth.realm(name)` | The realm in the login popup (default `Restricted`); returns the value in use. |

A name may not be empty or contain `:`, whitespace or control characters (it is sent as
`name:password`), and the password may not be empty. `addUser` never overwrites a
`users.json` it cannot parse — it reports the problem on `stderr` instead, so a hand-edited
file with a typo cannot silently wipe your accounts.

The file looks like this (`salt` and `hash` are hex; `iterations` is stored per account so
the cost can be raised later without invalidating existing passwords):

```json
{
  "algorithm": "pbkdf2-sha256",
  "users": [
    { "username": "admin", "salt": "ca63c036f511d4006c93ad68353677d7",
      "hash": "d4b7284ab9668ecbb548f8fd8048b44200368b258cd2b87d00807737c9aec472",
      "iterations": 100000 }
  ]
}
```

Accounts are read lazily, so `auth.addUser(...)` can simply run just before
`server.listen(...)`. A password that verified once is remembered in memory (as a hash of the
password, never the password itself), so the deliberately slow PBKDF2 work is paid for the
first request of a session instead of every request. Editing the file by hand? Call
`auth.reload()`.

#### Protecting paths

| Call | Description |
| --- | --- |
| `auth.protect(path)` | Require credentials for exactly this path. |
| `auth.protectPrefix(path)` | Require credentials for this path and everything below it. |
| `auth.unprotect(path)` | Remove an exact-path protection again. |
| `auth.unprotectPrefix(path)` | Remove a prefix protection again. |

Paths are normalised (`admin/panel/` becomes `/admin/panel`) and `auth.protectPrefix("/")`
gates the whole site. The check happens before routing, static files and error pages, so a
protected path never leaks whether it exists: `/secret/nope` answers `401` without
credentials and `404` with them. The `401` body comes from
[`server.error(401, ...)`](#servererrorcode-handler) like any other status code; only the
`WWW-Authenticate` header is added by the plugin.

### `view.Render(path, vars)`

Renders a small template file and returns the result as a string. `view.render` is an alias.
Template paths are read relative to the process working directory; a missing template raises
an error, which a route handler turns into a `500`.

```dlang
server.route("/", ["GET"], func(req, params, cookies, files)
    return view.Render("templates/page.html", {
        title: "Home",
        user: { name: "Ada" },
        items: ["a", "b"]
    });
end);
```

Template syntax:

| Syntax | Behaviour |
| --- | --- |
| `@include("partial.html")` | Inlines another template. Recursive, resolved relative to the including file (with a fallback to the working directory); cycles are skipped and nesting is limited to 32 levels |
| `{{ key }}` | Replaced with the matching variable. Strings, numbers and booleans are printed as-is |
| `{{ a.b }}` | Walks into nested tables |
| `{{ list }}` / `{{ table }}` | Serialized to JSON (convenient inside `<script>`) |

Unknown keys are left in the output as literal `{{key}}` text, which makes typos easy to
spot. Missing or cyclic includes are replaced with an HTML comment.

---

## Status codes

Responses use these standard reason phrases — anything else falls back to `OK`, while the
numeric status code is always the one you asked for:

`200 OK`, `201 Created`, `204 No Content`, `301 Moved Permanently`, `302 Found`,
`400 Bad Request`, `401 Unauthorized`, `403 Forbidden`, `404 Not Found`,
`405 Method Not Allowed`, `413 Payload Too Large`, `500 Internal Server Error`,
`501 Not Implemented`.

Responses are always sent with `Content-Length`, `Content-Type` and `Connection: close`.

---

## Request and response flow

```mermaid
flowchart TD
    A[Accept connection] --> B[Read headers + Content-Length body]
    B --> C{Body within limit?}
    C -- no --> D[413]
    C -- yes --> N{Protected path?}
    N -- "wrong or missing credentials" --> O[401 + WWW-Authenticate]
    N -- "not protected, or valid credentials" --> E[Parse query, headers, cookies and body<br/>urlencoded / JSON / multipart]
    E --> F{Route matches path?}
    F -- yes, method allowed --> G[Call handler with req, params, cookies, files]
    F -- yes, other method --> H[405]
    F -- no --> I{Static mount matches?}
    I -- yes --> J[Serve file with detected MIME type]
    I -- no --> K[404]
    G --> L[Send response: string, or table with body / contentType / statusCode]
    G -. handler threw .-> M[500 with error page]
```

---

## Limitations

- **One request at a time.** `listen` accepts and handles connections sequentially in a
  blocking loop; there is no thread pool or keep-alive.
- **Exact route paths only.** No `:parameters`, wildcards or regex routes; match on
  `req.path` yourself if you need them.
- **`Content-Length` only.** Chunked transfer encoding is not supported, and a request
  without `Content-Length` has no body.
- **In-memory bodies.** The whole body (up to `server.maxBody`) is buffered before a handler
  runs, and static files are read fully into memory before being sent.
- **HTTP/1.1 only**, no TLS. Put a reverse proxy in front if you need HTTPS.
- **Basic auth is only as safe as the transport.** The browser resends the password with
  every request (base64 encoded, not encrypted), so use it behind HTTPS — or keep it on a
  trusted network.
- **A wrong password costs a full PBKDF2 run.** `auth.iterations` defaults to 100 000, and
  the server handles one request at a time, so an attacker can keep it busy with bad
  credentials. Lower the iteration count or add rate limiting if that matters to you.
- **Global server state.** Routes, static mounts, error pages, the protected paths and the
  loaded accounts are process-wide, so one `listen` call serves one server per process.

---

## Project layout

```text
dlang-http/
├─ CMakeLists.txt                 # top-level project
├─ CMakePresets.json              # Windows / Linux / macOS configure presets
└─ dlang.http/
   ├─ CMakeLists.txt              # builds the `dlang.http` shared library
   ├─ entry.cpp                   # the entire plugin: server, routing, static files,
   │                              # uploads, error pages, Basic auth and the view engine
   ├─ json.hpp                    # nlohmann/json (used by the VM's table library)
   └─ dlang/                      # upstream dlang VM copy — do not edit
      ├─ vm.hpp, types/, libs/, parser/
      └─ src/                     # compiler, lexer, parser, VM implementation
```

Everything plugin-specific lives in `dlang.http/entry.cpp`; `dlang.http/dlang/` is a
verbatim copy of the dlang VM so the shared library can be built and link the VM
implementation.

---

## AI-generated code

This repository and this README were produced with the help of an AI assistant
(GitHub Copilot). That means:

- The code was **generated, then verified against its own test harnesses** (request parsing,
  multipart uploads, socket-level request handling, `http.save`/`saveTo`, and the auth
  module including its SHA-256/HMAC/PBKDF2 test vectors), but it has not been reviewed by a
  human maintainer.
- Comments and documentation may describe behaviour that has not been audited in every
  edge case, and the API is still evolving.
- Treat it as a starting point: review the source in `dlang.http/entry.cpp`, test it with
  your own inputs, and add a license before publishing or distributing it.

There is no warranty of any kind. Use it at your own risk.