This page is for people who contribute changes, add extractors, or integrate directly with Nogon HTTP and MCP interfaces.
The primary source repository is on GitHub. It is also mirrored on Codeberg.
Contributing to Nogon
Read the complete contribution guide before starting work. Keep each pull request focused on one concern, explain why the change is needed, and make sure relevant tests and checks pass. Open an issue before implementing a large architectural change, broad refactor, or development tooling change so the approach can be discussed first.
Ask questions when requirements or existing behavior are unclear. Review feedback concerns the work and the shared goal of improving Nogon. A useful contribution should provide more value to the project than the time required to review and maintain it.
Project Standards
Go code. Follow the GolangCI Lint v2 configuration in .golangci.toml.
Formatting uses goimports and gofumpt. Run focused package tests during
development and go test ./... before submitting a broad change.
Frontend code. Prettier uses single quotes, trailing commas, a print width of 100, and two space indentation. Prefer standard Tailwind scale classes. Check the shared component library before creating a new component.
Change scope. Preserve compatibility unless the change explicitly requires otherwise. Update tests and user documentation whenever behavior, command flags, configuration, query syntax, or public interfaces change. Keep release notes for the release process.
Licensing. Contributions are accepted under the project AGPLv3 license.
AI use. The contribution guide requires disclosure of every AI tool and the
extent of its use. The human contributor must understand the complete change
and remain its main author. Issue descriptions, pull request descriptions, and
discussion comments must be written by a human. AI tools must not be used for
good first issues. Coding agents should also follow the local AGENTS.md instructions.
Useful Development Links
| Resource | Purpose |
|---|---|
| Contribution guide | Submission standards, code style, and AI policy |
manage.sh | Common dependency, test, build, and artifact commands |
| Go module | Go version, module path, and backend dependencies |
| Go lint configuration | Enabled Go formatters and linters |
| Frontend workspace | npm workspaces, formatting, and frontend checks |
| Frontend notes | Web project structure, builds, previews, and shared components |
webui/build.sh | Embedded web application build process |
| API route registry | HTTP endpoint declarations and generated API documentation |
| Extractor template | Starting point for a new extractor |
| Issue tracker | Bug reports, proposals, and feature discussions |
| Security policy | Private security reporting guidance |
Source Layout
| Path | Purpose |
|---|---|
nogon.go | Program entry point |
cmd/ | Command implementations and the terminal interface |
client/ | HTTP client used by commands |
config/ | Configuration loading and validation |
files/ | Watched directory matching and file ownership |
files/extract/ | Parsing supplied file contents without document or index state |
server/ | HTTP server, models, indexing, crawling, extraction, and vector storage |
webui/app/ | Web application embedded in the Go binary |
webui/components/ | Shared Svelte components |
webui/ext/ | Browser extension |
webui/website/ | Public website and documentation |
scripts/ | Dataset utilities |
contrib/ | Service integration files |
The Go module is github.com/asciimoo/nogon. The root npm workspace includes
the web application, website, shared components, and browser extension.
Rename compatibility
Nogon retains identifiers used by existing installations and integrations. Preserve these when changing visible branding:
| Surface | Retained identifiers | Purpose |
|---|---|---|
| HTTP clients and sessions | hister cookie, hister:// origin, X-Hister-Public, X-Hister-Target-User-ID | Keep existing authentication and client requests working. |
| Browser state | hister* app preferences and extension storage keys, extension IDs and Chrome key | Preserve settings, server connections, and extension upgrades. |
| Search indexes | hister_keep_stopwords, hister_snippet, hister.* index metadata | Reopen existing indexes without changing their stored definitions. |
| Monitoring | hister_* metric names | Preserve queries and dashboards. |
| Configuration and services | Legacy environment aliases, config and data paths, service identities, container volumes | Reuse existing configuration, permissions, and data. |
| Distribution | Package command aliases, registry URLs, package identifiers | Keep existing consumers working while publication is coordinated separately. |
| Published content | Post slugs, old heading anchors, RSS GUIDs, historical examples and fixtures | Preserve links, subscriptions, and regression coverage. |
Internal helper symbols, temporary file prefixes, cache names, and test controls
use Nogon names. Test controls prefer NOGON_* and accept legacy HISTER_* aliases. Repository and package links should change only when their replacement
destinations are available.
The server accepts nogon://, X-Nogon-Public, and X-Nogon-Target-User-ID alongside the legacy identifiers. Nogon ownership headers
take precedence when both spellings are present, even when the Nogon value is
empty. Clients send both ownership header spellings for older servers. The CLI
retains the legacy origin so it can also connect to those servers.
Dataset utilities
Run sh scripts/fetch_dataset.sh to list available datasets. For crawled
datasets, the script indexes pages into the configured server and exports JSON
to its working directory. If the repository contains config.yml, it uses the
repository as its working directory.
Set NOGON to select an executable, for example:
NOGON=/path/to/nogon sh scripts/fetch_dataset.sh mdn-css A nonempty NOGON overrides the legacy HISTER variable. Without either
override, the script searches its working directory, the repository, then PATH,
preferring nogon over hister within each location. An invalid explicit path
fails without selecting another executable. Paths containing spaces are supported.
Run go test . -run '^TestDatasetScriptCommandAliases$' to check selection and
argument handling without crawling or indexing content.
Configuration loading
config.ResolvePath selects a main file. Pass the selected path to config.Load, or an empty
string to start without a main file. Loading composes defaults, managed selections from <app.directory>/config.web.yml, main YAML, environment, and explicit startup overrides in that
order. Resolve the data directory from authoritative sources before loading managed selections;
the managed file cannot change its own location. LoadForInspection and Config.Reload share candidate decoding without creating runtime files. Reload retains the original main file
selection, independently of the TUI preferences path.
Set NOGON__APP__LOG_LEVEL=debug to trace configuration loading before normal
logging is configured. The legacy HISTER__APP__LOG_LEVEL also works. The Nogon
variable takes precedence when present, including an explicitly empty value.
Config.Source(path) reports a setting’s origin and source lock without retaining its value.
Explicit empty and default valued settings remain distinguishable from omitted settings.
Collections share one source lock; extractor options are tracked individually. CLI overrides
use config.Override entries containing a path, typed value, and source. Load, LoadForInspection, and Config.Reload accept these entries after environment settings.
Loading applies values and provenance together and copies mutable inputs for each candidate.
Bind a CLI flag to its configuration path with bindConfigFlag beside its registration.
Only explicitly supplied flags become overrides. This metadata does not grant editing
permission or replace field validation.
Configuration editing API
GET /api/settings returns redacted field state and an opaque revision.
Field metadata includes a top level category and an optional display group.
The backend catalog defines groups and field order; clients preserve that order
when grouping fields within each category.
Catalog entries declare a ConfirmationPolicy alongside the field schema.
Its mode is acknowledge or access; its pure Describe function compares the
current configuration with the validated candidate and returns an empty string
when consent is unnecessary. Catalog requires_confirmation means the field
can require consent. Validation returns the decision for the actual draft.
Read only preview permission is separate from confirmation and preserves source
locks. The server owns acknowledgment binding and administrator checks.
Extractor options implement sdk.OptionDescriber; their schemas and defaults stay with the extractor.
The built in registry registers editable field schemas with config.RegisterExtractorSettings during
initialization so managed file loading uses the same types as draft validation without importing
the extractor runtime into config. Undescribed options and executable controls remain file managed. POST /api/settings/validate checks a draft without changing configuration or runtime resources. POST /api/settings/apply saves and applies supported edits across all configuration categories. Both POST
operations accept revision and an edits array with explicit path, action (set or reset), and a typed value for set. Access changes use the same catalog
and draft preparation as ordinary settings, with additional review and confirmation.
File managed restrictions still apply. These routes use administrative
authorization, CSRF protection for writes, and Cache-Control: no-store.
Secret fields use set with an action object such as {"action":"keep"}, {"action":"replace","value":"new credential"}, or {"action":"clear"}; reset remains a separate setting operation. Secret collection members use opaque references
scoped to the current revision. Read fresh state after a conflict before rebuilding a collection
edit. Do not submit redacted display text as a value or persist credentials in browser storage.
Public GET /api/config exposes only the canConfigure capability for Settings navigation,
not private editor state.
Validation can preview app.user_handling, app.access_token, app.public, and server.address, server.base_url, server.oauth_only, and described OAuth credential
and option fields. Its review contains a mode, redacted changes with path, before, and after, consequences, and an opaque acknowledgment.
Access reviews also include destination_url.
Source locks and file managed restrictions still apply. User handling requires
a usable administrator, including an independent local recovery administrator
when OIDC manages roles. Review sessions are established when authentication is disabled.
Apply requires that acknowledgment for sensitive drafts, including mixed drafts,
Reset, and Clear. The runtime binds it to the exact candidate, edits, revision,
and authenticated credential or initiating session. A new review replaces the
previous one. Submitting the issued acknowledgment consumes it, including when
the draft changed or administrator availability was lost. Reloads, saves, and
restarts invalidate reviews. Missing acknowledgments return 428 with acknowledgment_required; mismatches and replays return 409 with invalid_acknowledgment. Authority is rechecked after waiting for serialization.
Saves also recheck the credential after draining requests. Administrator
prerequisite failures for access changes return 422 with admin_required.
Acknowledgment mode uses ordinary persistence and activation. It covers enabling
SQL debugging or profiling, changing the embedding endpoint, and removing or
replacing sensitive content patterns. Disabling diagnostics, adding filters,
and unchanged effective values need no acknowledgment. The entire mixed draft
uses access mode if any effective change requires access verification.
Access activation returns 202 with a pending revision, deadline, and destination_url. authentication identifies required fresh proof: credentials, password, or oauth; oauth_provider identifies a specific provider when required. GET /api/settings exposes this state as pending_access without private proof tokens.
POST /api/settings/confirm and POST /api/settings/cancel accept the pending revision. Confirmation must reach the candidate host, base path, and listener;
reverse proxies must preserve the public Host header. Changed authentication policies
require a new login or an applicable current API token. OAuth verification requires
a login started after activation through the required provider. An existing session
or a login started before activation is insufficient. Authority is rechecked after
draining requests and administrator availability is checked before persistence.
When the candidate disables authentication, activation also returns a private confirmation_token, required by both completion operations. It is never returned
by reads and grants no access to other endpoints. If the activation response is lost,
wait for rollback. Clients must not forward stored credentials to a different origin.
Only one OAuth provider can change per draft. Client credentials, scopes, and PKCE options are supported. Provider endpoints, identity mappings, and providers that synchronize OIDC administrator roles remain file managed.
Access changes leave the confirmed YAML untouched until confirmation. Cancellation,
the two minute timeout, or a persistence failure restores the previous runtime when
possible; restart loads the confirmed file. Competing saves and reloads return 409 while confirmation is pending. Source changes before confirmation are not overwritten.
For ordinary changes, the runtime serializes saves with reloads, drains requests, stages private YAML
beside config.web.yml, and commits it before publishing the replacement runtime.
Saves with unchanged web selections preserve the file, runtime, and revision.
Adding or removing an explicit selection is saved even when its value equals the default.
The main file is never rewritten. Conflicts return 409; field errors return 422. Read settings again after success or a lost response. Persistence or
activation failures restore the previous configuration when possible; failed
recovery returns 503 and requires restarting the server. Rollback does not undo
indexed documents or other background work.
Runtime and persistence failures include a code and a safe error message:
| Code | HTTP status | Meaning |
|---|---|---|
sources_changed | 409 | Sources changed, including during activation or persistence. |
admin_required | 403 | The saving user lost administrator access while requests drained. |
persistence_failed | 500 | Staging or commit failed; the previous configuration remains active. |
recovery_failed | 503 | Recovery failed; the server requires manual recovery and restart. |
Other activation failures use apply_failed. An outdated editor revision uses stale_revision, separately from source changes.
Startup discards abandoned staging files. If restoring the managed file fails,
inspect config.web.yml and the private .config.web-recovery-* copies before
restarting.
Administrator availability is checked against the selected database before startup and activation, outside pure configuration loading. Missing administrators or independent local recovery accounts produce warnings; database inspection failures remain fatal. This allows OIDC login to restore administrator access. Account mutations still protect recovery accounts under the existing database write lock, while provider revocations remain authoritative.
Development Commands
Install the frontend dependencies once before running frontend commands:
./manage.sh install_js_deps Use these commands for common work:
| Goal | Command |
|---|---|
| Build the Go program | go build |
| Run Go tests | go test ./... |
| Run app browser tests | npm run test:browser -w @nogon/app |
| Run browser extension tests | npm test -w @nogon/ext |
| Rebuild embedded frontend assets | go generate |
| Build the complete Nogon binary | ./manage.sh build |
| Build the web application | npm run build -w @nogon/app |
| Build this documentation website | npm run build -w @nogon/website |
| Build the browser extension | npm run build -w @nogon/ext |
| Run the app development servers | npm run serve:app |
| Format and lint Go code | golangci-lint run --fix ./... |
| Check frontend formatting and linting | npm run check |
| Apply frontend formatting | npm run format |
go generate runs webui/build.sh. That script builds @nogon/app, replaces server/static/app, and copies the new frontend build into the embedded asset
directory. Run it before building the Go binary whenever embedded frontend code
changes.
npm run serve:app starts the web application with hot reload and the Go
server with automatic rebuilds. It installs air when that tool is not already
available.
Use focused tests while developing. Run the complete Go suite before handing off a broad or high risk change.
The app browser tests use Chromium with desktop and mobile layouts. Install the browser before the first run:
npm exec -w @nogon/app -- playwright install --with-deps chromium Playwright starts the app development server and the tests mock the HTTP API and
search WebSocket, so these tests do not require a running Go server. Set PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH to use an existing Chromium installation.
Pull requests run app browser tests and extension tests in separate jobs when
their workspace, shared components, root npm dependencies, or the test workflow
changes. CI rejects focused browser tests such as test.only and runs one browser
worker. Failed browser runs upload an app-browser-failures artifact containing
the HTML report, traces, and screenshots, retained for seven days.
Release Binaries
Tagged releases provide standalone nogon_VERSION_OS_ARCH binaries, with .exe on Windows, and nogon_VERSION_checksums.txt. GoReleaser applies binary archive names when uploading; local paths remain under the build directory.
Windows releases also include nogon_VERSION_windows_amd64.zip, containing only nogon.exe. Its checksum is included in the same checksum file. New releases
do not provide Hister binary downloads or an additional hister.exe in the ZIP.
Rolling builds use .goreleaser-rolling.yml and copy each executable to dist/nogon_cgo_OS_ARCH, with .exe on Windows. The upload step selects these
copies and leaves build metadata out of the release. WinGet continues to use
the existing asciimoo.hister package identifier and selects Nogon downloads.
Run go test . to check release filenames, rolling copies, upload selection,
and WinGet compatibility. To build a local rolling snapshot without publishing:
goreleaser check -f .goreleaser-rolling.yml
goreleaser build --snapshot --single-target --clean -f .goreleaser-rolling.yml Package Manager Rename Handoff
The package definitions were checked on 2026-11-17. Keep the current package identities for the first Nogon release so existing installations can upgrade.
Homebrew
contrib/homebrew/hister.patch prepares the upstream Hister formula for the first Nogon source release. It uses @nogon/app, installs nogon with
a hister alias, and preserves completions for both commands. Its tests use config create --full and exercise both names. The formula name, service command, HISTER_DATA_DIR, working directory, and log paths stay unchanged.
Apply the patch in a Homebrew core checkout together with the new source version, checksum, and bottle updates. The existing v0.21.0 source lacks the renamed workspace and current config command, so the patch is not a standalone update to that release. Check it against the current formula before applying:
git apply --check /path/to/nogon/contrib/homebrew/hister.patch
git apply /path/to/nogon/contrib/homebrew/hister.patch After updating release metadata, build with Homebrew, run brew audit --strict hister and brew test hister, and verify an existing brew services installation
keeps its data and service identity. Ruby syntax and the formula’s test block
have been checked locally against the current native executable; a full Homebrew
build and service upgrade still need verification.
A later formula rename also requires Homebrew’s rename mapping and migration checks. Do not introduce a second independent formula or service for this compatibility release.
WinGet
Both publishing workflows retain the asciimoo.hister package identifier and
select the Nogon standalone Windows executable. The installed command depends
on the published package manifest.
After building a stable release, use Python 3 to prepare local manifests from
its Windows ZIP. Replace VERSION with the actual release version:
python3 contrib/winget/prepare.py dist/nogon_VERSION_windows_amd64.zip dist/winget/VERSION The script checks that the archive contains only nogon.exe, calculates its
SHA256, and writes three manifests into a new directory. They set the display
name and command to Nogon and preserve asciimoo.hister and the current GitHub
release URLs. The script
does not publish manifests or overwrite an existing output directory. Snapshot
versions are not accepted.
Validate the output with winget validate --manifest PATH on Windows. Verify
fresh installation, upgrade from the existing portable package, the nogon command,
retained configuration and data, and uninstall before submitting the manifests.
Local schema validation does not establish that the upgrade works.
Prepare and review the archive manifest explicitly. Verify a subsequent automated update retains its portable command before either publishing workflow selects the ZIP.
Container Compatibility Checks
go test . runs the Dockerfile healthcheck with both environment variable
spellings. When Docker Compose is installed, it also resolves compose.yml to
check image overrides and the existing project volume identity. These tests do
not need the Docker daemon. The Docker pull request workflow builds the image
and checks command aliases, configuration paths, data directory overrides, and
startup through Compose.
The image keeps HISTER_DATA_DIR=/hister/data as its default so existing runtime
overrides still apply. The executable is /usr/local/bin/nogon, with aliases at /usr/local/bin/hister and /hister/hister. Compose retains its service, container,
and volume names. Registry addresses remain unchanged until replacement images
are published.
Nix and Service Compatibility Checks
Run nix build --no-link .#checks.x86_64-linux.compatibility to evaluate the
package and module aliases and exercise the package install hook. The check
verifies NixOS service identity, existing storage paths, mixed option names,
disabled services, and conflicting settings. It also evaluates the Home Manager
and Darwin module outputs for Linux and Darwin. These checks do not build the
full application or activate services; native launchd activation still needs
verification on macOS.
Module exports have stable identities so importing both names loads each module
once. Linux service names, Darwin agent names, and legacy environment defaults
remain unchanged. go test . -run TestSystemdServiceAliasInstallation uses systemctl --root with a temporary fixture to verify that enabling the supplied
unit creates nogon.service as an alias of hister.service. See the systemd alias documentation for how these names resolve to one unit.
ZIM Import Development
cmd/zim.go implements nogon import zim with the pure Go stazelabs/gozim reader, pinned in go.mod. It supports uncompressed, XZ, and Zstd content without an external executable. The importer checks header and pointer table bounds before opening the archive, disables memory mapping, and caches one cluster. It reads selected metadata and uses error returning directory lookups so damaged entries cannot silently disappear.
The reader decompresses complete clusters before checking individual article sizes. Its limits are 4 GiB of encoded cluster data and 1 GiB of decoded compressed data; the article limit is not a total memory limit. This is a newer library than libzim, so verify real archives and malformed input tests before changing the pin.
URL discovery reads canonical, Open Graph, and Twitter metadata, or absolute entry URLs. Relative metadata uses the HTML base or another known page URL; --base-url provides an optional fallback. Entries without a URL produce a warning and increment the skipped count. Collection names are not used to infer origins.
The importer makes two passes. The first retains entry indices, source URLs, and redirect targets; the second prepares documents and rewrites relative article links through that lookup. This handles forward links and archive paths that differ from website paths without retaining all HTML in memory. Redirect traversal is bounded, unresolved links are disabled, and page URL metadata remains absolute after removing HTML base elements. Normal authenticated submission and server extraction handle the prepared documents. Importer metadata retains only source and a nonempty zim_collection. The archive publication date is stored in Added; entry paths remain local to link resolution and warnings.
go test ./cmd -run TestZIM always runs integration tests against the checked in archives in cmd/testdata/zim. These fixtures were generated from the test articles with libzim 3.13.1. The legacy fixture converts the article namespace and cluster compression to XZ and was verified with libzim. Tests also cover operation with an empty executable path and malformed file offsets. Set NOGON_ZIM_LEGACY_FIXTURE to data/withns/small.zim from the official ZIM testing suite for an additional compatibility check.
HTTP API Development
The running server publishes its generated API documentation at /api. API
routes are declared as Endpoint values in server/api.go. Add new API routes
there so routing, authorization, CSRF handling, and API documentation remain in
sync.
Authenticated clients can send either of these headers:
Authorization: Bearer <access-token> X-Access-Token: <access-token> Browser Session Authentication
User sessions store the account’s user_id and user_session_version at login.
Password and OAuth logins both copy the version from the authenticated user
record. Each request checks that the stored version matches the account’s current SessionVersion. Sessions contain no password or password hash.
model.UpdatePassword saves the password hash and increments session_version in one database update. A failed update leaves both unchanged. Any code that
updates account passwords must preserve this atomic invalidation. A session
refresh saves the original version, so an in progress refresh cannot restore
access after a password reset. Personal access tokens are independent of the
session version.
The session_version column defaults to zero and is added to existing accounts
by AutoMigrate. Sessions created by earlier versions lack user_session_version and are accepted only while the account’s version is zero. An upgrade preserves
those sessions; a subsequent password reset invalidates them. Invalidated session
records remain until the normal expiration cleanup removes them.
User Administration Reads
The user administration endpoints require app.user_handling: true on the server
and an administrator’s personal token or authenticated session. Public mode does
not grant access to these endpoints.
| Endpoint | Response |
|---|---|
GET /api/users | A users array and an optional next_after_id cursor |
GET /api/users/{id} | Metadata for one account |
GET /api/users/{id}/token | The account’s current access token as {"token":"..."} |
Account metadata contains only id, username, is_admin, created_at, updated_at, and revision. Passwords, password hashes, tokens, rules, and OAuth identities are
excluded. Timestamps use RFC 3339 format. The token endpoint reads the existing
token without rotating it. Successful responses from all three endpoints use Cache-Control: no-store.
Listing accepts these optional query parameters:
| Parameter | Meaning |
|---|---|
username | Exact, case sensitive username. Wildcards are treated literally. |
limit | Page size from 1 to 100, with a default of 50 |
after_id | Return users whose IDs are greater than this value, with a default of 0 |
Results are ordered by ascending user ID. Pass next_after_id as the next request’s after_id, keeping the same filter. The final page omits next_after_id, and an
empty result uses "users": []. Pagination does not freeze the user database;
accounts created or changed between requests can affect subsequent pages.
GET /api/users?username=alice&limit=1
Authorization: Bearer <administrator-token> User IDs in paths must be positive integers. Unknown, repeated, or malformed list
parameters are rejected. Handler errors use a JSON code and readable error:
{ "code": "user_not_found", "error": "user not found" } | Status | Code | Meaning |
|---|---|---|
400 | invalid_request | Invalid ID or query parameters |
400 | user_handling_disabled | User handling is disabled on this server |
404 | user_not_found | No account exists for the requested ID |
500 | internal_error | The account lookup failed |
Authentication failures use the existing 403 response from the authorization
middleware. A username filter with no match returns an empty list, not 404.
Database error details are logged on the server and are not included in responses.
User Creation
POST /api/users creates an account on the running server. It requires app.user_handling: true and an existing administrator’s token or session, even
in public mode. It cannot create the first administrator without authentication.
POST /api/users
Authorization: Bearer <administrator-token>
Origin: hister://
Content-Type: application/json
{"username":"alice","password":"example-password","is_admin":false} The body must be a single JSON object. username and password are required
strings. is_admin is an optional boolean that defaults to false. Unknown
fields, null values, and trailing JSON are rejected.
Usernames are case sensitive and stored unchanged. They may include Unicode and
spaces, but must not be blank or contain control characters. Passwords must
contain 8 to 72 UTF-8 bytes; a Unicode character may occupy multiple bytes.
Passwords are hashed before storage. Creating an account does not enable password
login when server.oauth_only is set.
Success returns 201 Created with the same account metadata as GET /api/users/{id} and Cache-Control: no-store. The Location header points
to that account’s URL, including any configured base path. The response does not
include the password or access token. An administrator can explicitly retrieve
the generated token with GET /api/users/{id}/token.
Command line clients use Origin: hister:// with their administrator token, as
shown above. Browser session requests follow the existing CSRF checks. Requests
from a different origin must supply the session’s X-CSRF-Token, available from GET /api/config.
Errors use the same JSON code and error fields as the account read endpoints:
| Status | Code | Meaning |
|---|---|---|
400 | invalid_request | Invalid body, username, password, or admin flag |
400 | user_handling_disabled | User handling is disabled on this server |
409 | username_exists | The username is taken, including concurrent creation attempts |
413 | request_too_large | The request exceeds the 1 MiB body limit |
500 | internal_error | The account could not be saved |
Authentication and CSRF failures use the existing 403 responses. A conflicting
creation request leaves the existing account’s credentials and privileges
unchanged.
User Updates
PATCH /api/users/{id} applies account changes in one transaction. It requires
user handling, an authenticated administrator, and the same CSRF protection as
account creation.
Read the account first, then send its current revision with at least one change:
PATCH /api/users/2
Authorization: Bearer <administrator-token>
Origin: hister://
Content-Type: application/json
{"revision":1,"username":"alice-renamed","password":"replacement-password","is_admin":false,"regenerate_token":true} username, password, and is_admin are optional. Omitted fields keep their
values. An explicit false removes administrator privileges. Username and
password validation matches account creation. regenerate_token: true replaces
the access token; false alone does not count as a change. Null values, unknown
or repeated fields, and trailing JSON are rejected. The body limit is 1 MiB.
Success returns 200 OK with the updated account metadata and Cache-Control: no-store. Only when regenerate_token is true does the response
also contain token, holding the new access token. A client rotating its own
token must retain that replacement. Ordinary updates never return credentials.
Account revisions start at 1 and increase when a username, password,
administrator status, or token is updated, including through existing account
commands and profile token rotation. Rule changes do not change this revision.
A stale revision returns 409 without applying any part of the request. Fetch
the latest account, review the changes, and submit a new request with its current
revision. Do not retry an update blindly with a new revision.
Password changes invalidate the account’s existing browser sessions, including
OAuth sessions, while retaining its token unless rotation is also requested.
Setting a password does not bypass server.oauth_only. The last administrator
cannot be demoted or deleted.
Handler errors use the JSON code and error fields:
| Status | Code | Meaning |
|---|---|---|
400 | invalid_request | Invalid ID, revision, body, or field value |
400 | user_handling_disabled | User handling is disabled |
403 | admin_required | The caller lost administrator privileges before the update |
404 | user_not_found | The target account does not exist |
409 | stale_revision | The account changed since it was read |
409 | username_exists | Another account already uses the requested username |
409 | last_admin | The change would remove the last administrator |
409 | last_recovery_admin | OIDC role management requires this local recovery account |
413 | request_too_large | The request exceeds the body limit |
500 | internal_error | The update could not be saved |
Authorization failures before the handler use the existing middleware’s 403 response. Validation failures and transaction failures leave all account fields
unchanged.
User Token Rotation
POST /api/users/{id}/token lets account owners and administrators replace an
access token. It requires user handling and the same CSRF protection as account
updates. Get your user_id and revision from GET /api/profile; administrators
can read another account’s revision through GET /api/users/{id}.
POST /api/users/2/token
Authorization: Bearer <current-user-token>
Origin: hister://
Content-Type: application/json
{"revision":1} Send only the current revision. Success returns 200 OK with Cache-Control: no-store:
{ "token": "<new-access-token>", "revision": 2 } Save the returned token: the old token stops working immediately. Browser
sessions remain valid. A stale revision returns 409 with code stale_revision without changing the token. Refresh the account before retrying explicitly.
User Deletion
DELETE /api/users/{id} deletes an account. It requires user handling, an
authenticated administrator, and the same CSRF protection as account updates.
Set ?purge=true to remove the account’s indexed documents first; the default
is false.
Success returns 200 OK with {"id":2,"username":"alice","deleted_documents":3}.
An unknown ID returns 404. Conflicts return 409 with code purge_required when documents remain or last_admin when deleting the final administrator.
Purged documents are not restored if account deletion fails. See Data Lifecycle and Retention for retention limits.
Retained Data Purge
POST /api/purge requires administrator access in multi user mode and the normal
authentication and CSRF checks. The selection applies across all users. Supply
an exclusive before cutoff as an RFC3339 timestamp in the past and at least one
of versions, history, or crawl_jobs:
{
"before": "2026-01-01T00:00:00Z",
"versions": true,
"history": true,
"crawl_jobs": true,
"dry_run": true
} Success returns dry_run and integer counts for versions, history_entries, history_queries, history_urls, crawl_jobs, and crawl_urls. A dry run counts
rows without deleting them, including query and URL rows that would become
unreferenced. Otherwise the counts describe rows deleted in one SQL transaction.
A transaction failure rolls back all selected categories and returns 500.
Invalid cutoffs, empty selections, unknown fields, and trailing JSON return 400.
Version age uses created_at; history entry, orphan query and URL, and crawl job
age uses updated_at. Pinned history is retained. Only completed crawl jobs with
no pending or in progress URLs qualify. Current index documents, assets, vectors,
and embedding jobs are unaffected. See Purging Retained Data for user guidance and limitations.
Request Limits
API request bodies have a default limit of 1 MiB, including MCP requests and login requests. The following endpoints have their own limits:
| Requests | Body limit |
|---|---|
POST /api/add, POST /api/add_pdf, POST /api/batch | server.max_batch_body_size MiB, default 40 MiB |
POST /api/rules/preview | 1 MiB |
Limits apply to JSON and form data, including requests sent with chunked transfer
encoding. The PDF upload limit includes the base64 content and the surrounding
JSON. An oversized request returns HTTP 413 with a JSON body containing error, code: "request_too_large", and limit_bytes.
Search WebSocket messages have a limit of 64 KiB across all frames in a message.
An oversized message closes the connection with code 1009.
The HTTP server allows 10 seconds to receive request headers and two minutes to receive the entire request, including its body. It closes idle keepalive connections after one minute. These limits do not impose a duration limit on handlers such as reindexing after the request has been received.
POST /api/reindex returns 200 for completion, 422 for an embedding dimension mismatch, or 500 for
other rebuild failures. The dimension error body identifies the expected and
returned sizes without including embedding endpoint details. Disconnecting
cancels the rebuild, including active embedding requests.
Document Versions
GET /api/versions?url=<url> The endpoint returns stored version differences for the given URL, with the
newest record first. Each entry contains id, created_at, url, user_id, html_diff, and text_diff. The difference values use diff match patch patch strings.
Current Profile
GET /api/profile returns information about the currently authenticated user:
{
"user_id": 1,
"username": "alice",
"is_admin": true
} Document Language Handling
User configuration and correction workflows are described in Language Settings and Detection.
server/languages resolves language tags, aliases, detector candidates, and
Bleve analyzer routing. These are separate concerns: a valid document tag can
lack a detector model or a language analyzer. Regional tags share their base
language’s analyzer and index. Norwegian Bokmal (nb) uses analyzer no and index_no.db; Chinese, Japanese, and Korean share the CJK analyzer while keeping
separate indexes. Unknown languages and tags without an analyzer use index.db.
server/document runs Lingua on extracted text. Low accuracy uses trigram
models; high accuracy can use models for one through five letters on short
text. Both modes use trigrams for text with at least 120 letters. Models load
lazily and remain cached for the process lifetime.
The built in detector requires at least 12 Unicode letters or four letters
from the Han, Hiragana, Katakana, or Hangul scripts. It then requires a top
confidence score of at least 0.5 with all languages, or 0.8 with an explicit
candidate list, and a margin of at least 0.15 over the runner up. These are
heuristics; scores are relative to the candidate set, not calibrated
probabilities. An excluded language can still receive an incorrect confident
classification.
Document.ResolveLanguage tries a web title only when the body has insufficient
text, not when a longer body is ambiguous. File titles are excluded from this
fallback. A root HTML lang or xml:lang declaration can refine an agreeing
base language. Parsing is capped at 64 KiB, and child attributes are ignored.
Declarations cannot bypass disabled detection or override conflicting text. EvidenceLanguageDetector exposes transient scores and decision reasons;
implementations of the original LanguageDetector interface remain supported.
Resolution leaves document metadata unchanged.
The language_override document field persists a manual pin separately from language. Export and import preserve it. Ordinary resubmissions retain an
existing pin. /api/update accepts {"language":"auto"} in changes to clear
it and reevaluate saved content. Title edits also reevaluate unpinned web
documents. Updates load stored HTML to preserve agreeing regional declarations.
The authored regression corpus in server/document/testdata/language_corpus.json covers eleven languages and
short or empty inputs in both accuracy modes. It checks expected behavior,
not accuracy on representative browsing collections. Run the focused tests with:
go test ./server/languages ./server/document ./server/indexer/... ./cmd Extractor Development
Extractors are components that enrich metadata, produce searchable content, and render document previews. Indexing first runs every matching enricher in registration order. It then tries matching content extractors in registration order until one succeeds. Preview selection considers only extractors that declare the preview capability.
Extractor Decisions
Extract and Preview return explicit result values. Each result records one
of three decisions:
| Decision | Meaning |
|---|---|
| Success | The extractor handled the document successfully. |
| Fallback | The extractor was inconclusive, so the next matching extractor should be tried. |
| Abort | A fatal error occurred. The chain stops immediately and returns the error without trying a fallback. |
Use Extracted, ExtractFallback, or AbortExtraction for extraction
results. Use Previewed, PreviewFallback, or AbortPreview for preview
results. The result fields are private, so an implementation cannot return a
contradictory decision and error combination. The zero value is invalid.
When a caller names a preview extractor explicitly, that extractor must be
enabled, preview capable, and matched to the document. Selection starts there.
If it returns PreviewFallback, later matching preview extractors may still
provide the response.
If no extractor succeeds, ErrNoExtractor is returned.
Extractor Interface
A custom extractor can depend on the focused server/extractor/sdk package instead of importing Nogon configuration, document, and result
packages separately. It must implement this interface:
type Extractor interface {
// Name returns a human readable identifier used in logs and config.
Name() string
// Description returns a short human readable summary for clients.
Description() string
// Capabilities declares the extractor phases this implementation joins.
Capabilities() sdk.Capabilities
// Match reports whether this extractor applies to the given document.
// Extract and Preview are only called when Match returns true.
Match(*sdk.Document) bool
// Extract rewrites the document before it is added to the index.
// Return an explicit success, fallback, or abort result.
Extract(*sdk.Document) sdk.ExtractResult
// Preview returns a rendered representation suitable for display.
// Return an explicit success, fallback, or abort result.
Preview(*sdk.Document) sdk.PreviewResult
// GetConfig returns the extractor's current configuration.
// It must return sensible defaults before SetConfig is called.
GetConfig() *sdk.Config
// SetConfig applies user supplied configuration on top of defaults.
// Return an error for any unrecognised option key.
SetConfig(*sdk.Config) error
} Capabilities
Capabilities keep metadata enrichment, searchable content extraction, and preview rendering independent:
type Capabilities struct {
Enrich bool
Extract bool
Preview bool
} An enricher annotates every matching document and does not compete to select the searchable body. A content extractor can populate the title and text. A preview extractor can render the stored document. Most specialist extractors declare both content and preview capabilities.
Result Types
sdk.ExtractResult and sdk.PreviewResult describe the outcome of each phase.
Construct them with these helpers:
return sdk.Extracted()
return sdk.ExtractFallback(err)
return sdk.AbortExtraction(err)
return sdk.Previewed(sdk.PreviewResponse{Content: html})
return sdk.PreviewFallback(err)
return sdk.AbortPreview(err) Fallback errors are optional diagnostics. Abort helpers always carry an error.
Passing nil to an abort helper produces a descriptive error rather than an
ambiguous result.
Document
sdk.Document aliases the complete document.Document type. It is passed to Match, Extract, and Preview.
PreviewResponse
sdk.PreviewResponse carries the output of Preview:
type PreviewResponse struct {
Content string // HTML or plain text to render
Template string // optional custom frontend template name, leave blank for default
} Register an Extractor
Built in extractors are constructed by DefaultExtractors in server/extractor/registry.go.
Add an instance there before the generic fallbacks so it takes priority for
the pages it targets.
Applications can also create an isolated chain or extend a fresh default one:
registry := extractor.NewDefaultRegistry()
if err := registry.RegisterBefore("Readability", &MyExtractor{}); err != nil {
return err
}
if err := registry.Init(configuredExtractors); err != nil {
return err
} The package level extraction functions remain available and use extractor.DefaultRegistry().
Write a New Extractor
A ready to use starting point lives at server/extractor/extractors/_extractor_template/extractor.go.
The directory begins with _ so the Go toolchain ignores it during normal
builds, but the file itself is valid and fully commented Go.
- Copy
server/extractor/extractors/_extractor_template/toserver/extractor/extractors/<myname>/and remove the leading_. - Change the
packagedeclaration to match the new directory name. - Rename
TemplateExtractorto something descriptive, such asHackerNewsExtractor. - Update
matchURLPrefixand theMatchfunction for the target site. - Update
Capabilitiesto declare the phases the extractor supports. - Implement
Extractto populated.Title,d.Text, and optional metadata. - Implement
Previewto return sanitized HTML, or returnPreviewFallbackto reuse the generic readability preview. - Add an import and an extractor entry to
DefaultExtractorsinserver/extractor/registry.go, before the readability extractor.
Extractor Configuration Support
Embed sdk.ConfigSupport to provide GetConfig and SetConfig. Its zero value
enables the extractor and rejects every option key:
type MyExtractor struct {
sdk.ConfigSupport
} Use sdk.NewConfigSupport when an extractor has custom defaults. Keys present in
the default options map are accepted automatically:
func NewMyExtractor() *MyExtractor {
return &MyExtractor{
ConfigSupport: sdk.NewConfigSupport(sdk.Config{
Enable: true,
Options: map[string]any{
"timeout": 10,
},
}),
}
} Additional accepted keys without defaults can be passed after the default configuration. An extractor can implement its own configuration methods when applying configuration requires extra work.
Config merging from defaults and user values is performed automatically by Registry.Init before SetConfig is called. SetConfig always receives the
fully resolved configuration.
Extractor Guidelines
Avoid additional HTTP requests. Work with the HTML and metadata already
available in the Document struct wherever possible. Making extra requests
inside an extractor adds latency, increases network traffic, and can fail
silently in offline or restricted environments. Outbound requests also expose
the user IP address and browsing activity to external servers. Make such a
request only when there is no reasonable alternative.
Avoid embedding third party content. Strip or discard remote images,
videos, iframes, and other externally hosted media before returning content
from Extract or Preview wherever possible. Embedded content causes the
browser to contact an external server whenever a preview is opened, which can
expose the user IP address without their knowledge. When multimedia is
essential, prefer a placeholder button that loads it only after explicit user
action.
Use custom preview templates when they add value. If extracted content has
a clear structure, return a template name in PreviewResponse and build a
dedicated Svelte template for it. Code documentation, question and answer
threads, and recipes often benefit from a tailored layout.
The reusable issue template accepts JSON encoded from sdk.IssuePreview in PreviewResponse.Content. Set Template to "issue". It supports a title,
number, repository, URL, state, labels, a description in body, and an ordered comments array. The description and comments use sdk.IssueComment, with
optional author, authorURL, createdAt, and url fields plus bodyHTML.
Use plain text for metadata, absolute HTTP or HTTPS URLs for attribution links,
and sanitized HTML for bodyHTML. Resolve relative content URLs before
sanitizing. The web interface renders the layout, and the terminal client
converts the same payload into readable text. Include only comments present
in the saved page.
Test Extractors Against Live Websites
Live extractor cases are declared in server/extractor/live_cases.yaml.
Each case identifies a public URL, crawler backend, extractor, and a set of
stable content or metadata expectations. Prefer semantic invariants such as the
page type, a durable title fragment, and minimum content length over exact page
snapshots.
Run every live case with:
./manage.sh run_extractor_tests Pass a case name fragment to run a subset:
./manage.sh run_extractor_tests discourse The runner fetches each page through the configured Nogon crawler backend and
respects its robots rules. It checks the selected extractor directly, checks the
complete extractor chain unless run_chain is false, and checks previews when
preview expectations are present. Identical fetches are reused within a test
run.
The live suite uses the live Go build tag and is not part of a normal go test ./... run. The manifest structure is still validated during normal
tests, so invalid fields and unknown extractor names fail without making
network requests. Validation also requires at least one positive live case for
every registered extractor.
When a live case fails, the fetched HTML, extracted documents, preview, and a
summary are saved under /tmp/nogon-live-extractors. Set NOGON_LIVE_ARTIFACT_DIR to use another directory. These artifacts help
distinguish a website structure change from a transient fetch failure.
MCP Protocol Testing
The MCP endpoint uses Streamable HTTP at POST /mcp. Every interaction is a POST request with a JSON RPC 2.0 body. The server responds with a JSON object.
The authorization header is required only when authentication is enabled in
the Nogon configuration.
Initialize
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' After a successful response, send the initialized notification. The server
acknowledges it with 202 Accepted and no body:
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","method":"notifications/initialized"}' List Tools
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' Search
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search","arguments":{"query":"python async","limit":5}}}' If result.structuredContent.trusted.next_page_key is present, pass its value
as arguments.page_key in the next call. Treat the cursor as opaque and keep
the query, date filters, and semantic setting unchanged. Each response respects limit; the last page may be empty. Search cursors track both the index batch
and the position within the combined results so pagination does not skip
keyword matches when history fills a page. Semantic matches follow the keyword
batches, with earlier keyword and history matches excluded.
Include full text and language fields when testing optional result data:
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"search","arguments":{"query":"python async","limit":5,"fields":["text","language"]}}}' Preview
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"get_preview","arguments":{"url":"https://example.com/article"}}}' Use fields to select text, html, and metadata. Omitted fields default
to all three; an empty array returns only document identity and timestamps.
HTML preview rendering runs only when HTML is selected.
For bounded responses, set max_chars and continue with text_offset or html_offset. These values count Unicode code points after source normalization,
not UTF-8 bytes. Each returned body reports its length and whether more remains
in trusted.text_total_chars / trusted.html_total_chars and trusted.text_truncated / trusted.html_truncated. When more remains, use trusted.next_text_offset or trusted.next_html_offset for the next request.
Zero max_chars means unlimited, and offsets at or beyond the end return an
empty body without a continuation offset. Metadata is not truncated. HTML
portions may split tags or entities and must be reassembled before rendering.
Indexed History
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"get_history","arguments":{"mode":"indexed","limit":10}}}' Facet Counts
curl -s -X POST http://127.0.0.1:4433/mcp
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream"
-H "Authorization: Bearer <your-access-token>"
-d '{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"get_facets","arguments":{"query":"python","limit":5}}}' The response’s trusted.reported_total counts matching indexed documents.
Facet data is in untrusted_content[0].fields, with terms and date_histogram using the same shape as /api/facets. A term facet contains terms with term, count, and optional label values, plus an optional other count. Date buckets have name and count values and may be absent
when there are no matches. Facet source values remain inside the untrusted
content boundary in both structured content and the text fallback.
Supported Methods
The endpoint implements MCP specification version 2025-06-18 with these
methods:
| Method | Description |
|---|---|
initialize | Capability negotiation, required before any other call |
ping | Liveness check |
tools/list | Returns the list of available tools and their input schemas |
tools/call | Executes a tool by name with the provided arguments |
notifications/initialized | Acknowledged with 202 when sent as a notification |
notifications/cancelled | Acknowledged with 202 when sent as a notification |
Every tool advertises an outputSchema. Tool call results include the required content field and a structuredContent object conforming to that schema.
Clients should use structuredContent and enforce the untrusted_content trust markers when placing results into model context.
File Extraction
files/extract owns PDF, DOCX, Markdown, Org, HTML, and plain text parsing.
Construct a parser with extract.New(cfg.Indexer.PDFParser) and call Extract(ctx, name, content). The name selects a format; parsing uses the
supplied bytes. Results contain format, title, text, and HTML. Errors return
an empty result. Parser configuration is copied at construction, and content
limits apply to every call. Org file references are rejected.
server/extractor.ExtractFile adapts these results to a document and runs the
web extractor registry for saved HTML. It applies content fields only after
successful extraction and size checks. Both CLI snapshots and server indexing
use this adapter. File access, directory rules, timestamps, ownership, sensitive
content checks, language detection, and persistence remain with the callers.
The parser package has no dependency on documents, the database, or Bleve.
Documentation Maintenance
Write user documentation around settings, actions, and observable behavior. Keep implementation details, internal algorithms, and test design on this Developer Documentation page.
Add documentation pages under webui/website/src/content/docs and register
their slugs in webui/website/src/lib/docs-structure.ts. The docs search index,
page routes, previous and next links, and sitemap are generated from this
content during the website build.
The public website origin is set in webui/website/src/lib/config.ts. Canonical
URLs, social metadata, sitemap locations, robots instructions, and RSS links
use this value. RSS item GUIDs retain the original hister.org URLs as stable
identifiers so existing subscribers do not receive old posts again. Keep these
identifiers unchanged when updating the public domain.
When changing query syntax, update query builder tests and the query language
guide. When changing configuration, update the generated configuration
reference data, examples, and related operator guidance. When adding an API
route, declare it in server/api.go so the built in API documentation remains
complete.