Skip to main content

Rules

Rules let you control how Nogon indexes and surfaces documents. They live in {data_directory}/rules.json in single-user mode, or per-user in the database when user handling is enabled. The Rules tab in the web interface (and the TUI) is the easiest way to manage them.

Rule Types

Allow rules

Allow rules limit indexing to URLs matching at least one allow pattern. Patterns are combined with OR: a match against any entry is enough. Skip rules still take precedence, so you can allow a whole site while excluding particular paths.

An empty allow list permits any URL that does not match a skip rule. Removing the last allow rule restores that behavior. Existing rules files without an allow field continue to work as before.

For example, allow these sites:

^https://example.com/
^https://docs.example.org/

Then add ^https://docs\.example\.org/private/ as a skip rule to exclude private paths on the documentation site.

Allow rules apply to API submissions and browser imports as well as automatic browser capture. Editing rules does not immediately remove saved documents. Running nogon reindex removes documents excluded by the current allow or skip rules, except documents carrying an explicit manual override. In multiple user mode, reindexing uses each document owner’s rules; public documents use the instance rules.

Skip rules

Skip rules prevent matching URLs from being added to the index. When a URL matches any pattern in the skip list, Nogon silently discards the document during indexing and during nogon reindex.

Manual indexing in the browser extension overrides allow and skip rules for that submission and saves the choice with the document, so the page survives nogon reindex. Automatic submissions still respect the rules.

From the command line, use --ignore-rules with nogon index or any nogon import subcommand to save the same explicit override with each submitted document. For an already indexed URL, combine nogon index --ignore-rules with --force so that the document is submitted again. The override does not bypass sensitive content checks, robots rules, or crawl and file selection filters.

API clients can set metadata.ignore_skip_rules to the boolean true in JSON documents submitted to /api/add. For /api/add_pdf, it belongs inside document.metadata; for /api/batch, each add operation has its own metadata. The override is stored with the document and preserved in exported documents.

The override applies only to URL allow and skip rules. The metadata field keeps its existing name for compatibility. Authentication, ownership, sensitive content checks, and the exclusion of Nogon’s own URLs still apply. Accepted updates preserve an existing override unless they explicitly submit metadata.ignore_skip_rules: false. Explicit deletion still removes the document.

Use cases: block ad networks, login pages, cookie-consent walls, or any site you never want in your search results.

^https://ads.example.com
^https?://(login|mail).example.com/
.*?utm_source=

Priority rules

Priority rules push matching documents to the top of search results regardless of their relevance score. Nogon applies a large score boost to any document whose URL matches a priority pattern.

Use cases: always surface your personal wiki, your company’s internal documentation, or a trusted source before other results.

^https://wiki.example.com/
^https://docs.example.com/

Versioning rules

Versioning rules tell Nogon to track changes to a document every time it is re-indexed. When a URL matches a versioning pattern and the document content differs from the previously indexed version, Nogon stores diff match patch records for the HTML and text changes in the database.

Use cases: monitor pages for edits (privacy policies, documentation, news articles), build a personal changelog of sites you follow, or audit when a trusted resource last changed.

^https://example.com/privacy-policy$
^https://docs.example.com/

Viewing stored versions

Stored differences can reconstruct earlier document versions. Version history is available only after Nogon has recorded at least one change for that document.

Aliases

Aliases are query-time shortcuts: before Nogon executes a search, it replaces any alias keyword with its expanded form.

{
  "gh": "domain:github.com",
  "local": "type:file",
  "work": "domain:(internal.example.com|jira.example.com)"
}

With the work alias above, searching for work deployment is equivalent to domain:(internal.example.com|jira.example.com) deployment.

Pattern syntax

Allow, skip, priority, and versioning rules are matched against the full URL (including scheme, host, path, and query string). A few important details:

  • Patterns follow Go regular expression syntax. Look-ahead and look-behind are not supported.
  • Anchoring must include the scheme: ^https://foo.com or ^https?://(login|mail)\. are valid; ^foo.com is not.
  • The URL hash is stripped before matching (https://foo.com/#section becomes https://foo.com/).
  • Query-string parameters are not reordered; only utm_* parameters are stripped.
  • A trailing $ anchor will not match URLs that have a query string: /login$ does not match https://foo.com/login?auth=1.

Saving a skip rule does not delete existing documents by default. Optional deletion applies the rule to documents already in the index. Cancelling deletion leaves both the saved rule and the existing documents intact.

Storage

In single-user mode (user handling disabled), rules are saved to and read from rules.json in the configured data directory. Changes made through the web interface or API update this file directly.

In multi-user mode (user handling enabled), each user has a private copy of their rules stored in the database. Changes affect only the authenticated user and do not touch rules.json. See User Handling for details.