Lewis[1] is a river-style RSS/Atom feed reader for Lilush Shell. It frames feeds as a flowing stream rather than an inbox: items age, fade, and pass downstream into an archive. There is no concept of "unread" in Lewis. Items either sit in the river by virtue of being fresh, or they've drifted past the bend and live in the archive.
Lewis design is inspired by the The Last Quiet Thing post and ideas of Current RSS reader of the same author.
From Lilush Shell
lilpack install lewis
Bind the mode to an F-key in ~/.config/lilush/modes.json:
{ "F5": "lewis" }
The first time the mode opens it creates the
~/.local/share/lilush/lewis.mneme MNEME db. Add a few feeds and refresh:
add https://feeds.bbci.co.uk/news/rss.xml
add https://www.theguardian.com/world/rss -c news
add https://lobste.rs/rss
refresh all
flow
All commands are entered at the lewis mode prompt. Tab completion is wired for command names and feed ids.
| Command | Args | Description |
|---|---|---|
add | <url> [-c category] [--half-life DUR] | Subscribe to a feed |
rm | <N\|feed_id> | Unsubscribe and delete that feed's entries |
ls | [-c category] | List subscribed feeds (markdown table with order ids) |
refresh | [<N\|feed_id>\|all] | Conditional GET; ingest new entries |
flow | — | Open the river view (alt-screen) |
feed | <N\|feed_id> | Per-feed Beyond-style list view (alt-screen) |
read | <N\|feed_id> <entry_id> | Open one entry in the article reader |
search | <query> [-f N\|feed_id] | BM25 full-text search |
pin | <N\|feed_id> <entry_id> [--resurface DUR] | Pin (auto-saves) |
unpin | <N\|feed_id> <entry_id> | Unpin (does not clear saved) |
save | <N\|feed_id> <entry_id> | Mark saved (preserved against eviction) |
unsave | <N\|feed_id> <entry_id> | Clear saved flag (no-op if pinned) |
saved | — | List saved entries |
pins | — | List pinned entries |
drown | <N\|feed_id> <entry_id> | Drop the body and hide from the river — metadata stays in drowned so surface can recover it; the entry is now eligible for purge |
surface | <N> | Restore a drowned entry's visibility by its drowned-list ordinal (body stays gone) |
drowned | — | List drowned entries (numbered; feeds the ordinal surface takes) |
import | <path> | Import an OPML 2.0 subscriptions file |
export | <path> | Export subscriptions to OPML |
half-life | <N\|feed_id> <DUR\|auto> | Set per-feed half-life or revert to inferred |
cap | <N\|feed_id> <int\|none> | Set per-feed cap on visible river slots above the bend |
purge | — | Evict expired entries (respects pins / saved) |
similar | — | Reserved for CALM (see Planned) |
embed | — | Reserved for CALM (see Planned) |
DUR accepts 1h, 2d, 1w, 30d. A bare integer is seconds.
N is the order-id printed by ls (or by drowned for surface).
Order ids are 1-based positions in the canonical feed list (sorted by
added_ts asc) and re-number whenever a feed is added or removed —
they're a convenient handle, not a stable identifier. The 16-hex
feed_id form is still accepted everywhere a <N\|feed_id> is taken,
so scripts and copy-pastes from the feed_id column keep working.
The mode prompt completes by token position:
| Position | Completes from |
|---|---|
| Token 1 | command names (prefix-suffix on the typed letters) |
feed/rm/refresh/read/pin/unpin/save/unsave/drown/half-life/cap/similar/embed token 2 | feed match — see below |
surface token 2 | global drowned-list ordinal (display: N — [Feed Name] Title) |
read/pin/unpin/save/unsave/drown/similar token 3 | entry_id from the feed in token 2 |
half-life <fid> token 3 | auto / 1h / 6h / 1d / 3d / 1w / 30d / 90d |
cap <fid> token 3 | 5 / 10 / 20 / 50 / none |
refresh/embed token 2 | feed matches plus the literal all |
-c <category> value | existing categories pulled from feeds |
-f <feed_id> value | feed match (same matcher as token 2) |
--half-life/--resurface value | duration suggestions (no auto) |
The matcher tries, in order:
raw feed_id prefix on what you typed,
order-id prefix when the input is purely numeric,
case-insensitive substring against the feed title or URL.
The completion menu shows entries as N — <Feed title>. Accepting one
injects only the order-id N into the line — the dispatcher resolves
it back to the canonical feed_id at run time. So
feed bbc<TAB>
becomes
feed 4
once you accept the BBC News entry. Entry_ids still match the same way against entry titles and are injected verbatim, scoped to whichever feed is in token 2.
flow opens the river view in alt-screen. The river is a centered
column that adapts to terminal width — pager-style, capped at 100
cells, with the rest of the terminal left as padding.
Surface, Dim, and Faint tiers all share the same Title / Source /
Summary layout — title on its own line, source on the next line in a
recessed style, then the wrapped summary. Surface scales the title
through the big preset and clips it to fit the river column,
appending an ellipsis at the last word boundary that fits (the
Surface title is single-line by design — long titles read in full
once you press Enter). Dim drops the scaling; Faint runs the title
and summary through a braille mask whose hardness rises with age.
Beyond is the only one-liner.
[▶] Title
Source
Summary line one, wrapped to fit the column,
capped at three lines with an ellipsis if longer.
The leftmost two cells carry the marker:
| Glyph | Meaning |
|---|---|
▶ | Currently selected (anchored ~1/3 down the viewport) |
• | Pinned |
· | Saved (but not pinned) |
| Plain |
Age is conveyed by the title's color, not by an explicit time column:
title color fades continuously by visibility — fresh items land on
p.fg8 bold; old items ease into p.fg3. The summary sits one tier
behind the title (p.fg4 → p.fg2) so the hierarchy reads at a
glance. Beyond entries carry an explicit [YYYY-MM-DD] date stamp
since their visibility is too low for the gradient to be informative.
Each entry is assigned a tier from its visibility. Tiers map to render treatments:
| Tier | Visibility | Render |
|---|---|---|
| Surface | ≥ 0.7 | Title / Source / Summary, full color, scaled bold title |
| Dim | 0.4 .. 0.7 | Title / Source / Summary, normal-size bold title, slightly cooler |
| Faint | 0.15 .. 0.4 | Title / Source / Summary; title and summary braille-masked, source in mask anchor color |
| Beyond | < 0.15 | 1-line [YYYY-MM-DD] Title Source |
The river is sorted by visibility descending (with effective_pub_ts
as the tiebreaker), so tier transitions are monotone: Surface → Dim →
Faint → bend → Beyond. The bend is a row of ~ in the gutter color,
emitted once between the last river tier and the archive.
The Faint band has a soft entry: the last two Dim items above it pull
their color toward the mask anchor (no braille noise yet), then mask
hardness ramps from 5 at the top of Faint to 55 at the bottom.
The mask anchor also lerps from p.fg2 (warmer) to p.fg1 (colder)
across the band — even within the masked region, depth reads as a
gradient instead of a flat block.
To keep the river short, the band is capped at 3 Faint entries. Anything that would have been the 4th Faint entry or beyond renders in Beyond format instead and lives below the bend.
An entry's visibility depends only on:
now
its effective_pub_ts
its feed's half_life_seconds
Refreshing some other feed can add competing entries, but it does not change the visibility of entries that were already present. To preserve that invariant, Lewis selects river candidates from the active feed set first, computes visibility, sorts by river order, and only then applies the final river limit.
For each entry:
pub_ts = the feed-provided publish/update timestamp, or ingest time when missing
effective_pub_ts = pub_ts normally
effective_pub_ts = max(pub_ts, pin.last_surface_ts) when pinned
pub_ts is stable once the entry exists in the store: re-ingesting an
already-known entry may update title/summary/content, but it does not
change pub_ts and therefore does not make the entry fresh again.
Default half-life is 24 hours:
default_half_life = 86400 seconds
When half-life inference is enabled for a feed, Lewis tracks recent inter-post intervals and computes:
inferred_half_life = mean(post_intervals) * meta.half_life_multiplier
= mean(post_intervals) * 5.0 by default
clamped to [1h, 90d]
The interval history is not limited to entries that arrive in the same
refresh. Lewis also remembers the most recent seen pub_ts for each
feed and, when a later refresh brings a newer entry, records the gap
between that remembered timestamp and the earliest newly seen one.
That makes inference approximate publishing cadence rather than refresh
cadence: a feed that publishes one item per refresh still accumulates
interval evidence over time.
A manual override disables inference for that feed:
half-life <feed> <DUR> => set half_life_seconds and freeze it
half-life <feed> auto => re-enable inference
Manual values are clamped to the same [1h, 90d] range.
Lewis uses exponential decay:
visibility = 0.5 ^ ((now - effective_pub_ts) / half_life_seconds)
At age 0, visibility is 1.0. After one half-life it is 0.5; after
two half-lives it is 0.25; after three it is 0.125.
| Tier | Visibility | Approx age |
|---|---|---|
| Surface | >= 0.7 | < 0.515 half-lives |
| Dim | 0.4 .. 0.7 | 0.515 .. 1.322 half-lives |
| Faint | 0.15 .. 0.4 | 1.322 .. 2.737 half-lives |
| Beyond | < 0.15 | > 2.737 half-lives |
Pinned entries still use the same raw visibility formula, but river
ordering gives them a sort floor of 0.4, so they never sink below
Dim while pinned.
For the main river view, Lewis applies this pipeline:
Choose candidate feeds from the active stream, or all feeds.
Remove drowned entries.
Compute each entry's visibility.
Sort by visibility descending, with pinned entries floored at 0.4 for sorting.
Tie-break by effective_pub_ts descending.
Deduplicate by normalized link.
Apply the per-source diversity pass.
Apply per-feed river_cap above the bend.
Limit the Faint band to 3 entries.
Render Beyond below the bend.
That ordering matters. In particular, stream filtering happens before limiting, and visibility sorting happens before the final river cut, so slow high-visibility feeds are not starved just because faster feeds were refreshed more recently.
Streams are adaptive terciles over the current subscription set's half-life distribution:
fast = the third of feeds with the shortest half-lifes
medium = the middle third
slow = the third with the longest half-lifes
Membership is recomputed on entry to the river and after full refreshes, because ingest can change inferred half-lifes. With fewer than 3 feeds, streams are disabled.
R inside the river fetches only the active stream when a stream filter
is active. Press a first to drop back to all feeds, then R, if you
want a full sweep across every subscription.
r is store-only: it never fetches and therefore never changes
half-lifes. It only re-reads from the store and re-renders the current
view against the current time.
| Action | Network? | Scope | Can change half-lives? | Can add entries? |
|---|---|---|---|---|
r | no | current view | no | no |
R in stream | yes | active stream feeds | yes, for fetched feeds | yes |
R in all feeds | yes | all feeds | yes | yes |
refresh <feed> | yes | one feed | yes | yes |
refresh all | yes | all feeds | yes | yes |
A 304 Not Modified response is a successful check, not an error. It:
updates fetch timestamps
updates validators when present
clears prior fetch error state
does not add entries
does not change half-life by itself
Visibility and retention are separate concerns. An entry can be well past the bend and still remain stored for a long time.
Purge horizon is based on the longest feed half-life currently present:
purge_horizon = max(feed.half_life_seconds) * meta.purge_age_multiplier
= max(feed.half_life_seconds) * 20 by default
Pinned entries and saved entries are preserved. Pins imply saved. Unsaved drowned entries remain eligible for purge too; drowning hides an entry from river/feed/search views, but it is not a retention mechanism.
After the visibility sort, a soft per-feed cap reorders the view: at most 3 entries from any one feed appear within any sliding window of 10 consecutive positions. Excess entries from a saturated feed are deferred and re-emitted once their feed drops back under the cap, so a high-volume feed (Hacker News, Lobsters) can't crowd the surface tier with a single source. Visibility ordering within a feed is preserved — we only reorder across feeds. Pinned entries bypass the cap so a pin burst always surfaces together.
river_capcap <feed_id> <N> sets a hard ceiling on how many of a feed's
entries can occupy slots above the bend (Surface + Dim + Faint).
Set this for high-volume feeds when the diversity cap alone leaves
too many of their entries visible — for example, cap hacker-news 5
keeps at most 5 HN entries in the visible river regardless of how
many fresh ones the feed has. cap <feed_id> none removes the cap.
When a capped feed has more above-bend candidates than its cap,
Lewis picks the survivors via visibility-weighted random sampling
(A-Res with weight = visibility): fresher entries are more likely
to survive, but every candidate has a nonzero chance, so a different
subset rotates in on each light refresh (r in the river view).
Pinned entries bypass the cap. The Beyond tier and the per-feed /
search views are not affected — the cap only governs the visible
river.
The river groups every feed together by default. With many subscriptions of mixed cadence — daily news, weekly newsletters, hourly aggregators — fast feeds drown out slow ones in the visible band. Streams carve the river into three speed classes:
| Stream | Bucket |
|---|---|
fast | The third of feeds with the shortest half-lifes |
medium | The middle third |
slow | The third with the longest half-lifes |
Streams are computed automatically as adaptive terciles over the
current subscription's half-life distribution — fast is whatever
"fast" means in your set. The partition is recomputed on entry to
the view and on every R (full refresh, half-lifes can drift after
ingest). Fewer than 3 feeds disables streams (Tab flashes a
reminder, the river falls back to all feeds).
The river opens on the fast stream by default. Press Tab to cycle through the streams:
fast → medium → slow → fast
Press a to drop the filter and show all feeds; press Tab again
to drop back into the fast stream. The active stream name is shown
on the left of the status line; "all feeds" is implied when the
prefix is absent. Light refresh (r), pin (p), save (s), drown
(d), and selection moves all preserve the active stream.
Pressing R while a stream is active only fetches the feeds in that
stream — switch to "all feeds" with a first if you want a full
sweep. Light r is store-only and never fetches, so its scope is
purely the current view regardless.
Pinning sets last_surface_ts = pub_ts and auto-saves the entry.
The timeline sorted set is re-scored to the pin's effective publish
time, so the entry rejoins the river at the top. While pinned, an
entry's "sort floor" is 0.4 — even if its raw visibility decays,
it never sinks below the Dim tier visually.
A pinned entry resurfaces every resurface_seconds (default 7 days,
overridable per-pin via --resurface). The bump fires on entry to
the mode and once per flow open.
Saving without pinning preserves the entry against eviction but does
not change its position in the river. Pin implies save; unpin does
not clear saved (call unsave separately).
drown is the mirror of pin: instead of bumping an entry to the
top, it deletes the body. The summary, feed_content, and any cached
fetched_content are dropped, the FT index entry is removed, any
active pin is cleared, and saved is set to false. What stays is
metadata — title, link, comments link, author, dates, tags — enough
to remember the entry existed. The metaphor: you can't revive a
dead entry, but you can still do an autopsy.
Drowned entries are absent from the river, the per-feed feed <fid>
listing, and search. They show up only in drowned and via
tab-complete on surface. surface restores visibility but does not
restore the body — opening the entry afterwards falls back to the
article reader stub that points at the original link.
Drowning does not preserve the entry against purge. Drowned
entries are removed from the active river timeline immediately, but
unsaved drowned entries still remain eligible for purge by age. If you
want to keep something forever, save it explicitly.
The cursor anchors roughly one-third down the viewport — j and k
advance or retreat which entry holds that position. The river itself
is scrolled and re-painted around the anchor; selection follows. Near
the start of the river the selection rides at row 1, and near the end
it sinks past the anchor toward the bottom of the viewport (standard
scroll-off in both directions).
| Key | Action |
|---|---|
j / ↓ | Next entry becomes the top |
k / ↑ | Previous entry becomes the top |
g / Home | Jump to entry 1 |
G / End | Jump to the first entry past the bend (or last if no bend) |
J / PgDn | Page step — advance ~one viewport |
K / PgUp | Page step backwards |
r | Light refresh — recompute the river (no fetch) |
R | Full refresh — refetch feeds in the active stream (or all when on "all feeds") |
Tab | Cycle stream filter: fast → medium → slow → fast |
a | Show all feeds (drop stream filter) |
Enter | Open the top entry in the article reader |
f | Open the top entry's URL in the system browser |
c | Open the top entry's comments URL when present (RSS <comments> / Atom rel="replies") |
p | Pin/unpin the top entry |
s | Save/unsave the top entry |
d | Drown the top entry (removes it from the river; use surface <fid> <eid> to restore) |
q / Esc | Leave the view |
Opening in a browser uses the registered desktop handler for
https/http (the same mechanism xdg-open consults). If no
handler is registered, the status line flashes "no browser handler
registered".
Several transitions in the river view animate rather than snap:
Scroll — j/k/J/K/g/G slide the viewport one or more
lines per frame until the new selected entry rests at the anchor
row. During motion every viewport row outside the selected entry's
band is masked with braille noise, with hardness ramping from soft
adjacent to the selection to hard at the viewport edges (warm anchor
above, cold below). The selected entry itself stays crisp, so the
eye locks onto the focal entry while the rest of the river blurs.
Single-line steps are not animated; page and edge jumps are capped
to ~300 ms total regardless of distance.
Drown — pressing d keeps the just-drowned entry on screen as
a "ghost" that descends one row per frame, dissolving into braille
noise and bleeding toward the cold mask anchor as it sinks. The
river behind it is already the post-drown view, so the entry
visibly leaves the stream.
Open — pressing Enter keeps the selected entry crisp while
every other viewport row dissolves into spaces, then the article
reader takes over. The fetch for the article body runs concurrently
with the dissolve so the pager opens with cached content.
Light refresh — pressing r dissolves every row outside the
selected entry's band into spaces (~180 ms), the river is re-read
from the store and re-laid-out against the current time, then the
new river materialises by reversing the dissolve (~180 ms). No
network activity; the visible reorder is purely visibility decay.
Stream cycle — pressing Tab reuses the light-refresh dissolve:
the river fades out around the current selection, the filter
advances to the next stream, the records are re-read with the new
filter applied, and the (likely different) subset materialises in
place.
Full refresh — pressing R plays a cold mask wave that
descends row-by-row over the river before alt-screen tears down for
the network fetch (~300 ms). Per-feed status lines print at the
prompt during the fetch. When the river returns, a clean front
rises from the bottom of the viewport, peeling the mask away over
~420 ms, revealing the post-fetch river.
A keystroke during any animation cuts it short and dispatches
immediately. Set LEWIS_REDUCED_MOTION=1 (or true/yes/on) to
disable all animations and restore instant transitions; the variable
is read fresh on each keystroke, so it can be flipped without
restarting the mode.
feed <feed_id> and search <query> both open an interactive
alt-screen list. Each row uses the river's Beyond format
([YYYY-MM-DD] Title); the per-feed view drops the source column
since every row shares the feed, the search view keeps it so you
can see which source produced each hit. The heading line carries
the feed title (or the search query) over a ~-row separator.
| Key | Action |
|---|---|
j / ↓ | Next entry |
k / ↑ | Previous entry |
g / Home | Jump to first entry |
G / End | Jump to last entry |
J / PgDn | Page step forward |
K / PgUp | Page step backward |
Enter | Open the selected entry in the article reader |
f | Open the selected entry's URL in the system browser |
c | Open the selected entry's comments URL when present |
p | Pin/unpin the selected entry |
s | Save/unsave the selected entry |
r | Refresh just this feed (per-feed view only) |
q / Esc | Leave the view |
Both views fall back to the original markdown-archive document
through the shell pager when run non-interactively (e.g.
lilush -c "feed <fid>" or lilush -c "search foo"), so scripted
output keeps working unchanged.
read <feed_id> <entry_id> (and pressing Enter on the top of the
river) opens the entry in the shell pager. The body is selected in
this order:
feed_content — the full body shipped by the feed (content:encoded
in RSS, <content> in Atom), converted to markdown via
markdown.from_html at ingest time.
fetched_content — for entries whose feed shipped no full body,
lewis.fetch.article pulls the linked HTML on first read and
markdown.from_html converts it; the markdown is cached on the
entry record so the next read is instant.
Stub — when the feed shipped no full content and the entry has
no link (or extraction failed), a single line nudges toward the
original URL.
The cached extraction is permanent for the life of the entry. Entries
ingested with an older extractor keep their previous body until the
river evicts them via half-life; re-adding the feed (or deleting the
entry rows) forces a re-ingest with the current pipeline.
Document shape handed to the pager:
# <Entry title>
*<feed.title> · <author> · <pub_ts>*
[Original link](<entry.link>)
[Comments](<entry.comments_link>) <!-- only when the feed shipped a comments URL -->
---
<body>
This section summarizes the same rules described earlier in Age, visibility, half-life, and retention. The formulas below are the implementation details behind that model.
visibility(entry, now) = 0.5 ^ ((now - effective_pub_ts) / half_life_seconds)
effective_pub_ts is pub_ts for normal entries, and
max(pub_ts, pin.last_surface_ts) for pinned entries.
Tier thresholds (lewis.age.tier): 0.7 / 0.4 / 0.15. Pinned
entries have a tier floor of Dim regardless of raw visibility, and a
sort floor of 0.4 in the river ordering.
Per feed, Lewis tracks the last 10 inter-post intervals. The inferred half-life is:
half_life_seconds = mean(post_intervals) * meta.half_life_multiplier (default 5.0)
clamped to [1h, 90d]
Intervals come from both:
gaps between multiple newly seen entries in the same refresh, and
gaps across refreshes via the feed's remembered latest seen pub_ts
So a feed that produces one post per refresh still accumulates interval history over time.
Recomputed on every successful refresh that yields ≥ 1 new entry with a valid publish timestamp.
half-life <feed_id> <DUR> overrides the value and freezes it
(half_life_inferred = false); half-life <feed_id> auto reverts to
inferred. Manual overrides are clamped to the same [1h, 90d] envelope
as inference, so a fat-finger value (30s, 200d) lands at the
nearest bound rather than a degenerate visibility regime.
purge removes entries older than the current purge horizon, where:
purge_horizon = max over feeds of (half_life * meta.purge_age_multiplier) (default 20.0)
Pinned entries and entries with saved = true are exempt. Unsaved
drowned entries are also eligible for purge. The operation also clears
FT index entries.
A daily feed with half_life ≈ 24h keeps about 20 days of history;
a weekly feed with half_life ≈ 7d keeps about 4.5 months.
The single entry point to MNEME. Every other lewis module talks to this layer; nothing else opens the database directly.
local store = require("lewis.store")
local s, err = store.open() -- defaults to ~/.local/share/lilush/lewis.mneme
Public methods on the returned handle:
| Method | Notes |
|---|---|
s:meta() | Read the meta keyspace (schema version, defaults, multipliers) |
s:subscribe(url, opts?) | opts = { category, half_life } → feed_id |
s:unsubscribe(feed_id) | |
s:list_feeds(opts?) | opts = { category } |
s:get_feed(feed_id) / s:get_entry(feed_id, entry_id) | |
s:ingest(feed_id, parsed_feed) | Atomic batch; returns {new, updated} |
s:record_fetch_error(feed_id, msg) | Bumps consecutive_errors |
s:update_fetch_meta(feed_id, { etag, last_modified }) | |
s:update_entry_content(feed_id, entry_id, fetched) | Re-indexes FT |
s:river(opts?) | opts = { limit = 200, before_ts } |
s:archive_feed(feed_id, opts?) | Plain chronological |
s:search(query, opts?) | opts = { top_k = 20, feed_id } |
s:pin(feed_id, entry_id, opts?) | Auto-saves; opts = { resurface } |
s:unpin(feed_id, entry_id) | Does not clear saved |
s:save(feed_id, entry_id) / s:unsave(feed_id, entry_id) | |
s:list_pins() | Sorted by last_surface_ts desc |
s:list_saved(opts?) | Saved (and not drowned); opts = { feed_id }; sorted by ingested_ts desc |
s:tick_pins() | Bumps eligible pins; returns count |
s:drown(feed_id, entry_id) | Drops the body, removes the FT entry, clears any pin, and clears saved |
s:surface(feed_id, entry_id) | Restores visibility; the body is not restored |
s:list_drowned(opts?) | opts = { feed_id }; sorted by ingested_ts desc |
s:purge() | Returns { evicted = N } |
s:close() |
| Keyspace | Type | Purpose |
|---|---|---|
feeds | KV | Subscription metadata (one record per feed_id) |
entries | KV | Entry payloads keyed by feed_id:entry_id |
timeline | sorted set (all) | Scored by effective_pub_ts |
pins | KV | Pin records keyed by feed_id:entry_id |
search | FT | BM25 index over title \n summary \n feed_content |
meta | KV | Schema version, defaults, multipliers, last purge time |
Defaults written into meta on first open:
| Key | Default |
|---|---|
default_half_life | 86400 (24h) |
default_resurface | 604800 (7d) |
half_life_multiplier | 5.0 |
purge_age_multiplier | 20.0 |
feed_id = sha256(canonical_url)[:16] (hex). Trailing slashes and
scheme are normalised before hashing, so https://x.com/rss and
https://x.com/rss/ produce the same id.
entry_id is taken from the feed's <guid> / <id> when present
(sanitised), otherwise from sha256(link || title || pub_ts)[:16].
Re-ingesting the same entry_id updates mutable fields (title,
summary, content) but never pub_ts — a publisher editing a
post does not bring it back to the top of the river.
lewis.fetch.feed(record, opts?) and lewis.fetch.feeds(records, opts?)
implement conditional GET, gzip decoding, redirect following, and
per-host single-flight gating.
Defaults:
| Setting | Value |
|---|---|
| Per-feed timeout | 30 s |
| Per-article timeout | 15 s |
| Redirects | up to 5 |
| Per-host concurrency | 1 (single-flight queue) |
Accept-Encoding | always gzip |
Conditional GET sends If-None-Match (etag) and If-Modified-Since
based on the stored fetch meta. A 304 returns
(nil, headers, "not_modified"); the caller leaves entries untouched
and updates only fetch meta.
lewis.fetch.article(url, opts?) shares the same per-host gate, so
a feed fetch and a concurrent article fetch on the same host
serialise.
| Module | Public surface |
|---|---|
lewis.xml | Streaming pull-style XML parser. parser(text, opts?) and parser_from_iterator(reader, opts?) yield decl / start / end / text / cdata events. UTF-8, Latin-1, Windows-1252 transcoding. Hard limits on depth (64), name length (256), attribute value length (64 KB), text chunk (1 MB). Rejects DOCTYPE external entities. |
lewis.feed | parse(xml_text) → { format, feed, entries }. Detects RSS 2.0, Atom 1.0, RDF/RSS 1.0. summary is HTML-stripped to plain text and trimmed to 500 chars; feed_content is converted to markdown via markdown.from_html and capped at 256 KB. Falls back to ingested_ts when the feed's pub_ts is unparseable. |
lewis.dates | RFC 822 + RFC 3339 parser with the usual real-world deviations (missing tz, two-digit years, trailing junk). |
lewis.opml | parse(text) → { title, outlines = [...] }; serialize(tree) → OPML 2.0 string. Folders nest one level deep on export (categories collapse). |
lewis.age | visibility, tier, effective_pub_ts, infer_half_life, should_resurface. Pure, no I/O. |
lewis.streams | compute(feeds) partitions feeds into fast/medium/slow buckets via adaptive terciles over half_life_seconds; next_filter(current) cycles nil → fast → medium → slow → nil. Pure, no I/O. |
lewis.fetch | feed, feeds, article. See above. |
lewis.store | The MNEME wrapper. See above. |
lewis.view.river | render_buffer(records, opts?) (pure) and run(opts) (the interactive driver). |
lewis.view.list | Interactive Beyond-style list view backing feed and search. render_buffer(entries, opts?) (pure) and run(opts) (the interactive driver). |
lewis.view.archive | Markdown-rendered chronological list. Non-interactive fallback for feed and search. |
lewis.mode.lewis | The shell mode object. Wires commands.dispatch to the prompt and exposes the runtime context (store handle, fetch loop, pager). |
lewis.mode.lewis.article | Article reader: builds the markdown document with the feed_content → fetched_content → stub fallback chain. |
lewis.theme | Section builder for theme.subscribe("lewis"). Carries the bend / pin / save / selection / summary roles; the rest of the river uses the live palette directly for continuous lerps. |
lewis.completion.slash | Position-aware completion driver (commands, feed_id, entry_id, flag values, duration suggestions). Smart-matches feed_id by prefix or by case-insensitive title substring. |
lewis.completion.source.commands / feeds / entries | Data sources backing the driver. feeds exposes list(), list_full(), categories(); entries exposes list_for_feed(fid). |
Lewis registers a theme section through the lilpack manifest. The
section subscribes to the active palette and is rebuilt on
theme.set(...).
TSS roles wired in this build:
| Path | Purpose |
|---|---|
lewis.info, lewis.error, lewis.ok, lewis.warning | Prompt-side status messages |
lewis.feed_id, lewis.feed_title, lewis.category | ls, pins, listings |
lewis.river.surface.excerpt | Summary-line tint |
lewis.river.source | Surface-tier source line, italic and recessed |
lewis.river.status | Bottom status line, recessed |
lewis.river.bend | The ~ row between river and archive |
lewis.river.pin | • marker |
lewis.river.save | · marker |
lewis.river.selection | ▶ marker |
prompts.lewis.* | Mode prompt segments |
Title, time, source, summary, mask anchor, and beyond colors are
derived directly from the live palette (fg1 … fg8, text,
warm1, fg2, fg5) inside the renderer, with continuous RGB
lerps by visibility. There is no per-tier static role — themes
adjust the palette and the river follows.
Lewis already reserves a commands slot for embed [<feed_id>|all]
and similar <feed_id> <entry_id>. Both will be wired to a CALM
embedding model loaded by lilush:
embed walks entries and writes a vector per entry into a separate
MNEME embeddings keyspace, keyed identically to the FT index
(feed_id:entry_id). The vector source is title || "\n" || summary.
The keyspace is gated on the lilush build path (lilu does not load
CALM), so the vector layer is a no-op there.
similar looks up the entry's vector and returns top-k cosine
neighbours, presented through the archive view.
The expected payoff is duplicate clustering (the same story from two outlets ends up adjacent in vector space) and a better "more like this" navigation than BM25 over short summaries can offer.
Dynamic river flow (lewisd)
Cross-feed deduplication (the same article from two sources). The CALM embedding layer is the most likely vehicle.
Per-category half-life defaults (currently per-feed only).