🏭 Big public data on a small server

Public data is free, and it is enormous. Medical Price & Bill Coach, my hospital price tool, reads the price files US hospitals are required to publish: single CSVs (tables saved as plain text, commas between the fields) of 500 MB, JSON files (data written as plain text) near 1 GB, and a 1.8 GB zip that unpacks to a 43 GB CSV. Its server had about 2 GB of memory to spare, shared with other apps.

This lesson grows the build.mjs idea from Build static-first into a pipeline that runs on your server from Deploy behind one front door, refreshes itself every week and can't half-break the live site. I'm a data person, not a 10× engineer, and nearly every rule below broke on me first.

Start from the questions, not the file. My price tool answers price questions for about 100 common procedures, so its allowlist (the short list of things your app answers questions about) holds their codes, and every other row is dropped as it streams past. That turns about 60 GB of weekly downloads into about 210 MB it actually uses. The 43 GB CSV took 22 minutes, read straight out of its zip, and was never unpacked.

Standard-library Python, one script per job

For big files I use Python, because its standard library (the modules every Python install includes) downloads files and reads CSV, zip and gzip (compressed) files, while Node has no CSV or zip reader built in. The pipelines behind Ticket Heat and Who Owns Phoenix use nothing else: the machine I built them on had neither pandas nor requests, two popular add-ons, so they run anywhere.

Python isn't part of Set up your workshop. Install Python 3 from python.org (on Windows, tick Add python.exe to PATH on the installer's first screen) and reopen VS Code, your code editor. In its terminal, py --version (Mac: python3 --version) should print Python 3. and a version number. On Windows the command is py: plain python is often not recognized, or offers you the Microsoft Store. Prefer Node? Ask Claude for the same scripts in Node, and expect it to add one package, for zips.

Three habits keep reruns cheap. Cache every slow step (save the result to disk and reuse it): the raw pull of property records behind Who Owns Phoenix is about 113 MB, and with it and every geocoder answer cached (a geocoder turns addresses into map coordinates), a rerun takes seconds instead of hours of downloading. Give the script a --selfcheck flag (an option typed after the command) that tests its logic on tiny made-up files, offline. And give each job its own small script in a tools/ folder, safe to rerun at any time.

One rule file, and lookups out of the raw folder

Keep every rule in one file that everything reads. The List keeps its thresholds, rate tables, deadlines and the official revenue estimate in policy.json, and its pipeline, its calculations and its FAQ all read it. When my own docs drifted into calling the tax a sliding rate (it's a flat percentage), the numbers people saw stayed right, because they come from that file. A check fails once a deadline in it has passed, so it gets reviewed every year.

And keep lookup tables out of any folder your import reads in bulk, or they become fake records. The List's three (sales, assessed values and listing links) sit outside its raw-input folder on purpose and are joined in by name. A manifest, one file naming every source to import, avoids the trap by design.

Open your app's folder in VS Code and ask Claude Code, the AI assistant working in that folder:

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath: nothing runs yet. The allowlist is the small answer written down, a plain list like ["73721", "70553", "45378"], and the manifest is the only list of files the pipeline will ever read. Each entry's region decides which output file its rows land in, and .gitignore (the list of files git never stores) keeps downloads out of your repo.

[
  {"id": "HOSP-A", "region": "AZ", "url": "https://example.org/prices/hosp-a.csv"},
  {"id": "HOSP-B", "region": "AZ", "url": "https://example.org/download?id=2"},
  {"id": "HOSP-C", "region": "CA", "file": "raw/HOSP-C.csv"}
]

• CSV: loop over Python's csv.reader, keeping a row only if its code is on the allowlist.
• Zip: open the biggest CSV inside with zipfile and stream it from there. Unpacking first is how disks fill up.
• JSON: for item in ijson.items(f, 'some.path.item') instead of json.load(f). ijson is a package, not standard library: py -m pip install ijson (Mac: python3 -m pip install ijson). On the server, ask Claude to put it in a virtual environment (a private package folder for one project), because recent Ubuntu refuses installs outside one, and to run the job with that environment's Python.
• Downloads: shutil.copyfileobj(response, file) writes to disk a chunk at a time.

A process (a running program) killed, or a server frozen, on a big file usually means a .read(), json.load or pandas.read_csv on the whole thing.

Pin the types

Tools that guess a column's type guess "text" when it's empty. One hospital whose kept rows had no cash prices at all gave the price tool an all-empty column. DuckDB guessed text for that file, combining all the files turned the whole column into text, median() returned a string, and the price card crashed. So write numbers as numbers: the script below turns "$1,250.00" into 1250.0 and anything unreadable into null, never into text.

If you grow into Parquet, cast every number column in the query that writes each file (CAST(cash_price AS DOUBLE)), and read defensively:

SELECT median(TRY_CAST(cash_price AS DOUBLE))
FROM read_parquet('data/rates/**/*.parquet');

Now the script:

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath: a loop that holds one row at a time. Look for these parts in what Claude wrote; its names will differ:

def read_csv(raw):
    text = io.TextIOWrapper(raw, encoding='utf-8-sig', errors='replace', newline='')  # drops a BOM (an invisible file-start marker); step 3 explains this line
    yield from csv.reader(text)                         # quoted commas stay inside their field


def keep_rows(rows, src, allow):
    """Stream rows past, keeping only allowlisted codes. Returns (kept rows, rows read)."""
    header = [h.strip().lower() for h in next(rows)]
    cols = [src.get('code_col', 'code').lower(), src.get('price_col', 'price').lower()]
    for c in cols:
        if c not in header:
            raise ValueError(f'no {c!r} column; this file starts {header[:6]}')
    ci, pi = header.index(cols[0]), header.index(cols[1])   # by name, never by position
    kept, read = [], 0
    for row in rows:                                    # one row in memory at a time
        read += 1
        if len(row) > max(ci, pi) and row[ci].strip() in allow:
            kept.append([src['id'], row[ci].strip(), to_number(row[pi])])
    return kept, read

yield hands rows over one at a time, so nothing ever holds the whole file. A source whose columns have other names gets "code_col" and "price_col" in its manifest entry. Check it yourself in VS Code's terminal (PowerShell or Git Bash), in your app's folder; on a Mac, type python3 instead of py:

py tools/ingest.py --selfcheck
py tools/ingest.py

You'll see one line per source in your manifest, with your own names and counts. On a big file, keep Task Manager open (Ctrl+Shift+Esc; Mac: Activity Monitor): Python's memory should stay roughly level from start to finish. If it climbs with the file, something is loading it whole.

Files that are supposed to follow a standard rarely all do. When the price tool expanded to five more states, all of these turned up within about a week:

• Another layout: a "wide" CSV with one group of columns per insurer, where the standard's example is "tall", one row per price. Turn each wide row into one record per insurer.
• A byte-order mark at the start of JSON exports: encoding='utf-8-sig' drops it.
• Latin-1 text instead of UTF-8: errors='replace' turns an odd byte into � instead of ending the run.
• Gzip nobody asked for: servers compressing the download anyway.
• Stray control bytes (invisible characters numbered 0 to 31) that crashed the JSON parser: filter them out of the stream first.
• A broken certificate chain (the padlock's proof of identity) on a public file. The only exception I allow is a per-source insecure: true flag for public, non-sensitive files, never a global switch.

So ingest.py judges a file by its first bytes, never its name: PK means zip, 1F 8B means gzip, and <html means you saved a web page.

def open_rows(path):
    """Yield CSV rows from a plain, gzipped or zipped file, whatever its name says."""
    with open(path, 'rb') as f:
        head = f.read(64)
    if head.lstrip().lower().startswith((b'<!doctype html', b'<html')):
        raise ValueError('this "file" is a web page: open the link in a browser and find the real file')
    if head.startswith(b'PK'):                          # a zip, even with no .zip in the link
        with zipfile.ZipFile(path) as z:
            csvs = [m for m in z.infolist() if m.filename.lower().endswith('.csv')]
            if not csvs:
                raise ValueError('a zip with no .csv inside')
            with z.open(max(csvs, key=lambda m: m.file_size)) as raw:   # read inside the zip
                yield from read_csv(raw)
    elif head.startswith(b'\x1f\x8b'):                  # gzipped, asked for or not
        with gzip.open(path) as raw:
            yield from read_csv(raw)
    else:
        with open(path, 'rb') as raw:
            yield from read_csv(raw)

To peek at a link yourself, in Git Bash, on a Mac or on the server (replace the example addresses with yours):

curl -sL 'https://example.org/download?id=2' | head -c 8 | od -c
curl -sIL 'https://example.org/download.aspx?id=1' | grep -i '^location'

P K first means zip; { or [ means JSON. The second line prints each address a redirect passes through: store the last one. For a landing page, open it in a browser and find the real file link. Then, after every run, read the per-source lines and treat any zero, or a sudden drop, as a bug until you can explain it. Mark a source you know is empty with "known_empty": true so the zero list stays meaningful.

City portals, silent filters and blocked downloads

Many city open-data portals run CKAN, software with an API (a web address that answers programs with data) that accepts SQL. Ticket Heat's portal answered 403 (forbidden) until requests carried a normal browser's User-Agent, returned at most 32,000 rows per query, dropped the connection partway through long pulls, and kept each year in a separate table.

Ask Claude, replacing DATASET_URL with your dataset's page address: "On DATASET_URL find each year's resource id, then pull every one from the portal's CKAN SQL endpoint, 32,000 rows at a time with OFFSET, until a page comes back short. Retry each page three times, 2, 5 and 10 seconds apart, save each page as it arrives, and print rows per year." OFFSET skips the rows you already have; a short page means you've reached the end.

A filter that matches nothing never errors. Print counts per year and per category after every step, trim and uppercase text before comparing (s.strip().upper()), match patterns like LIKE '%CRASH%' (anything containing CRASH) instead of exact lists, and test the longer phrase first: NO INJUR before INJUR, or every no-injury crash counts as an injury.

If a site blocks your server, don't fight its bot protection. Across the two sites built on my price-tool code, about thirty public price files couldn't be fetched from the server: some hosts blocked the hosting company's whole address range, others showed bot checks or timed out. Fetched in a normal browser, every one came through. Note each file's source link and checksum (a fingerprint of its exact bytes: sha256sum FILE in Git Bash or on the server, shasum -a 256 FILE on a Mac, replacing FILE with its name) in a DELIVERY.md, copy the file into the pipeline's raw/ folder, and point its manifest entry at it: "file": "raw/HOSP-C.csv". Hand-fetched files never refresh themselves, so set a calendar reminder.

EXAMPLE — YOU ASK CLAUDE CODE

Check it yourself: in tools/ingest.py (Ctrl+F, Mac: Cmd+F, finds it), change utf-8-sig to utf-8, save, and run py tools/ingest.py --selfcheck. It should fail instead of printing selfcheck ok; mine ended (yours may word it differently):

Undo (Ctrl+Z, Mac: Cmd+Z), save and rerun: selfcheck ok. A check you've never seen fail may test nothing. Each time a new kind of file bites, add it as one more case.

Never write new data over the files visitors are reading. Build into a staging folder, data.build/, and only once every source has been read, move each finished file into data/ with one rename. Downloads work the same way: save to file.part, rename when complete.

def publish(by_region, regions, held, build, out):
    """Swap each region's finished file into the live folder: one rename each."""
    for region in regions:
        rows = by_region.get(region)
        if region in held or not rows:                  # a failed source holds its whole region back
            print(f'{region}: not published, so the live file stays as it was')
            continue
        tmp = os.path.join(build, region + '.json')
        with open(tmp, 'w', encoding='utf-8') as f:
            json.dump(rows, f, separators=(',', ':'))
        os.replace(tmp, os.path.join(out, region + '.json'))   # old file or new, never half
        print(f'{region}: {len(rows)} rows swapped in')
    dates = {f[:-5]: time.strftime('%Y-%m-%d', time.localtime(os.path.getmtime(os.path.join(out, f))))
             for f in sorted(os.listdir(out)) if f.endswith('.json') and f != 'regions.json'}
    with open(os.path.join(build, 'regions.json'), 'w') as f:
        json.dump(dates, f)                             # the page's menu: only regions with data
    os.replace(os.path.join(build, 'regions.json'), os.path.join(out, 'regions.json'))

The night the price tool's refresh died 2.5 hours in (step 5 has the story), nothing half-written reached the site. Here, a region with a failed source also keeps last week's file, and the log says why. The price tool takes the other trade-off, letting a failed source drop out until the next good run and catching it in the per-source lines; for a first pipeline, holding the region back is the safer default.

The site never waits on the job, either. Your page reads whatever files exist; its menu reads regions.json, written from the files that exist, never from config; and a region without a file says "No data for this area yet". The price tool's container starts only its web server and runs imports separately: if startup waited for an import, one broken hospital link would trap it in a crash-and-restart loop.

A new market is config, not code

Splitting output by region made growth cheap. The price tool went from Phoenix to all of Arizona with no code change (27 more hospitals in its config and a re-import), then added five more states the same way, each appearing on the site the moment its data landed, without a restart. Here, a new region is a few manifest entries plus py tools/ingest.py --region XX, which rebuilds that region alone.

Re-download only what changed

The script keeps those values in a small sidecar file beside each download (cache/HOSP-A.meta). Never use "the file already exists, so skip it": that freezes your data forever and never notices a hospital publishing new prices. A host that sends neither header gets downloaded every time; accept it. On the price tool's 50 GB server, disk was tighter than bandwidth, so big raw files are deleted after import and downloaded again next week.

def download(url, dest):
    """Fetch url into dest only if the server says it changed. Returns what happened."""
    req = urllib.request.Request(url, headers={'User-Agent': UA})
    if os.path.exists(dest) and os.path.exists(dest + '.meta'):
        for name, value in load_json(dest + '.meta').items():
            if value:
                req.add_header(name, value)             # "only if it changed since this"
    for wait in (2, 5, 10, 0):                          # connections drop: four tries in all
        try:
            with urllib.request.urlopen(req, timeout=60) as resp, open(dest + '.part', 'wb') as f:
                shutil.copyfileobj(resp, f)             # straight to disk, so memory stays flat
                seen = {'If-None-Match': resp.headers.get('ETag'),
                        'If-Modified-Since': resp.headers.get('Last-Modified')}
            break
        except urllib.error.HTTPError as e:
            if e.code == 304:
                return 'unchanged'                      # 304 Not Modified: our copy is current
            if e.code < 500 or not wait:
                raise                                   # a 403 or 404 won't fix itself
        except OSError:
            if not wait:
                raise
        time.sleep(wait)
    os.replace(dest + '.part', dest)                    # only a complete download replaces the old copy
    with open(dest + '.meta', 'w') as f:
        json.dump(seen, f)
    return 'downloaded'

Database imports: prove a rerun writes nothing

If your pipeline writes into a database instead of rebuilding files, give every record a unique key (its source plus the source's own ID) and a hash of its cleaned content with the keys sorted, then insert new records, update changed ones and skip the rest. Prove it by running twice. On an unlisted public-health map of mine, the first real run pulled 365 FDA recalls and the second wrote zero, all unchanged; the one rejected record, with an empty recall number, showed up in an admin error log with its reason instead of vanishing.

Test the exact address you send, too. Some APIs read + as a space. On that map I typed +TO+ into a query; the encoder turned my plus signs into %2B, and the FDA's API answered 500 (a server error). Write real spaces and let the encoder make the pluses:

from urllib.parse import urlencode
print(urlencode({'search': 'report_date:[20250101 TO 20260101]'}))
# search=report_date%3A%5B20250101+TO+20260101%5D

EXAMPLE — YOU ASK CLAUDE CODE

Check it yourself: run it twice; the second run's downloaded sources say unchanged. Then switch off your Wi-Fi and run it again. Each downloaded source prints FAILED after its retries (about 20 seconds each), each region holding one says not published, and that region's file in data/ keeps its old modified time (ls -l data in Git Bash or on a Mac, dir data in PowerShell). Regions built only from local files still rebuild. That's the whole promise: a refresh can fail, and the site doesn't notice.

The price tool's imports take from 13 minutes to over 2 hours per state, and AI coding tools and SSH sessions often cut a command off long before that (after 10 minutes, in my setup). So long jobs run detached: in the background, not tied to your terminal, writing to a log file. Python holds back printed output when it writes to a file instead of a screen, so run python3 -u (unbuffered), or the log stays empty and the job looks hung. And make sure nothing is already running before you start a copy.

So ingest.py needs a lock (a marker only one running copy can hold): a second copy prints another ingest is already running and stops. Linux frees the lock when the job ends, even in a crash, so a dead run can't block next week's.

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath, in your two terminal tabs, the laptop one and the one logged in with ssh root@YOUR_VPS_IP (replace YOUR_VPS_IP with your server's address, and myapp with your app's short name; no raw/ folder yet? leave raw out of the scp line):

ssh root@YOUR_VPS_IP 'mkdir -p /root/myapp-pipeline'
scp -r tools raw root@YOUR_VPS_IP:/root/myapp-pipeline/

python3 --version
pgrep -af ingest.py
cd /root/myapp-pipeline && nohup python3 -u tools/ingest.py --out /root/myapp/data >> ingest.log 2>&1 &
tail -f /root/myapp-pipeline/ingest.log

No python3? apt install python3. pgrep prints nothing when no copy is running. nohup keeps the job alive after you log out, >> appends to the log, 2>&1 sends errors there too, and the final & runs it in the background. tail -f follows the log; Ctrl+C stops watching, not the job. The job writes into /root/myapp/data, inside the folder your app's file server already serves, so each file is live the moment it's renamed in, while the code, cache and lock stay in /root/myapp-pipeline, outside the site.

Every Monday, by itself

EXAMPLE — YOU ASK CLAUDE CODE

[Unit]
Description=myapp weekly data refresh

[Service]
Type=oneshot
TimeoutStartSec=0
OOMScoreAdjust=500
WorkingDirectory=/root/myapp-pipeline
StandardOutput=append:/root/myapp-pipeline/ingest.log
ExecStart=-/usr/bin/python3 -u tools/ingest.py --region AZ --out /root/myapp/data
ExecStart=-/usr/bin/python3 -u tools/ingest.py --region CA --out /root/myapp/data

[Unit]
Description=Run myapp-ingest every Monday at 03:00

[Timer]
OnCalendar=Mon 03:00
Persistent=true

[Install]
WantedBy=timers.target

oneshot runs to the end, then stops, and TimeoutStartSec=0 lifts the time limit. Each ExecStart line is one region (use yours) in its own process, which frees its memory when it ends; the leading - lets the next region run even if one fails. Persistent=true runs a Monday the server missed while it was off, once it boots. 03:00 is server time, often UTC, world time (timedatectl shows yours). If another app's heavy job already runs Mondays, use Tue 03:00: the two sites on my price-tool code refresh on different nights, so their imports never squeeze memory together.

systemctl daemon-reload && systemctl enable --now myapp-ingest.timer
systemctl list-timers myapp-ingest.timer

You should see one row whose NEXT column is the coming Monday at 03:00:00. To run it now, systemctl start --no-block myapp-ingest (same lock, same log, same memory rules), and systemctl is-active myapp-ingest prints activating while it runs. To stop the schedule, systemctl disable --now myapp-ingest.timer.

When memory runs out

Your service already does two of those: one run per region, and OOMScoreAdjust=500. Add the swap once. Ask Claude: "Check memory, swap and free disk on my server. If there's no swap and at least 5 GB free, add a 4 GB swap file that survives reboots, set swappiness to 10, and show me free -h and swapon --show." Underneath, it checks first:

swapon --show
df -h /

If swapon --show printed nothing (no swap yet) and df shows at least 5G under Avail, it runs:

fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
grep -q '^/swapfile ' /etc/fstab || echo '/swapfile none swap sw 0 0' >> /etc/fstab
echo 'vm.swappiness=10' > /etc/sysctl.d/99-swappiness.conf && sysctl -p /etc/sysctl.d/99-swappiness.conf
free -h

Line one makes and switches on a 4 GB swap file, line two keeps it after reboots (only once, even if you paste it twice), and line three tells Linux to prefer real memory. free -h should now show about 4.0Gi on its Swap line. (A few container-based server types forbid swap files, and swapon will say so.) If your site ever restarts overnight for no visible reason, run journalctl -k | grep -i oom; a job that ended with exit status 137 was killed, usually by the OOM killer.

Joins: where confident wrong answers come from

A wrong match looks exactly like a right one, so match on the strongest evidence first and leave anything doubtful unmatched:

1. The street address each source states about itself, normalized (tidied into one standard spelling).
2. Name similarity, but only between records with the same parent organization.
3. A mutual best match: A's best candidate is B, and B's best is A.
4. Everything else goes on an unmatched list for a person to check. Never guess.

Same ID, different level

Check that an ID means the same thing on both sides. In The List's data, every co-op row, the building's own and each apartment's, carries the building's parcel ID (the city's number for the lot). Joining tax bills on it gave every apartment its whole building's bill, and the citywide total came to about $21 billion, against about $33 billion for the city's entire property-tax levy. That outside figure, your alarm from Numbers you can defend, caught it at once. Before any join, count the rows sharing each ID on each side: more than one means a level mismatch, and the building's figure has to be divided out.

A fallback key, and the match type recorded

The List's sales data files an apartment sale under the building's lot, while the roll lists the apartment's own lot, so joining on ID alone matched 12 of 12,375 condos. A fallback keyed on borough plus a cleaned address (apartment part stripped) took coverage to 23,737 of 31,541 properties: 12,359 exact and 11,378 same-building matches. Every row stores its match_type, and the page words it to fit: "last recorded sale in this building".

Guard every normalization rule with a test. Retrying a range like 120-124 Main St at its low end added 1,341 matches, but Queens house numbers are hyphenated too (141-01) and aren't ranges. Without the guard "only when the second number is bigger", about 1,076 Queens addresses would have pointed at the wrong buildings in Manhattan.

import re

def address_keys(addr):
    """'120-124 Main St, 4B' -> ['120-124 MAIN ST', '120 MAIN ST']"""
    a = ' '.join(addr.split(',')[0].upper().split())  # drop the unit after a comma, tidy spaces
    keys = [a]
    m = re.match(r'(\d+)-(\d+) (.+)', a)
    if m and int(m[2]) > int(m[1]):                   # a range: also try its low end
        keys.append(m[1] + ' ' + m[3])
    return keys

assert address_keys('120-124 Main St, 4B') == ['120-124 MAIN ST', '120 MAIN ST']
assert address_keys('141-01 Example Ave') == ['141-01 EXAMPLE AVE']   # Queens: not a range

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath: each row tries the strongest key first and records which one hit, so the page can say "this building" instead of claiming "this apartment". Open review.csv and read ten rows. If most are real matches your rules missed, add a rule with a guard and a test; if they're genuinely doubtful, they stay unmatched.

Same spirit, two more cases. Before adding "everything" from a reference source, diff it against what you have after folding case and accents (comparing é as e): a published slang dictionary of about 230 terms, checked against Acento's 214 existing headwords, held 67 genuinely new terms (134 new cards), not 230. Use a book like that as a checklist of terms, and write your own definitions. And to put addresses on a map, geocode in 1,000-row batches, cache every answer and drop points outside your area's bounding box (the rectangle around it), as in Maps, charts and dashboards.

Pay-per-call APIs

Some APIs charge per call, and a bug can make you pay twice. Print the cost before spending anything: a private keyword tool of mine plans every run first, and showed 220 queries for about 22 cents before fetching one. Save every raw response to disk before parsing it, named after its query, so re-scoring after a bug reads the cache for free. And put the provider behind a small interface (one class with a search(query) method, one subclass per vendor), so switching vendors touches one file.

Ask Claude: "Before any paid call, print how many calls this run makes times the price per call, and wait for my yes. Cache each raw response in cache/api/ under a hash of its query, and read the cache first."

Do it with Claude: the prompts

The core of this lesson is three prompts:

1. "Write tools/ingest.py with only the standard library: stream each source in sources.json row by row, keep rows whose code is in allowlist.json, write data.build/, then swap each region's file into data/ with one rename. Print kept rows per source."
2. "Run it detached with python3 -u, append to ingest.log, and refuse to start if another copy holds the lock."
3. "Add a weekly systemd timer with Persistent=true for Monday at 03:00."

What's happening underneath: the site only ever reads finished files, so a crash mid-build leaves visitors on last week's complete data. An AI can report a pipeline as working while some sources quietly import nothing; the per-source lines are your proof, not its summary. Put the routine in NOTES.md:

## Data pipeline
Laptop: py tools/ingest.py --selfcheck, then py tools/ingest.py   (Mac: python3)
Server: systemctl start --no-block myapp-ingest, then tail -f /root/myapp-pipeline/ingest.log
After every run: grep 'zero or failed' /root/myapp-pipeline/ingest.log | tail -n 2
  -> none, or each name explained here (marked known_empty if it's empty for good)
Schedule: Mondays 03:00 server time; other apps' heavy jobs on other nights.
Hand-fetched files: listed in DELIVERY.md, refreshed by calendar reminder.

Your turn

• py tools/ingest.py --selfcheck (Mac: python3) prints selfcheck ok, and fails with an error instead while utf-8-sig is changed to utf-8.
• A real run prints one kept= line per source in sources.json and writes one small JSON file per region plus regions.json, and every zero or FAILED is fixed, or explained in NOTES.md.
• A second run says unchanged for every downloaded source, and a run with your Wi-Fi off says not published for each region with a downloaded source, and leaves that region's file in data/ with its old modified time.
• On the server, systemctl list-timers myapp-ingest.timer shows next Monday at 03:00, swapon --show lists your swap file, and a run started with systemctl start --no-block myapp-ingest ends with zero or failed: none in ingest.log.
• Two of your datasets (or your data and a lookup, like sales or geocodes) are joined with a match_type on every row, the count per type is printed, unmatched rows sit in review.csv, and the 141-01 assert passes.

Next, Live feeds without getting banned: data that changes by the minute, fetched politely, where a rate limit must never be saved as "no data".