reworked it all

This commit is contained in:
2026-09-18 18:13:05 +02:00
parent fcd5aaeb6c
commit d7ef929160
8 changed files with 1199 additions and 254 deletions
+113 -141
View File
@@ -10,204 +10,176 @@ No Spotify. No YouTube Music. No ads. Just your music, your server, your rules.
## What This Stack Does ## What This Stack Does
- **Streams** your music library to any device via a Spotify-like interface - **Streams** your library to any device through Navidrome
- **Automatically discovers** new music weekly based on your listening habits - **Discovers** new music every week from **two independent sources**: ListenBrainz Weekly Exploration and Last.fm recommendations. If one of them is down, the other still delivers
- **Downloads** discovered tracks at high quality (FLAC/WAV preferred, 320kbps minimum) from the Soulseek network - **Downloads** the discoveries from Soulseek at high quality (FLAC/WAV preferred, 320 kbps minimum), through a VPN
- **Scrobbles** everything you listen to Last.fm and ListenBrainz - **Keeps only this week and last week** of discoveries. Older playlists and their files are removed, except tracks you starred or added to a playlist of your own
- **Routes all P2P traffic through a VPN** so your real IP is never exposed - **Heals itself**: a Soulseek session that drops is reconnected, and a stuck container is restarted, without you noticing
---
## Components ## Components
| Component | Role | | Component | Role |
|---|---| |---|---|
| **Navidrome** | Music server — indexes your library, streams to clients | | **Navidrome** | Music server (installed separately, e.g. Unraid Community Apps) |
| **Substreamer** | Mobile app (iOS/Android) — plays music from Navidrome | | **Gluetun** | VPN gateway. All Soulseek traffic exits via ProtonVPN |
| **Pangolin** | Reverse proxy — exposes Navidrome securely to the internet | | **slskd** | Soulseek client, living inside gluetun's network |
| **Gluetun** | VPN gateway container — all Soulseek traffic exits via ProtonVPN | | **autoheal** | Restarts gluetun / slskd when Docker marks them unhealthy |
| **slskd** | Soulseek client — searches and downloads music from the P2P network | | **Explo** | Searches slskd, downloads, builds the Navidrome playlist |
| **autoheal** | Watches slskd/gluetun health status and force-restarts them if they hang | | **discovery/discover.py** | Runs inside the Explo container and decides *what* Explo imports and *when*; adds Last.fm, outage handling and retention |
| **Explo** | Discovery engine — connects ListenBrainz recommendations to slskd | | **Lidarr** | Optional, for whole-album grabs |
| **ListenBrainz** | Open-source scrobbler + recommendation engine |
| **Last.fm** | Secondary scrobbler for stats and social features |
--- ## How Discovery Works
## Prerequisites Once a day (`DISCOVERY_SCHEDULE`, default 00:15) the orchestrator runs a **tick**. A tick only does work that is still missing, so it is safe at any frequency and after any restart:
- Unraid NAS (or any Linux server with Docker)
- ProtonVPN account (paid, for P2P/WireGuard support)
- Soulseek account — free at [slsknet.org](https://www.slsknet.org)
- ListenBrainz account — free at [listenbrainz.org](https://listenbrainz.org)
- Last.fm account — free at [last.fm](https://www.last.fm)
- Last.fm API key — free at [last.fm/api/account/create](https://www.last.fm/api/account/create)
- Substreamer app on your phone
- Pangolin or any reverse proxy for external access
---
## Folder Structure
``` ```
/mnt/user/appdata/ tick
├── gluetun/ ← VPN state (auto-populated) ├─ ListenBrainz: is there a Weekly Exploration playlist I have not imported yet?
├── slskd/ │ yes → check slskd is logged in → Explo imports it → "Weekly-Exploration-2026-Week38"
│ └── slskd.yml ← slskd config (copy from repo) │ no → nothing to do LB down → note it, try again next tick
└── explo/ ├─ Last.fm: does this week's playlist exist yet?
├── .env ← Explo config (copy from .env.example) │ no → pull recommendations → hand them to Explo → "Lastfm-Recommended-2026-Week38"
└── config/ ← Explo's playlist cache + cover art (auto-populated, must persist) └─ Prune: keep the newest KEEP_WEEKS per source, remove the rest
/mnt/user/data/media/music/
├── incomplete/ ← In-progress downloads (Navidrome ignores these)
├── explo/ ← Explo-managed discovery downloads
└── (your library)/ ← Everything else
``` ```
Things worth knowing:
- **ListenBrainz is tracked by playlist ID, not by calendar.** If ListenBrainz publishes late, or is down on Tuesday, the playlist is simply picked up on the first tick after it appears. If it never appears, last week's playlist is *not* re-imported under a new name. A playlist that keeps failing is attempted 3 times, then left alone.
- **Last.fm recommendations** come from the same feed the Last.fm website player uses (`last.fm/player/station/user/<you>/recommended`). It is public, needs no login, and returns Last.fm's own picks of artists you have not played. It is not an official API, so with `LASTFM_API_KEY` set there is a second path built purely on the official API (artists similar to your last three months, minus everyone you already know). Tracks recommended in earlier weeks are remembered and not repeated.
- **`LASTFM_MODE`**: `always` gives you two playlists a week; `fallback` only steps in, from Wednesday on, in weeks where ListenBrainz has produced nothing; `off` disables it.
- **slskd preflight.** Before Explo starts, the orchestrator checks that slskd is really logged in to Soulseek and nudges it if not. If it stays down, the run is postponed to the next tick. Without this, one bad night means fifty YouTube rips instead of fifty FLACs.
- **No jams.** Weekly Jams and Daily Jams are gone. `PRUNE_LEGACY_JAMS=true` cleans up what the old setup left behind.
### Retention
With `KEEP_WEEKS=2` you always have this week's and last week's playlist per source. On each tick, anything older goes: the Navidrome playlist and the matching folder under `music/explo/`.
Before a folder is deleted, every file in it is checked against your **starred tracks and every playlist that is staying**. Matches are moved to `music/explo/Keepers/` instead of deleted, so starring a track is how you say "keep this one". Only folders the pipeline created (`Weekly-Exploration-*`, `Custom-Lastfm-*`) are ever touched.
Preview it first:
```bash
docker exec explo python3 /ainulindale/discover.py prune --dry-run
```
`PRUNE_FILES=false` removes old playlists but leaves all files on disk.
## Why slskd Stays Connected Now
The old setup had three gaps, which together explain "slskd drops and autoheal does nothing":
1. **slskd's healthcheck never looked at Soulseek.** It only asks whether the web server answers (and the image gives itself a 60 minute start period). Logged out of Soulseek with the UI up counts as healthy, so autoheal had nothing to act on. `slskd/healthcheck.sh` replaces it and reports the Soulseek login state. On a drop it first asks slskd to reconnect; only three failed checks in a row lead to a restart.
2. **A gluetun restart strands slskd.** slskd borrows gluetun's network namespace. When gluetun restarts, slskd is left holding the old, dead one: localhost still works (so the old check passed) but nothing reaches the internet. The new check notices that gluetun's control server has vanished from localhost and fails, and autoheal restarts slskd into the new namespace.
3. **slskd did not know about the VPN.** slskd has a native gluetun integration that was not switched on. It now polls gluetun every few seconds, drops the Soulseek session when the tunnel drops, logs in again when it is back, and applies the forwarded port by itself when port forwarding is on.
When the tunnel itself is down, the healthcheck stays green on purpose: restarting slskd cannot fix a VPN, and gluetun repairs its own tunnel.
One thing no healthcheck can fix: **do not log in with the same Soulseek account anywhere else** (Nicotine+, SoulseekQt on a desktop). The server kicks the older session, and the two clients will keep throwing each other out.
--- ---
## Setup ## Setup
### 1. Clone the repo ### 1. Clone and configure
```bash
git clone https://github.com/yourusername/ainulindale
cd ainulindale
```
### 2. Create your config files
```bash ```bash
git clone https://github.com/Quinta0/Ainulindale
cd Ainulindale
cp .env.example .env cp .env.example .env
``` ```
Fill in `.env` with your WireGuard private key, paths, and VPN country. Everything lives in that one `.env`: VPN key, Soulseek login, Navidrome login, ListenBrainz and Last.fm names, retention. Generate the two internal keys with `openssl rand -hex 24`.
Copy `slskd.yml` to your appdata folder: Run `docker compose` from this folder. The compose file mounts `./discovery` and `./slskd/healthcheck.sh` by relative path.
```bash
cp slskd.yml /mnt/user/appdata/slskd/slskd.yml
```
Fill in your Soulseek credentials and choose a web UI username/password. ### 2. Folders
Copy the Explo env block from `.env.example` into a separate file:
```bash
cp .env.example /mnt/user/appdata/explo/.env
```
Fill in your ListenBrainz username, Navidrome URL/credentials, and slskd API key.
### 3. Create required folders
```bash ```bash
mkdir -p /mnt/user/appdata/{gluetun,slskd,explo/config} APPDATA_PATH=/mnt/user/appdata; MUSIC_PATH=/mnt/user/data/media/music # same as in .env
mkdir -p /mnt/user/data/media/music/{explo,incomplete} mkdir -p $APPDATA_PATH/{gluetun,slskd,explo/config}
mkdir -p $MUSIC_PATH/{explo,slskd/incomplete}
touch $MUSIC_PATH/slskd/incomplete/.ndignore # Navidrome skips half-downloaded files
cp slskd/slskd.yml $APPDATA_PATH/slskd/slskd.yml
``` ```
### 4. Get your ProtonVPN WireGuard key ```
$APPDATA_PATH/
├── gluetun/
├── slskd/slskd.yml ← tuning only, no secrets
└── explo/config/ ← Explo cache + ainulindale-state.json (must persist)
1. Log into [account.proton.me](https://account.proton.me) → VPN → Downloads $MUSIC_PATH/
2. Select **WireGuard** protocol and a **P2P-capable server** ├── slskd/ ← slskd downloads (manual ones show up in Navidrome from here)
3. Generate the config and copy the `PrivateKey` value into your `.env` │ └── incomplete/
├── explo/
### 5. Start the stack │ ├── Weekly-Exploration-2026-Week38/
│ ├── Custom-Lastfm-2026-38/
```bash │ └── Keepers/ ← rescued favourites
# Start VPN first and verify it connects └── (your library)
docker compose up -d gluetun
docker compose logs -f gluetun
# Look for: "Public IP address is x.x.x.x"
# Then start slskd
docker compose up -d slskd
``` ```
Open `http://your-nas-ip:5030`, log in with your slskd credentials, go to **Options → API Keys**, generate a key and paste it into `/mnt/user/appdata/explo/.env` as `SLSKD_API_KEY`. ### 3. ProtonVPN key
### 6. Install Navidrome via Unraid Community Apps [account.proton.me](https://account.proton.me) → VPN → Downloads → WireGuard, pick a P2P server, copy `PrivateKey` into `.env`.
Add these environment variables in the Navidrome container template: ### 4. Navidrome
| Variable | Value | | Variable | Value |
|---|---| |---|---|
| `ND_LASTFM_APIKEY` | your Last.fm API key |
| `ND_LASTFM_SECRET` | your Last.fm shared secret |
| `ND_LASTFM_ENABLED` | `true` | | `ND_LASTFM_ENABLED` | `true` |
| `ND_LASTFM_APIKEY` / `ND_LASTFM_SECRET` | your Last.fm API key and secret |
| `ND_SCANSCHEDULE` | `1m` | | `ND_SCANSCHEDULE` | `1m` |
### 7. Start Explo and autoheal Link both Last.fm and ListenBrainz in Navidrome's user settings so both services learn from what you play.
### 5. Start and verify
```bash ```bash
docker compose up -d explo autoheal docker compose up -d
docker compose ps # gluetun and slskd should turn "healthy"
docker inspect slskd --format '{{range .State.Health.Log}}{{.Output}}{{end}}' | tail -1
# → ok: Connected, LoggedIn
docker exec explo python3 /ainulindale/discover.py status
docker exec explo python3 /ainulindale/discover.py lastfm-preview # what Last.fm would give you
docker exec explo python3 /ainulindale/discover.py run # a full tick now, instead of waiting for 00:15
``` ```
Unlike the rest of the stack, Explo doesn't need any external scheduler (no Unraid User Scripts, no cron). It runs continuously and schedules Weekly Exploration and Daily Jams itself via `WEEKLY_EXPLORATION_SCHEDULE` / `DAILY_JAMS_SCHEDULE` in `docker-compose.yml`, and `EXECUTE_ON_START=false` means restarting or updating the container never triggers an extra, out-of-schedule run. This is what keeps it from downloading the same week twice — see [Troubleshooting](#troubleshooting) below for the full explanation. ## Migrating From the Previous Version
`autoheal` also just runs continuously in the background; it watches slskd and gluetun and restarts either one automatically if Docker marks it unhealthy. 1. `docker compose down`
2. **Replace `slskd.yml`.** slskd lets the YAML file override environment variables, so an old file with `soulseek:`, `web:` or `directories:` blocks silently wins over `.env`. Back it up, copy the new one in.
3. Move your settings from `appdata/explo/.env` into the root `.env` (names differ slightly, see `.env.example`). The old file is no longer read.
4. Set `GLUETUN_API_KEY` and `SLSKD_API_KEY`. Gluetun's control port 8000 is no longer published on the host; nothing outside the stack needs it.
5. New downloads land in `music/slskd/`. Existing files stay where they are. Set `SLSKD_DOWNLOADS_PATH=$MUSIC_PATH` to keep the old behaviour.
6. `docker compose up -d`, then run `prune --dry-run` (above). The first real prune will remove every `Weekly-Exploration-*` week except the newest two, so **star what you want to keep first**. Set `PRUNE_LEGACY_JAMS=true` for one run to clear the old jams playlists and folders.
--- The first tick re-creates this week's ListenBrainz playlist once (it has no record of the old setup having imported it). Tracks already on disk are recognised, not downloaded again.
## How Discovery Works
```
You listen in Substreamer
↓
Navidrome scrobbles to Last.fm + ListenBrainz
↓
ListenBrainz builds your taste profile over time
↓
Every Monday night: ListenBrainz generates Weekly Exploration playlist
↓
Tuesday 00:15: Explo pulls recommendations
↓
slskd searches Soulseek network (behind ProtonVPN)
↓
Downloads FLAC/WAV/320kbps+ to /mnt/user/data/media/music
↓
Navidrome scans every 1 minute → appears in Substreamer
↓
You discover new music 🎵
```
---
## Troubleshooting ## Troubleshooting
### slskd hangs and needs a manual restart every ~24h **slskd keeps restarting.** `docker inspect slskd --format '{{json .State.Health.Log}}'` shows what the healthcheck saw. "cannot read Soulseek state" means `SLSKD_API_KEY` is not the key slskd is using (old `slskd.yml` still in place?). "logged out of Soulseek" every time, with `docker logs slskd` mentioning another client, means the account is in use elsewhere.
This was almost always one of two things, both addressed in the current compose file: **slskd is stuck in "Restarting".** It validates its settings on start and exits on the first bad one; `docker logs slskd --tail 40` names it. Usual causes: `PORT_FORWARDING` set to `on` instead of `true`, a missing `music/slskd/incomplete` folder, an `SLSKD_API_KEY` under 16 characters.
- **slskd itself getting stuck** (memory pressure / stuck transfers) — fixed by pinning `slskd/slskd:0.25.1`, which bundles a round of upstream fixes for stuck and failing transfers. **slskd never logs in, log says it is waiting for the VPN.** `GLUETUN_API_KEY` is empty or differs between the two containers, or `PORT_FORWARDING=true` on a Proton plan/server without port forwarding.
- **gluetun's tunnel silently hanging** — since slskd runs inside gluetun's network namespace, a frozen gluetun looks identical to a frozen slskd. Fixed by pinning `qmcgaw/gluetun:v3.41.1`, which resolves a healthcheck race condition that could make gluetun hang completely.
Both containers already ship a Docker healthcheck, but **Docker does not restart a container just because it's unhealthy** — that gap is why a manual restart was needed. The new `autoheal` service closes it: it watches both containers (via the `autoheal=true` label) and force-restarts whichever one goes unhealthy, with no manual intervention. **Tick says "slskd did not log in ... Skipping this run".** Working as intended; it retries on the next tick. `REQUIRE_SLSKD=false` lets runs go ahead on YouTube alone.
If slskd still misbehaves after this, check `docker ps` for its health status and `docker logs slskd`/`docker logs gluetun` before restarting manually. **Last.fm gives 0 tracks.** Open `https://www.last.fm/player/station/user/<you>/recommended` in a browser. Empty JSON means the account has too few scrobbles, or the profile's listening data is private. Set `LASTFM_API_KEY` to enable the API-based path.
### Weekly Exploration downloaded multiple times **Prune says it could not remove a folder.** Files from the old root-owned setup while Explo now runs with `PUID`/`PGID`. `chown -R` the `music/explo` folder.
The old setup ran two one-shot `explo-weekly` / `explo-daily` containers (`restart: "no"`, `EXECUTE_ON_START=true`) that were triggered by an external Unraid User Scripts cron. That design had two compounding problems: **Upgrading Explo.** The Last.fm bridge writes Explo's playlist cache (`config/cache/custom-*.json`), an internal format. It is pinned to `v1.2.0`; after bumping, run `discover.py run` once and watch the log. If Explo ever ships Last.fm support itself ([issue #135](https://github.com/LumePart/Explo/issues/135)), the bridge can retire.
1. Every container restart (a host reboot, a Docker update, the scheduler firing slightly out of sync) re-ran Explo immediately because of `EXECUTE_ON_START=true`, regardless of whether that week's playlist already existed.
2. Explo's playlist cache had nowhere to persist between runs (no `/opt/explo/config` volume), so each fresh container had no memory of "I already built this week's playlist" and would build it again.
The current compose runs a single, always-on `explo` container with its own internal cron (`WEEKLY_EXPLORATION_SCHEDULE` / `DAILY_JAMS_SCHEDULE`), `EXECUTE_ON_START=false`, and a persistent `/opt/explo/config` volume, plus the newer `v1.1.0` image which includes further upstream fixes for scheduled-job bugs. Together this means Explo only ever runs on its own schedule, and remembers what it already did across restarts.
**One-time cleanup:** this fix prevents *future* duplicates, but it won't retroactively merge the duplicate `Weekly-Exploration-2026-WeekXX` playlists already sitting in Navidrome. Delete the extras once, manually, from the Navidrome UI (or via its API) — new runs going forward should stay to one playlist per period.
---
## Tips ## Tips
- **ListenBrainz needs time** — it takes a few weeks of scrobbling before Weekly Exploration playlists are generated. Speed this up by importing your Google Takeout YouTube Music watch history via [ytm-extractor](https://community.metabrainz.org/t/ytm-extractor-import-youtube-music-listens-to-listenbrainz/707619). - **ListenBrainz needs a few weeks** of scrobbles before Weekly Exploration appears. Last.fm recommendations work with whatever history your account already has, which makes it a good bridge in the meantime.
- **Manual downloads** — search for any artist or album directly in the slskd web UI at `http://your-nas-ip:5030`. Downloads land straight in your Navidrome library. - **Manual downloads**: search in the slskd UI at `http://your-nas-ip:5030`; results land in `music/slskd/`.
- **Playlist import** — use [Soundiiz](https://soundiiz.com) to transfer playlists from YouTube Music/Spotify directly into Navidrome. Tracks already in your library get matched automatically. - **Sharing**: slskd shares your library back. The more you share, the better your queue position with other peers.
- **Quality** — slskd is tried first for every download. YouTube is only used as a last resort fallback for tracks not found on Soulseek. - **Port forwarding** noticeably improves Soulseek results. If your Proton plan has it, set `PORT_FORWARDING=true` (one switch for gluetun and slskd; it must be `true`/`false`, slskd does not start on `on`/`off`).
- **Sharing** — slskd shares your music library back to the Soulseek network. The more you share, the better your download priority from other peers.
- **ProtonVPN** — the WireGuard private key in Gluetun pins you to a specific server. Make sure to use a P2P-capable server when generating your WireGuard config on ProtonVPN's site.
--- ---
## License ## License
MIT — do whatever you want with it. MIT. Do whatever you want with it.
+815
View File
@@ -0,0 +1,815 @@
#!/usr/bin/env python3
"""
Ainulindale discovery orchestrator.
Runs inside the Explo container (python 3.12, stdlib only) and drives the Explo
binary instead of Explo's built-in cron. One idempotent "tick" does this:
1. Preflight make sure slskd is actually logged in to Soulseek, otherwise
Explo would silently fall back to YouTube for the whole week.
2. ListenBrainz import Weekly Exploration, but only if ListenBrainz has
published a playlist we have not imported yet.
3. Last.fm build a playlist from Last.fm recommendations and hand it to
Explo as a "custom" playlist (always, or only as a fallback).
4. Prune keep the newest N weeks per source, remove older playlists
and their files; starred / playlisted tracks are rescued.
Because every step checks state first, the tick can run as often as you like
and after any restart without ever producing duplicates.
Usage: discover.py run | status | prune [--dry-run] | lastfm-preview
"""
from __future__ import annotations
import datetime as dt
import fcntl
import hashlib
import json
import os
import random
import re
import secrets
import shutil
import subprocess
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
# --------------------------------------------------------------------------
# Configuration (all from environment, see .env.example)
# --------------------------------------------------------------------------
def env(name: str, default: str = "") -> str:
return os.environ.get(name, default).strip()
def env_bool(name: str, default: bool) -> bool:
v = env(name)
return default if v == "" else v.lower() in ("1", "true", "yes", "on")
def env_int(name: str, default: int) -> int:
try:
return int(env(name, str(default)))
except ValueError:
return default
CONFIG_DIR = env("WEB_DATA_PATH", "/opt/explo/config") # Explo reads its playlist cache from here
DATA_DIR = env("DOWNLOAD_DIR", "/data")
SLSKD_DIR = env("SLSKD_DIR", "/slskd")
EXPLO_BIN = env("EXPLO_BIN", "/opt/explo/explo")
EXPLO_ENV = env("WEB_ENV_PATH", "/opt/explo/.env")
STATE_FILE = os.path.join(CONFIG_DIR, "ainulindale-state.json")
LOCK_FILE = os.path.join(CONFIG_DIR, "ainulindale.lock")
KEEPERS_DIR = os.path.join(DATA_DIR, "Keepers")
LB_USER = env("LISTENBRAINZ_USER")
LB_TOKEN = env("LISTENBRAINZ_USER_TOKEN")
LB_ENABLED = env_bool("LISTENBRAINZ_ENABLED", True)
LB_MAX_ATTEMPTS = env_int("LISTENBRAINZ_MAX_ATTEMPTS", 3)
LASTFM_USER = env("LASTFM_USER")
LASTFM_API_KEY = env("LASTFM_API_KEY")
LASTFM_MODE = env("LASTFM_MODE", "always").lower() # always | fallback | off
LASTFM_STATIONS = [s for s in env("LASTFM_STATIONS", "recommended").split(",") if s.strip()]
LASTFM_TRACKS = env_int("LASTFM_TRACKS", 30)
LASTFM_FALLBACK_DAY = env_int("LASTFM_FALLBACK_DAY", 3) # ISO weekday, 3 = Wednesday
LASTFM_HISTORY = 1000 # remembered recommendations, so weeks do not repeat each other
NAVIDROME_URL = env("SYSTEM_URL").rstrip("/")
NAVIDROME_USER = env("SYSTEM_USERNAME")
NAVIDROME_PASS = env("SYSTEM_PASSWORD")
SLSKD_URL = env("SLSKD_URL").rstrip("/")
SLSKD_API_KEY = env("SLSKD_API_KEY")
USES_SLSKD = "slskd" in env("DOWNLOAD_SERVICES", "slskd,youtube").split(",")
REQUIRE_SLSKD = env_bool("REQUIRE_SLSKD", True)
SLSKD_WAIT_MIN = env_int("SLSKD_WAIT_MINUTES", 10)
KEEP_WEEKS = max(1, env_int("KEEP_WEEKS", 2))
PRUNE_FILES = env_bool("PRUNE_FILES", True)
KEEP_FAVOURITES = env_bool("KEEP_FAVOURITES", True)
PRUNE_LEGACY_JAMS = env_bool("PRUNE_LEGACY_JAMS", False)
EXPLO_TIMEOUT_MIN = env_int("EXPLO_TIMEOUT_MINUTES", 480)
USER_AGENT = "Mozilla/5.0 (compatible; Ainulindale/2.0; +https://github.com/Quinta0/Ainulindale)"
AUDIO_EXT = {".flac", ".wav", ".mp3", ".m4a", ".ogg", ".opus", ".aac", ".wma", ".aiff", ".ape"}
def log(msg: str) -> None:
print(f"{dt.datetime.now():%Y-%m-%d %H:%M:%S} [ainulindale] {msg}", flush=True)
# --------------------------------------------------------------------------
# Small HTTP helper
# --------------------------------------------------------------------------
class HttpError(Exception):
pass
def http_json(url, *, method="GET", headers=None, params=None, timeout=20, retries=2, allow_empty=False):
if params:
url = f"{url}{'&' if '?' in url else '?'}{urllib.parse.urlencode(params)}"
hdrs = {"User-Agent": USER_AGENT, "Accept": "application/json"}
hdrs.update(headers or {})
last = None
for attempt in range(retries + 1):
try:
data = b"" if method in ("PUT", "POST") else None
req = urllib.request.Request(url, method=method, headers=hdrs, data=data)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read()
if not body.strip():
if allow_empty:
return None
raise HttpError("empty response")
return json.loads(body)
except urllib.error.HTTPError as e:
last = HttpError(f"HTTP {e.code} from {urllib.parse.urlsplit(url).netloc}")
if e.code in (400, 401, 403, 404):
break
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
last = HttpError(f"{type(e).__name__}: {e}")
if attempt < retries:
time.sleep(3 * (attempt + 1))
raise last
# --------------------------------------------------------------------------
# State
# --------------------------------------------------------------------------
def load_state() -> dict:
try:
with open(STATE_FILE, encoding="utf-8") as f:
state = json.load(f)
except (OSError, ValueError):
state = {}
state.setdefault("listenbrainz", {})
state.setdefault("lastfm", {})
state["lastfm"].setdefault("history", [])
return state
def save_state(state: dict) -> None:
os.makedirs(CONFIG_DIR, exist_ok=True)
tmp = STATE_FILE + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump(state, f, indent=2, ensure_ascii=False)
os.replace(tmp, STATE_FILE)
def iso_week(d: dt.date | None = None) -> tuple[int, int]:
y, w, _ = (d or dt.date.today()).isocalendar()
return y, w
def week_key(d: dt.date | None = None) -> str:
y, w = iso_week(d)
return f"{y}-W{w:02d}"
# --------------------------------------------------------------------------
# slskd
# --------------------------------------------------------------------------
class Slskd:
def __init__(self, url: str, key: str):
self.url, self.key = url, key
def server_state(self) -> dict:
return http_json(f"{self.url}/api/v0/server", headers={"X-API-Key": self.key}, timeout=10, retries=0)
def connect(self) -> None:
http_json(f"{self.url}/api/v0/server", method="PUT", headers={"X-API-Key": self.key},
timeout=10, retries=0, allow_empty=True)
def wait_until_logged_in(self, minutes: int) -> bool:
"""True once slskd reports Connected+LoggedIn. Nudges a reconnect while waiting."""
deadline = time.monotonic() + minutes * 60
nudged_at = None # not 0.0: monotonic() itself can be < 120 right after a host boot
while True:
try:
st = self.server_state()
if st.get("isConnected") and st.get("isLoggedIn"):
return True
desc = st.get("state", "unknown")
transitioning = st.get("isConnecting") or st.get("isLoggingIn")
if not transitioning and (nudged_at is None or time.monotonic() - nudged_at > 120):
log(f"slskd is '{desc}', asking it to reconnect")
try:
self.connect()
except HttpError as e:
log(f"reconnect request failed: {e}")
nudged_at = time.monotonic()
except HttpError as e:
log(f"slskd not reachable at {self.url}: {e}")
if time.monotonic() >= deadline:
return False
time.sleep(20)
def slskd_ready() -> bool:
if not USES_SLSKD:
return True
if not (SLSKD_URL and SLSKD_API_KEY):
log("SLSKD_URL / SLSKD_API_KEY not set, skipping slskd preflight")
return True
if Slskd(SLSKD_URL, SLSKD_API_KEY).wait_until_logged_in(SLSKD_WAIT_MIN):
return True
if REQUIRE_SLSKD:
log(f"slskd did not log in to Soulseek within {SLSKD_WAIT_MIN} min. Skipping this run so the week "
"is not filled with YouTube rips; the next tick will retry.")
return False
log("slskd is not logged in, continuing anyway because REQUIRE_SLSKD=false")
return True
# --------------------------------------------------------------------------
# ListenBrainz
# --------------------------------------------------------------------------
JSPF_PLAYLIST = "https://musicbrainz.org/doc/jspf#playlist"
def lb_latest_playlist(user: str, patch: str = "weekly-exploration") -> dict | None:
"""Newest 'created for you' playlist of the given type: {'mbid', 'date', 'title'} or None."""
headers = {"Authorization": f"Token {LB_TOKEN}"} if LB_TOKEN else {}
best = None
offset = 0
while True:
page = http_json(
f"https://api.listenbrainz.org/1/user/{urllib.parse.quote(user)}/playlists/createdfor",
params={"count": 50, "offset": offset}, headers=headers, timeout=30)
items = page.get("playlists", [])
for item in items:
pl = item.get("playlist", {})
meta = pl.get("extension", {}).get(JSPF_PLAYLIST, {}).get("additional_metadata", {})
if meta.get("algorithm_metadata", {}).get("source_patch") != patch:
continue
date = pl.get("date", "")
if best is None or date > best["date"]:
best = {"mbid": pl.get("identifier", "").rstrip("/").rsplit("/", 1)[-1],
"date": date, "title": pl.get("title", "")}
offset += len(items)
if not items or offset >= page.get("playlist_count", 0):
return best
def lb_is_fresh(state: dict) -> bool:
"""Has a ListenBrainz playlist published in the last 7 days been imported?"""
date = state["listenbrainz"].get("imported_date", "")
try:
published = dt.datetime.fromisoformat(date.replace("Z", "+00:00"))
except ValueError:
return False
return dt.datetime.now(dt.timezone.utc) - published < dt.timedelta(days=7)
# --------------------------------------------------------------------------
# Last.fm
# --------------------------------------------------------------------------
def track_key(artist: str, title: str) -> str:
return re.sub(r"[^\w]+", "", f"{artist}|{title}".lower(), flags=re.UNICODE)
def lastfm_station(user: str, station: str, want: int, seen: set[str]) -> list[dict]:
"""
Last.fm's own recommendation engine, via the JSON the website player uses.
Undocumented but public and stable for years; every call returns a fresh
batch, so we poll until we have enough unseen tracks.
"""
url = f"https://www.last.fm/player/station/user/{urllib.parse.quote(user)}/{station}"
out: list[dict] = []
dry = 0
for _ in range(25):
if len(out) >= want or dry >= 3:
break
batch = http_json(url, timeout=30).get("playlist", [])
if not batch:
break
added = 0
for t in batch:
title = (t.get("name") or t.get("_name") or "").strip()
artists = [a.get("name") or a.get("_name") or "" for a in t.get("artists", [])]
artists = [a.strip() for a in artists if a.strip()]
if not title or not artists:
continue
key = track_key(artists[0], title)
if key in seen:
continue
seen.add(key)
out.append({"title": title, "artist": ", ".join(artists), "mainArtist": artists[0], "release": ""})
added += 1
dry = 0 if added else dry + 1
time.sleep(1)
return out[:want]
def lastfm_api(method: str, **params) -> dict:
params.update({"method": method, "api_key": LASTFM_API_KEY, "format": "json"})
data = http_json("https://ws.audioscrobbler.com/2.0/", params=params, timeout=20)
if "error" in data:
raise HttpError(f"Last.fm API error {data['error']}: {data.get('message')}")
time.sleep(0.25) # stay well under the 5 req/s limit
return data
def lastfm_similar(user: str, want: int, seen: set[str]) -> list[dict]:
"""
Fallback built only on the official API: artists similar to what you played
in the last 3 months, minus every artist you already know, one top track each.
"""
known = set()
for page in (1, 2):
top = lastfm_api("user.gettopartists", user=user, period="overall", limit=1000, page=page)
artists = top.get("topartists", {}).get("artist", [])
known.update(a["name"].lower() for a in artists)
if len(artists) < 1000:
break
recent = lastfm_api("user.gettopartists", user=user, period="3month", limit=30)
scores: dict[str, float] = {}
for seed in recent.get("topartists", {}).get("artist", []):
try:
sim = lastfm_api("artist.getsimilar", artist=seed["name"], limit=15, autocorrect=1)
except HttpError:
continue
for a in sim.get("similarartists", {}).get("artist", []):
if a["name"].lower() not in known:
scores[a["name"]] = scores.get(a["name"], 0.0) + float(a.get("match") or 0)
ranked = sorted(scores, key=scores.get, reverse=True)
out: list[dict] = []
for artist in ranked:
if len(out) >= want:
break
try:
top = lastfm_api("artist.gettoptracks", artist=artist, limit=5, autocorrect=1)
except HttpError:
continue
candidates = [t["name"] for t in top.get("toptracks", {}).get("track", []) if t.get("name")]
random.shuffle(candidates)
for title in candidates:
key = track_key(artist, title)
if key not in seen:
seen.add(key)
out.append({"title": title, "artist": artist, "mainArtist": artist, "release": ""})
break
return out
def lastfm_fill_albums(tracks: list[dict]) -> None:
"""Optional nicety when an API key is present: album names make Explo's matching stricter."""
for t in tracks:
try:
info = lastfm_api("track.getinfo", artist=t["mainArtist"], track=t["title"], autocorrect=1)
t["release"] = info.get("track", {}).get("album", {}).get("title", "") or ""
except HttpError:
pass
def lastfm_recommendations(state: dict, want: int) -> list[dict]:
seen = set(state["lastfm"]["history"])
tracks: list[dict] = []
for station in LASTFM_STATIONS:
if len(tracks) >= want:
break
try:
got = lastfm_station(LASTFM_USER, station.strip(), want - len(tracks), seen)
log(f"Last.fm station '{station}': {len(got)} new tracks")
tracks += got
except HttpError as e:
log(f"Last.fm station '{station}' failed: {e}")
if len(tracks) < want and LASTFM_API_KEY:
try:
got = lastfm_similar(LASTFM_USER, want - len(tracks), seen)
log(f"Last.fm similar-artists (API): {len(got)} new tracks")
tracks += got
except HttpError as e:
log(f"Last.fm API fallback failed: {e}")
if tracks and LASTFM_API_KEY:
lastfm_fill_albums(tracks)
return tracks
# --------------------------------------------------------------------------
# Explo bridge
# --------------------------------------------------------------------------
def lb_playlist_name(d: dt.date | None = None) -> str:
y, w = iso_week(d)
return f"Weekly-Exploration-{y}-Week{w}" # exactly how Explo names it with --replace-playlist=false
def lastfm_ids(d: dt.date | None = None) -> tuple[str, str]:
y, w = iso_week(d)
# digits only after the prefix: Explo title-cases the id to build the download folder name
return f"custom-lastfm-{y}-{w}", f"Lastfm-Recommended-{y}-Week{w}"
def register_custom_playlist(pid: str, name: str, tracks: list[dict]) -> None:
"""Write the two files `explo --playlist=custom-*` reads (Explo v1.2 cache format)."""
cache_dir = os.path.join(CONFIG_DIR, "cache")
os.makedirs(cache_dir, exist_ok=True)
with open(os.path.join(cache_dir, f"{pid}.json"), "w", encoding="utf-8") as f:
json.dump({"tracks": tracks}, f, ensure_ascii=False)
meta_path = os.path.join(CONFIG_DIR, "custom-playlists.json")
try:
with open(meta_path, encoding="utf-8") as f:
meta = json.load(f)
if not isinstance(meta, list):
meta = []
except (OSError, ValueError):
meta = []
meta = [m for m in meta if m.get("id") != pid]
meta.append({"id": pid, "name": name, "source": "lastfm", "refresh_days": 0,
"last_fetched": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")})
with open(meta_path, "w", encoding="utf-8") as f:
json.dump(meta, f, indent=2, ensure_ascii=False)
def unregister_custom_playlist(pid: str) -> None:
try:
os.remove(os.path.join(CONFIG_DIR, "cache", f"{pid}.json"))
except OSError:
pass
meta_path = os.path.join(CONFIG_DIR, "custom-playlists.json")
try:
with open(meta_path, encoding="utf-8") as f:
meta = [m for m in json.load(f) if m.get("id") != pid]
with open(meta_path, "w", encoding="utf-8") as f:
json.dump(meta, f, indent=2, ensure_ascii=False)
except (OSError, ValueError):
pass
def run_explo(*flags: str) -> int:
cmd = [EXPLO_BIN, "--config", EXPLO_ENV, *flags]
log("running: " + " ".join(cmd))
try:
return subprocess.run(cmd, cwd=os.path.dirname(EXPLO_BIN) or ".", timeout=EXPLO_TIMEOUT_MIN * 60).returncode
except subprocess.TimeoutExpired:
log(f"Explo exceeded {EXPLO_TIMEOUT_MIN} min and was stopped")
return 124
except OSError as e:
log(f"could not start Explo: {e}")
return 127
# --------------------------------------------------------------------------
# Navidrome (Subsonic API)
# --------------------------------------------------------------------------
class Navidrome:
def __init__(self, url: str, user: str, password: str):
self.url, self.user, self.password = url, user, password
def call(self, endpoint: str, **params) -> dict:
salt = secrets.token_hex(8)
params.update({"u": self.user, "s": salt, "t": hashlib.md5((self.password + salt).encode()).hexdigest(),
"v": "1.16.1", "c": "ainulindale", "f": "json"})
resp = http_json(f"{self.url}/rest/{endpoint}", params=params, timeout=30).get("subsonic-response", {})
if resp.get("status") != "ok":
raise HttpError(f"Navidrome {endpoint}: {resp.get('error', {}).get('message', 'failed')}")
return resp
def playlists(self) -> list[dict]:
return self.call("getPlaylists").get("playlists", {}).get("playlist", []) or []
def playlist_entries(self, pid: str) -> list[dict]:
return self.call("getPlaylist", id=pid).get("playlist", {}).get("entry", []) or []
def starred(self) -> list[dict]:
return self.call("getStarred2").get("starred2", {}).get("song", []) or []
def delete_playlist(self, pid: str) -> None:
self.call("deletePlaylist", id=pid)
def scan(self) -> None:
self.call("startScan")
def navidrome() -> Navidrome | None:
if NAVIDROME_URL and NAVIDROME_USER:
return Navidrome(NAVIDROME_URL, NAVIDROME_USER, NAVIDROME_PASS)
return None
# --------------------------------------------------------------------------
# Retention
# --------------------------------------------------------------------------
# (label, playlist-name regex, download-folder regex, how many to keep)
def families() -> list[tuple[str, re.Pattern, re.Pattern, int]]:
fams = [
("ListenBrainz weekly exploration",
re.compile(r"^Weekly-Exploration-(\d{4})-Week(\d{1,2})$"),
re.compile(r"^Weekly-Exploration-(\d{4})-Week(\d{1,2})$"), KEEP_WEEKS),
("Last.fm recommendations",
re.compile(r"^Lastfm-Recommended-(\d{4})-Week(\d{1,2})$"),
re.compile(r"^custom-lastfm-(\d{4})-(\d{1,2})$", re.I), KEEP_WEEKS),
]
if PRUNE_LEGACY_JAMS:
jams = re.compile(r"^(?:Weekly|Daily)-Jams(?:-(\d{4})-(?:Week|Day)(\d{1,3}))?$")
fams.append(("legacy jams", jams, jams, 0))
return fams
def split_keep(names: list[str], rx: re.Pattern, keep: int) -> tuple[list[str], list[str]]:
def order(n: str):
g = rx.match(n).groups()
return int(g[0] or 0), int(g[1] or 0)
# a set: the same name can exist twice in Navidrome (duplicates left by older setups)
# and must only ever use up one of the "keep" slots
ranked = sorted({n for n in names if rx.match(n)}, key=order, reverse=True)
return ranked[:keep], ranked[keep:]
def rescue_or_delete(folder: str, protected: set[tuple[int, str]], dry: bool) -> tuple[int, int]:
"""Delete an expired download folder; files you starred or playlisted move to Keepers/ instead."""
rescued = removed = 0
for root, _dirs, files in os.walk(folder):
for fn in files:
path = os.path.join(root, fn)
ext = os.path.splitext(fn)[1].lower()
try:
size = os.path.getsize(path)
except OSError:
continue
if ext in AUDIO_EXT and (size, ext.lstrip(".")) in protected:
rescued += 1
if not dry:
os.makedirs(KEEPERS_DIR, exist_ok=True)
dest = os.path.join(KEEPERS_DIR, fn)
if os.path.exists(dest):
stem, e = os.path.splitext(fn)
dest = os.path.join(KEEPERS_DIR, f"{stem}.{int(time.time())}{e}")
shutil.move(path, dest)
else:
removed += 1
if not dry:
shutil.rmtree(folder, ignore_errors=True)
if os.path.isdir(folder):
log(f" could not fully remove {folder} (permissions? files from an older root-run setup?)")
return rescued, removed
def sweep_empty_slskd_dirs(dry: bool) -> None:
"""Explo moves finished files out of slskd's folder; drop empty shells older than a day."""
if not os.path.isdir(SLSKD_DIR) or os.path.isdir(os.path.join(SLSKD_DIR, "explo")):
return # an explo/ folder in here means slskd downloads into the library root: hands off
cutoff = time.time() - 86400
for name in os.listdir(SLSKD_DIR):
path = os.path.join(SLSKD_DIR, name)
if name == "incomplete" or not os.path.isdir(path):
continue
try:
if not os.listdir(path) and os.path.getmtime(path) < cutoff:
log(f" removing empty slskd folder {name}")
if not dry:
os.rmdir(path)
except OSError:
pass
def prune(dry: bool = False) -> None:
tag = "[dry-run] " if dry else ""
nd = navidrome()
doomed_playlists: list[dict] = []
all_playlists: list[dict] = []
if nd:
try:
all_playlists = nd.playlists()
except HttpError as e:
log(f"prune: Navidrome unreachable ({e}); nothing pruned this time")
return
try:
folders = [d for d in os.listdir(DATA_DIR) if os.path.isdir(os.path.join(DATA_DIR, d))]
except OSError:
folders = []
doomed_folders: list[str] = []
for label, pl_rx, dir_rx, keep in families():
kept, old = split_keep([p["name"] for p in all_playlists], pl_rx, keep)
doomed_playlists += [p for p in all_playlists if p["name"] in old]
for name in kept: # duplicates of a week that stays: keep the fullest copy only
twins = sorted((p for p in all_playlists if p["name"] == name),
key=lambda p: int(p.get("songCount") or 0), reverse=True)
if len(twins) > 1:
log(f"{tag}prune {label}: {len(twins) - 1} duplicate(s) of {name}")
doomed_playlists += twins[1:]
_, old_dirs = split_keep(folders, dir_rx, keep)
doomed_folders += old_dirs
if old or old_dirs:
log(f"{tag}prune {label}: {len(old)} playlist(s), {len(old_dirs)} folder(s) beyond the newest {keep}")
if not doomed_playlists and not doomed_folders:
sweep_empty_slskd_dirs(dry)
return
# Anything starred, or sitting in a playlist that survives, must not be deleted.
# Navidrome hides real paths by default, so files are matched by exact byte size + extension.
protected: set[tuple[int, str]] = set()
if nd and PRUNE_FILES and KEEP_FAVOURITES and doomed_folders:
try:
doomed_ids = {p["id"] for p in doomed_playlists}
songs = nd.starred()
for p in all_playlists:
if p["id"] not in doomed_ids:
songs += nd.playlist_entries(p["id"])
protected = {(int(s["size"]), str(s.get("suffix", "")).lower()) for s in songs if s.get("size")}
except HttpError as e:
log(f"prune: could not read favourites ({e}); keeping files this time to be safe")
doomed_folders = []
for p in doomed_playlists:
log(f"{tag} delete playlist {p['name']}")
if not dry:
try:
nd.delete_playlist(p["id"])
except HttpError as e:
log(f" failed: {e}")
m = re.match(r"^Lastfm-Recommended-(\d{4})-Week(\d{1,2})$", p["name"])
if m: # forget the matching entry in Explo's custom playlist registry
unregister_custom_playlist(f"custom-lastfm-{m.group(1)}-{int(m.group(2))}")
if PRUNE_FILES:
for d in doomed_folders:
rescued, removed = rescue_or_delete(os.path.join(DATA_DIR, d), protected, dry)
log(f"{tag} delete folder {d}: {removed} file(s) removed, {rescued} kept in Keepers/")
if doomed_folders and nd and not dry:
try:
nd.scan()
except HttpError:
pass # needs an admin user; Navidrome's own scheduled scan will catch up
sweep_empty_slskd_dirs(dry)
# --------------------------------------------------------------------------
# The tick
# --------------------------------------------------------------------------
def delete_playlist_named(name: str) -> None:
nd = navidrome()
if not nd:
return
try:
for p in nd.playlists():
if p["name"] == name:
log(f"removing earlier partial playlist {name}")
nd.delete_playlist(p["id"])
except HttpError as e:
log(f"could not check for an existing '{name}' playlist: {e}")
def step_listenbrainz(state: dict) -> str:
"""Returns: imported | current | waiting | down | failed | disabled"""
if not (LB_ENABLED and LB_USER):
return "disabled"
lb = state["listenbrainz"]
try:
latest = lb_latest_playlist(LB_USER)
except HttpError as e:
log(f"ListenBrainz is not answering: {e}")
return "down"
if not latest or not latest["mbid"]:
log("ListenBrainz has no Weekly Exploration playlist for this user yet")
return "waiting"
if latest["mbid"] == lb.get("imported_mbid"):
log(f"ListenBrainz: newest playlist ({latest['date'][:10]}) is already imported")
return "current" if lb_is_fresh(state) else "waiting"
if lb.get("attempt_mbid") == latest["mbid"] and lb.get("attempts", 0) >= LB_MAX_ATTEMPTS:
log(f"ListenBrainz: giving up on playlist {latest['mbid']} after {LB_MAX_ATTEMPTS} failed attempts")
return "failed"
log(f"ListenBrainz: new playlist '{latest['title']}' ({latest['date'][:10]})")
if not slskd_ready():
return "failed"
lb["attempts"] = lb.get("attempts", 0) + 1 if lb.get("attempt_mbid") == latest["mbid"] else 1
lb["attempt_mbid"] = latest["mbid"]
save_state(state)
delete_playlist_named(lb_playlist_name())
rc = run_explo("--playlist=weekly-exploration", "--replace-playlist=false")
if rc != 0:
log(f"Explo exited with code {rc} on the ListenBrainz import (attempt {lb['attempts']}/{LB_MAX_ATTEMPTS})")
return "failed"
lb.update({"imported_mbid": latest["mbid"], "imported_date": latest["date"],
"imported_at": dt.datetime.now().isoformat(timespec="seconds"), "attempts": 0})
save_state(state)
return "imported"
def step_lastfm(state: dict, lb_status: str) -> str:
if LASTFM_MODE == "off" or not LASTFM_USER:
return "disabled"
lf = state["lastfm"]
if lf.get("week") == week_key():
log("Last.fm: this week's playlist already exists")
return "current"
if LASTFM_MODE == "fallback":
if lb_status in ("imported", "current"):
return "not needed"
if dt.date.today().isoweekday() < LASTFM_FALLBACK_DAY:
log("Last.fm fallback: giving ListenBrainz a little longer before stepping in")
return "waiting"
log(f"Last.fm fallback: ListenBrainz is '{lb_status}', stepping in")
tracks = lastfm_recommendations(state, LASTFM_TRACKS)
if len(tracks) < min(5, LASTFM_TRACKS):
log(f"Last.fm gave only {len(tracks)} usable tracks; will retry on the next tick")
return "failed"
if not slskd_ready():
return "failed"
pid, name = lastfm_ids()
register_custom_playlist(pid, name, tracks)
rc = run_explo(f"--playlist={pid}") # replace-playlist defaults to true: a re-run overwrites, never duplicates
if rc != 0:
log(f"Explo exited with code {rc} on the Last.fm import")
return "failed"
lf["week"] = week_key()
lf["history"] = (lf["history"] + [track_key(t["mainArtist"], t["title"]) for t in tracks])[-LASTFM_HISTORY:]
save_state(state)
return "imported"
def tick() -> int:
state = load_state()
lb_status = step_listenbrainz(state)
lf_status = step_lastfm(state, lb_status)
try:
prune()
except Exception as e: # pruning must never take the run down with it
log(f"prune failed: {type(e).__name__}: {e}")
state = load_state()
state["last_tick"] = {"at": dt.datetime.now().isoformat(timespec="seconds"),
"listenbrainz": lb_status, "lastfm": lf_status}
save_state(state)
log(f"done: listenbrainz={lb_status} lastfm={lf_status}")
return 75 if "failed" in (lb_status, lf_status) else 0
def status() -> int:
state = load_state()
print(json.dumps({k: v for k, v in state.items() if k != "lastfm"}, indent=2))
lf = dict(state["lastfm"])
lf["history"] = f"{len(lf.get('history', []))} remembered tracks"
print(json.dumps({"lastfm": lf}, indent=2))
if USES_SLSKD and SLSKD_URL and SLSKD_API_KEY:
try:
print("slskd:", Slskd(SLSKD_URL, SLSKD_API_KEY).server_state().get("state"))
except HttpError as e:
print("slskd: unreachable -", e)
if LB_USER:
try:
print("listenbrainz newest:", lb_latest_playlist(LB_USER))
except HttpError as e:
print("listenbrainz: down -", e)
return 0
def main() -> int:
cmd = sys.argv[1] if len(sys.argv) > 1 else "run"
if cmd == "status":
return status()
if cmd == "lastfm-preview":
if not LASTFM_USER:
print("LASTFM_USER is not set")
return 2
for t in lastfm_recommendations(load_state(), LASTFM_TRACKS):
print(f"{t['artist']} - {t['title']}" + (f" [{t['release']}]" if t["release"] else ""))
return 0
if cmd not in ("run", "prune"):
print(__doc__)
return 2
os.makedirs(CONFIG_DIR, exist_ok=True)
with open(LOCK_FILE, "w") as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
log("another run is still in progress, skipping")
return 0
if cmd == "prune":
prune(dry="--dry-run" in sys.argv)
return 0
return tick()
if __name__ == "__main__":
sys.exit(main())
+63
View File
@@ -0,0 +1,63 @@
#!/bin/sh
# Replaces Explo's /start.sh. Same image, same binary, but instead of one cron
# line per playlist there is a single scheduled "tick" of discover.py, which
# decides what (if anything) needs doing. See discover.py for the logic.
set -u
PUID="${PUID:-0}"
PGID="${PGID:-0}"
SCHEDULE="${DISCOVERY_SCHEDULE:-15 0 * * *}"
ENV_FILE="${WEB_ENV_PATH:-/opt/explo/.env}"
ENV_DUMP=/run/ainulindale.env
SCRIPT=/ainulindale/discover.py
say() { echo "[setup] $*"; }
# yt-dlp ages quickly; refresh at start (best effort) and nightly, like upstream does
pip install --disable-pip-version-check --root-user-action=ignore --no-cache-dir --upgrade yt-dlp >/dev/null 2>&1 \
&& say "yt-dlp is up to date" || say "WARN: could not update yt-dlp (no network yet?)"
if [ "$PUID" != "0" ] || [ "$PGID" != "0" ]; then
groupmod -o -g "$PGID" explo
usermod -o -u "$PUID" explo
RUNNER="su-exec explo"
mkdir -p /opt/explo/config
chown -R explo:explo /opt/explo/config
chown explo:explo /opt/explo /data 2>/dev/null
else
RUNNER=""
fi
# All settings arrive as container environment variables (from the stack's .env),
# so Explo only needs this file to exist.
[ -e "$ENV_FILE" ] || : > "$ENV_FILE"
[ -n "$RUNNER" ] && chown explo:explo "$ENV_FILE" 2>/dev/null
if [ -n "$RUNNER" ] && ! $RUNNER test -w /data; then
say "WARN: /data is not writable for ${PUID}:${PGID}; downloads will fail until you chown the explo music folder"
fi
# cron jobs start with a bare environment; hand them ours
export -p > "$ENV_DUMP"
chmod 600 "$ENV_DUMP"
JOB=". $ENV_DUMP; cd /opt/explo && $RUNNER python3 -u $SCRIPT run >> /proc/1/fd/1 2>&1"
{
echo "$SCHEDULE $JOB"
echo "59 23 * * * pip install --disable-pip-version-check --root-user-action=ignore --no-cache-dir --upgrade yt-dlp >> /proc/1/fd/1 2>&1"
} > /etc/crontabs/root
chmod 600 /etc/crontabs/root
say "discovery tick scheduled: $SCHEDULE ($(date +%Z))"
if [ "${EXPLO_WEB_UI:-false}" = "true" ]; then
say "starting Explo web UI on ${WEB_ADDR:-:7288} (viewing only; do not add schedules there)"
WEB_UI=true $RUNNER /opt/explo/explo --config "$ENV_FILE" &
fi
if [ "${RUN_ON_START:-false}" = "true" ]; then
say "RUN_ON_START=true, running one tick now (safe: already-imported weeks are skipped)"
(cd /opt/explo && $RUNNER python3 -u "$SCRIPT" run) &
fi
exec crond -f -l 2
+128 -38
View File
@@ -1,86 +1,176 @@
services: services:
# ---------------------------------------------------------------------------
# VPN gateway. slskd lives inside this container's network namespace.
# ---------------------------------------------------------------------------
gluetun: gluetun:
image: qmcgaw/gluetun:v3.41.1 image: qmcgaw/gluetun:v3.41.3
container_name: gluetun container_name: gluetun
cap_add: cap_add:
- NET_ADMIN - NET_ADMIN
devices: devices:
- /dev/net/tun:/dev/net/tun - /dev/net/tun:/dev/net/tun
environment: environment:
- VPN_SERVICE_PROVIDER=protonvpn VPN_SERVICE_PROVIDER: protonvpn
- VPN_TYPE=wireguard VPN_TYPE: wireguard
- WIREGUARD_PRIVATE_KEY=${WIREGUARD_PRIVATE_KEY} WIREGUARD_PRIVATE_KEY: ${WIREGUARD_PRIVATE_KEY}
- WIREGUARD_ADDRESSES=${WIREGUARD_ADDRESSES} WIREGUARD_ADDRESSES: ${WIREGUARD_ADDRESSES}
- SERVER_COUNTRIES=${VPN_COUNTRY:-Switzerland} SERVER_COUNTRIES: ${VPN_COUNTRY:-Switzerland}
- TZ=${TZ:-Europe/Zurich} TZ: ${TZ:-Europe/Zurich}
- VPN_PORT_FORWARDING=${VPN_PORT_FORWARDING:-off} # One switch for both containers. Must be true/false: gluetun also accepts
- VPN_PORT_FORWARDING_PROVIDER=protonvpn # on/off, but slskd reads the same value and only understands true/false.
- PORT_FORWARD_ONLY=${VPN_PORT_FORWARDING:-off} VPN_PORT_FORWARDING: ${PORT_FORWARDING:-false}
- VPN_PORT_FORWARDING_STATUS_FILE=/gluetun/forwarded_port VPN_PORT_FORWARDING_PROVIDER: protonvpn
PORT_FORWARD_ONLY: ${PORT_FORWARDING:-false}
# Control server on :8000. slskd polls it (over localhost, they share a
# network namespace) to follow the tunnel state and the forwarded port.
# Every route needs this key; the port is deliberately NOT published.
HTTP_CONTROL_SERVER_AUTH_DEFAULT_ROLE: '{"auth":"apikey","apikey":"${GLUETUN_API_KEY}"}'
ports: ports:
- 5030:5030 - 5030:5030 # slskd web UI
- 8000:8000
volumes: volumes:
- ${APPDATA_PATH}/gluetun:/gluetun - ${APPDATA_PATH}/gluetun:/gluetun
restart: unless-stopped restart: unless-stopped
labels: labels:
- autoheal=true - autoheal=true
# ---------------------------------------------------------------------------
# Soulseek client
# ---------------------------------------------------------------------------
slskd: slskd:
image: slskd/slskd:0.25.1 image: slskd/slskd:0.26.0
container_name: slskd container_name: slskd
network_mode: "service:gluetun" network_mode: "service:gluetun"
depends_on: depends_on:
gluetun: gluetun:
condition: service_healthy condition: service_healthy
restart: true # compose restarts slskd whenever it restarts gluetun
environment: environment:
- SLSKD_REMOTE_CONFIGURATION=true SLSKD_REMOTE_CONFIGURATION: "true"
SLSKD_SLSK_USERNAME: ${SOULSEEK_USERNAME}
SLSKD_SLSK_PASSWORD: ${SOULSEEK_PASSWORD}
SLSKD_USERNAME: ${SLSKD_WEB_USERNAME}
SLSKD_PASSWORD: ${SLSKD_WEB_PASSWORD}
SLSKD_API_KEY: ${SLSKD_API_KEY} # used by Explo, the orchestrator and the healthcheck
SLSKD_DOWNLOADS_DIR: /downloads
SLSKD_INCOMPLETE_DIR: /downloads/incomplete
# Native gluetun integration: slskd waits for the tunnel before logging in,
# drops the Soulseek session when the tunnel drops, logs back in when it
# returns, and (with port forwarding) applies the new listen port itself.
SLSKD_VPN: "true"
SLSKD_VPN_PORT_FORWARDING: ${PORT_FORWARDING:-false}
SLSKD_VPN_GLUETUN_URL: http://localhost:8000
SLSKD_VPN_GLUETUN_API_KEY: ${GLUETUN_API_KEY}
volumes: volumes:
- ${APPDATA_PATH}/slskd:/app - ${APPDATA_PATH}/slskd:/app
- ${MUSIC_PATH}:/downloads - ${SLSKD_DOWNLOADS_PATH:-${MUSIC_PATH}/slskd}:/downloads
- ${MUSIC_PATH}:/music:ro - ${MUSIC_PATH}:/music:ro
- ./slskd/healthcheck.sh:/healthcheck.sh:ro
healthcheck:
# The image's own check only looks at the web server and has a 60 minute
# start period. This one follows the Soulseek login (see the script).
test: ["CMD", "bash", "/healthcheck.sh"]
interval: 60s
timeout: 20s
retries: 3
start_period: 3m
restart: unless-stopped restart: unless-stopped
labels: labels:
- autoheal=true - autoheal=true
# ---------------------------------------------------------------------------
# Restarts anything labelled autoheal=true once Docker marks it unhealthy
# ---------------------------------------------------------------------------
autoheal: autoheal:
image: willfarrell/autoheal:latest image: willfarrell/autoheal:latest
container_name: autoheal container_name: autoheal
restart: unless-stopped restart: unless-stopped
environment: environment:
- AUTOHEAL_CONTAINER_LABEL=autoheal AUTOHEAL_CONTAINER_LABEL: autoheal
- AUTOHEAL_INTERVAL=30 AUTOHEAL_INTERVAL: 30
- AUTOHEAL_START_PERIOD=120 AUTOHEAL_START_PERIOD: 120
- AUTOHEAL_DEFAULT_STOP_TIMEOUT=30 AUTOHEAL_DEFAULT_STOP_TIMEOUT: 30
TZ: ${TZ:-Europe/Zurich}
volumes: volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro - /var/run/docker.sock:/var/run/docker.sock
# ---------------------------------------------------------------------------
# Discovery: the Explo image, driven by discovery/discover.py
# ---------------------------------------------------------------------------
explo: explo:
image: ghcr.io/lumepart/explo:v1.1.0 image: ghcr.io/lumepart/explo:v1.2.0 # discover.py writes Explo's v1.2 playlist cache; re-test before bumping
container_name: explo container_name: explo
entrypoint: ["/bin/sh", "/ainulindale/entrypoint.sh"]
environment: environment:
- TZ=${TZ:-Europe/Zurich} TZ: ${TZ:-Europe/Zurich}
- WEB_UI=false # PUID: ${PUID:-99} # opt-in: run as a normal user instead of root.
- EXECUTE_ON_START=false # PGID: ${PGID:-100} # chown $APPDATA_PATH/explo and $MUSIC_PATH/explo first.
- WEEKLY_EXPLORATION_SCHEDULE=15 0 * * 2
- WEEKLY_EXPLORATION_FLAGS=--playlist=weekly-exploration --persist # --- schedule -----------------------------------------------------------
- WEEKLY_JAMS_SCHEDULE=30 0 * * 1 DISCOVERY_SCHEDULE: ${DISCOVERY_SCHEDULE:-15 0 * * *}
- WEEKLY_JAMS_FLAGS=--playlist=weekly-jams --persist RUN_ON_START: ${RUN_ON_START:-false}
- DAILY_JAMS_SCHEDULE=15 1 * * * EXPLO_WEB_UI: ${EXPLO_WEB_UI:-false}
- DAILY_JAMS_FLAGS=--playlist=daily-jams --persist
# --- sources ------------------------------------------------------------
DISCOVERY_SERVICE: listenbrainz
LISTENBRAINZ_DISCOVERY: playlist
LISTENBRAINZ_USER: ${LISTENBRAINZ_USER}
LISTENBRAINZ_USER_TOKEN: ${LISTENBRAINZ_USER_TOKEN:-}
LASTFM_USER: ${LASTFM_USER:-}
LASTFM_API_KEY: ${LASTFM_API_KEY:-}
LASTFM_MODE: ${LASTFM_MODE:-always}
LASTFM_STATIONS: ${LASTFM_STATIONS:-recommended}
LASTFM_TRACKS: ${LASTFM_TRACKS:-30}
# --- retention ----------------------------------------------------------
KEEP_WEEKS: ${KEEP_WEEKS:-2}
PRUNE_FILES: ${PRUNE_FILES:-true}
KEEP_FAVOURITES: ${KEEP_FAVOURITES:-true}
PRUNE_LEGACY_JAMS: ${PRUNE_LEGACY_JAMS:-false}
# --- Navidrome ----------------------------------------------------------
EXPLO_SYSTEM: subsonic
SYSTEM_URL: ${NAVIDROME_URL}
SYSTEM_USERNAME: ${NAVIDROME_USERNAME}
SYSTEM_PASSWORD: ${NAVIDROME_PASSWORD}
PLAYLISTNAME_FORMAT: week # the pruner recognises playlists by this naming
USE_SUBDIRECTORY: "true" # one folder per playlist, which is what gets pruned
# --- downloading --------------------------------------------------------
DOWNLOAD_SERVICES: ${DOWNLOAD_SERVICES:-slskd,youtube}
SLSKD_URL: http://gluetun:5030 # slskd has no address of its own, it answers on gluetun's
SLSKD_API_KEY: ${SLSKD_API_KEY}
SLSKD_MIGRATE_DOWNLOADS: "true"
REQUIRE_SLSKD: ${REQUIRE_SLSKD:-true}
EXTENSIONS: ${EXTENSIONS:-flac,wav,mp3}
MIN_BITRATE: ${MIN_BITRATE:-320}
MIN_BIT_DEPTH: ${MIN_BIT_DEPTH:-16}
TRACK_EXTENSION: mp3
YOUTUBE_API_KEY: ${YOUTUBE_API_KEY:-}
SLEEP: 2
LOG_LEVEL: ${LOG_LEVEL:-INFO}
volumes: volumes:
- ${APPDATA_PATH}/explo/.env:/opt/explo/.env - ./discovery:/ainulindale:ro
- ${APPDATA_PATH}/explo/config:/opt/explo/config - ${APPDATA_PATH}/explo/config:/opt/explo/config # playlist cache + orchestrator state, must persist
- ${MUSIC_PATH}/explo:/data - ${MUSIC_PATH}/explo:/data
- ${MUSIC_PATH}:/slskd - ${SLSKD_DOWNLOADS_PATH:-${MUSIC_PATH}/slskd}:/slskd
# ports:
# - 7288:7288 # only with EXPLO_WEB_UI=true
healthcheck:
test: ["CMD-SHELL", "pgrep crond >/dev/null"]
interval: 5m
timeout: 5s
restart: unless-stopped restart: unless-stopped
lidarr: lidarr:
image: lscr.io/linuxserver/lidarr:latest image: lscr.io/linuxserver/lidarr:latest
container_name: lidarr container_name: lidarr
environment: environment:
- PUID=${PUID:-99} PUID: ${PUID:-99}
- PGID=${PGID:-100} PGID: ${PGID:-100}
- TZ=${TZ:-Europe/Zurich} TZ: ${TZ:-Europe/Zurich}
volumes: volumes:
- ${APPDATA_PATH}/lidarr:/config - ${APPDATA_PATH}/lidarr:/config
- ${MUSIC_PATH}:/music - ${MUSIC_PATH}:/music
ports: ports:
- 8686:8686 - 8686:8686
restart: unless-stopped restart: unless-stopped
-48
View File
@@ -1,48 +0,0 @@
# =========================================================
# Ainulindale — .env.example
# Copy this file to .env and fill in your values
# =========================================================
# --- Paths ---
APPDATA_PATH=/mnt/user/appdata
MUSIC_PATH=/mnt/user/data/media/music
# --- Unraid user/group (for lidarr) ---
PUID=99
PGID=100
# --- VPN (ProtonVPN WireGuard) ---
WIREGUARD_PRIVATE_KEY=your_wireguard_private_key_here
WIREGUARD_ADDRESSES=10.x.x.x/32
VPN_COUNTRY=Switzerland
TZ=Europe/Zurich
# --- VPN port forwarding (optional, requires a P2P-capable ProtonVPN plan) ---
# Leave as "off" unless you specifically need port forwarding for slskd.
VPN_PORT_FORWARDING=off
# =========================================================
# Explo config — copy the block below into a separate
# file at: $APPDATA_PATH/explo/.env
# =========================================================
# --- Discovery ---
DISCOVERY_SERVICE=listenbrainz
LISTENBRAINZ_USER=your_listenbrainz_username
LISTENBRAINZ_DISCOVERY=playlist
# --- Navidrome (Subsonic API) ---
EXPLO_SYSTEM=subsonic
SYSTEM_URL=http://192.168.1.x:4533
SYSTEM_USERNAME=your_navidrome_username
SYSTEM_PASSWORD=your_navidrome_password
# --- Downloaders (slskd first, YouTube fallback) ---
DOWNLOAD_SERVICES=slskd,youtube
TRACK_EXTENSION=mp3
# --- slskd ---
SLSKD_URL=http://slskd:5030
SLSKD_API_KEY=your_slskd_api_key_min_16_chars
MIGRATE_DOWNLOADS=true
RENAME_TRACK=true
# --- Quality gates (FLAC/WAV preferred, 320kbps minimum) ---
EXTENSIONS=flac,wav,mp3
MIN_BITRATE=320
MIN_BIT_DEPTH=16
# --- YouTube fallback (optional) ---
# YOUTUBE_API_KEY=your_youtube_data_api_key
# --- Misc ---
SLEEP=2
LOG_LEVEL=INFO
-27
View File
@@ -1,27 +0,0 @@
soulseek:
username: your_soulseek_username
password: your_soulseek_password
web:
authentication:
username: your_chosen_web_username
password: your_chosen_web_password
api_keys:
explo_key:
key: your_api_key_minimum_16_chars_long # paste this into .env as SLSKD_API_KEY
role: readwrite
cidr: 0.0.0.0/0,::/0
directories:
incomplete: /downloads/incomplete
downloads: /downloads
transfers:
download:
slots: 50
upload:
slots: 20
shares:
directories:
- /music
+53
View File
@@ -0,0 +1,53 @@
#!/bin/bash
# slskd's stock healthcheck only asks "is the web server up?". slskd can sit
# there for days, web UI fine, logged out of Soulseek, and Docker calls it
# healthy, so autoheal never acts. This check asks the question that matters:
#
# web API dead -> unhealthy (restart)
# logged in to Soulseek -> healthy
# logged out, gluetun unreachable -> unhealthy: gluetun was restarted and this
# container is stuck in its old, dead network
# namespace. Only a restart re-attaches it.
# logged out, VPN tunnel down -> healthy: nothing slskd can do, gluetun heals
# itself and slskd's VPN integration reconnects.
# logged out, VPN fine -> ask slskd to reconnect, report unhealthy.
# Docker needs several failures in a row before it
# flags the container, so a reconnect that works
# never causes a restart.
PORT="${SLSKD_HTTP_PORT:-5030}"
KEY="${SLSKD_API_KEY##*;}" # tolerate the "role=..;cidr=..;key" form
GLUETUN="${SLSKD_VPN_GLUETUN_URL:-http://localhost:8000}"
API="http://localhost:${PORT}"
get() { wget -q -T 5 -t 1 -O - "$@"; }
get "${API}/health" >/dev/null || { echo "slskd web API is not responding"; exit 1; }
# without an API key we cannot look any deeper; behave like the stock check
[ -n "$KEY" ] || { echo "ok (SLSKD_API_KEY not set, Soulseek state not checked)"; exit 0; }
state="$(get --header "X-API-Key: ${KEY}" "${API}/api/v0/server")" \
|| { echo "cannot read Soulseek state from the API (wrong SLSKD_API_KEY?)"; exit 1; }
if jq -e '.isConnected and .isLoggedIn' >/dev/null 2>&1 <<<"$state"; then
echo "ok: $(jq -r '.state' <<<"$state")"
exit 0
fi
if [ "${SLSKD_VPN:-false}" = "true" ]; then
vpn="$(get --header "X-API-Key: ${SLSKD_VPN_GLUETUN_API_KEY:-}" "${GLUETUN}/v1/publicip/ip")" \
|| { echo "gluetun control server unreachable: stale network namespace, restart required"; exit 1; }
if [ -z "$(jq -r '.public_ip // empty' <<<"$vpn" 2>/dev/null)" ]; then
echo "VPN tunnel is down, waiting for gluetun"
exit 0
fi
fi
if jq -e '.isConnecting or .isLoggingIn' >/dev/null 2>&1 <<<"$state"; then
echo "Soulseek login in progress"
else
get --method=PUT --body-data='' --header "X-API-Key: ${KEY}" "${API}/api/v0/server" >/dev/null
echo "logged out of Soulseek ($(jq -r '.state' <<<"$state")), reconnect requested"
fi
exit 1
+27
View File
@@ -0,0 +1,27 @@
# slskd tuning. Copy to $APPDATA_PATH/slskd/slskd.yml
#
# No secrets in here: the Soulseek login, web UI login, API key, download
# folders and the gluetun integration are all set from the stack's .env through
# environment variables in docker-compose.yml.
#
# Careful: slskd lets this file WIN over environment variables. If you keep an
# older slskd.yml that still has soulseek:/web:/directories: blocks, those old
# values silently override .env. Start from this file instead.
shares:
directories:
- /music
# never offer half-finished downloads to other users
# (adjust if you changed SLSKD_DOWNLOADS_PATH to somewhere outside the default)
- "!/music/slskd/incomplete"
transfers:
download:
slots: 50
retry: # new in slskd 0.26: failed downloads retry on their own
partial: resume
attempts: 3
delay: 5000
max_delay: 60000
upload:
slots: 20