Quinta/Ainulindale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

reworked it all

Browse Source
This commit is contained in:
Quinta
2026-09-18 18:13:05 +02:00
parent fcd5aaeb6c
commit d7ef929160
8 changed files with 1199 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+113 -141
View File
@@ -10,204 +10,176 @@ No Spotify. No YouTube Music. No ads. Just your music, your server, your rules.
## What This Stack Does
- **Streams** your music library to any device via a Spotify-like interface
- **Automatically discovers** new music weekly based on your listening habits
- **Downloads** discovered tracks at high quality (FLAC/WAV preferred, 320kbps minimum) from the Soulseek network
- **Scrobbles** everything you listen to Last.fm and ListenBrainz
- **Routes all P2P traffic through a VPN** so your real IP is never exposed
---
- **Streams** your library to any device through Navidrome
- **Discovers** new music every week from **two independent sources**: ListenBrainz Weekly Exploration and Last.fm recommendations. If one of them is down, the other still delivers
- **Downloads** the discoveries from Soulseek at high quality (FLAC/WAV preferred, 320 kbps minimum), through a VPN
- **Keeps only this week and last week** of discoveries. Older playlists and their files are removed, except tracks you starred or added to a playlist of your own
- **Heals itself**: a Soulseek session that drops is reconnected, and a stuck container is restarted, without you noticing
## Components
| Component | Role |
|---|---|
| **Navidrome** | Music server — indexes your library, streams to clients |
| **Substreamer** | Mobile app (iOS/Android) — plays music from Navidrome |
| **Pangolin** | Reverse proxy — exposes Navidrome securely to the internet |
| **Gluetun** | VPN gateway container — all Soulseek traffic exits via ProtonVPN |
| **slskd** | Soulseek client — searches and downloads music from the P2P network |
| **autoheal** | Watches slskd/gluetun health status and force-restarts them if they hang |
| **Explo** | Discovery engine — connects ListenBrainz recommendations to slskd |
| **ListenBrainz** | Open-source scrobbler + recommendation engine |
| **Last.fm** | Secondary scrobbler for stats and social features |
| **Navidrome** | Music server (installed separately, e.g. Unraid Community Apps) |
| **Gluetun** | VPN gateway. All Soulseek traffic exits via ProtonVPN |
| **slskd** | Soulseek client, living inside gluetun's network |
| **autoheal** | Restarts gluetun / slskd when Docker marks them unhealthy |
| **Explo** | Searches slskd, downloads, builds the Navidrome playlist |
| **discovery/discover.py** | Runs inside the Explo container and decides *what* Explo imports and *when*; adds Last.fm, outage handling and retention |
| **Lidarr** | Optional, for whole-album grabs |
---
## How Discovery Works
## Prerequisites
- Unraid NAS (or any Linux server with Docker)
- ProtonVPN account (paid, for P2P/WireGuard support)
- Soulseek account — free at [slsknet.org](https://www.slsknet.org)
- ListenBrainz account — free at [listenbrainz.org](https://listenbrainz.org)
- Last.fm account — free at [last.fm](https://www.last.fm)
- Last.fm API key — free at [last.fm/api/account/create](https://www.last.fm/api/account/create)
- Substreamer app on your phone
- Pangolin or any reverse proxy for external access
---
## Folder Structure
Once a day (`DISCOVERY_SCHEDULE`, default 00:15) the orchestrator runs a **tick**. A tick only does work that is still missing, so it is safe at any frequency and after any restart:
```
/mnt/user/appdata/
├── gluetun/ ← VPN state (auto-populated)
├── slskd/
│ └── slskd.yml ← slskd config (copy from repo)
└── explo/
├── .env ← Explo config (copy from .env.example)
└── config/ ← Explo's playlist cache + cover art (auto-populated, must persist)
/mnt/user/data/media/music/
├── incomplete/ ← In-progress downloads (Navidrome ignores these)
├── explo/ ← Explo-managed discovery downloads
└── (your library)/ ← Everything else
tick
├─ ListenBrainz: is there a Weekly Exploration playlist I have not imported yet?
│ yes → check slskd is logged in → Explo imports it → "Weekly-Exploration-2026-Week38"
│ no → nothing to do LB down → note it, try again next tick
├─ Last.fm: does this week's playlist exist yet?
│ no → pull recommendations → hand them to Explo → "Lastfm-Recommended-2026-Week38"
└─ Prune: keep the newest KEEP_WEEKS per source, remove the rest
```
Things worth knowing:
- **ListenBrainz is tracked by playlist ID, not by calendar.** If ListenBrainz publishes late, or is down on Tuesday, the playlist is simply picked up on the first tick after it appears. If it never appears, last week's playlist is *not* re-imported under a new name. A playlist that keeps failing is attempted 3 times, then left alone.
- **Last.fm recommendations** come from the same feed the Last.fm website player uses (`last.fm/player/station/user/<you>/recommended`). It is public, needs no login, and returns Last.fm's own picks of artists you have not played. It is not an official API, so with `LASTFM_API_KEY` set there is a second path built purely on the official API (artists similar to your last three months, minus everyone you already know). Tracks recommended in earlier weeks are remembered and not repeated.
- **`LASTFM_MODE`**: `always` gives you two playlists a week; `fallback` only steps in, from Wednesday on, in weeks where ListenBrainz has produced nothing; `off` disables it.
- **slskd preflight.** Before Explo starts, the orchestrator checks that slskd is really logged in to Soulseek and nudges it if not. If it stays down, the run is postponed to the next tick. Without this, one bad night means fifty YouTube rips instead of fifty FLACs.
- **No jams.** Weekly Jams and Daily Jams are gone. `PRUNE_LEGACY_JAMS=true` cleans up what the old setup left behind.
### Retention
With `KEEP_WEEKS=2` you always have this week's and last week's playlist per source. On each tick, anything older goes: the Navidrome playlist and the matching folder under `music/explo/`.
Before a folder is deleted, every file in it is checked against your **starred tracks and every playlist that is staying**. Matches are moved to `music/explo/Keepers/` instead of deleted, so starring a track is how you say "keep this one". Only folders the pipeline created (`Weekly-Exploration-*`, `Custom-Lastfm-*`) are ever touched.
Preview it first:
```bash
docker exec explo python3 /ainulindale/discover.py prune --dry-run
```
`PRUNE_FILES=false` removes old playlists but leaves all files on disk.
## Why slskd Stays Connected Now
The old setup had three gaps, which together explain "slskd drops and autoheal does nothing":
1. **slskd's healthcheck never looked at Soulseek.** It only asks whether the web server answers (and the image gives itself a 60 minute start period). Logged out of Soulseek with the UI up counts as healthy, so autoheal had nothing to act on. `slskd/healthcheck.sh` replaces it and reports the Soulseek login state. On a drop it first asks slskd to reconnect; only three failed checks in a row lead to a restart.
2. **A gluetun restart strands slskd.** slskd borrows gluetun's network namespace. When gluetun restarts, slskd is left holding the old, dead one: localhost still works (so the old check passed) but nothing reaches the internet. The new check notices that gluetun's control server has vanished from localhost and fails, and autoheal restarts slskd into the new namespace.
3. **slskd did not know about the VPN.** slskd has a native gluetun integration that was not switched on. It now polls gluetun every few seconds, drops the Soulseek session when the tunnel drops, logs in again when it is back, and applies the forwarded port by itself when port forwarding is on.
When the tunnel itself is down, the healthcheck stays green on purpose: restarting slskd cannot fix a VPN, and gluetun repairs its own tunnel.
One thing no healthcheck can fix: **do not log in with the same Soulseek account anywhere else** (Nicotine+, SoulseekQt on a desktop). The server kicks the older session, and the two clients will keep throwing each other out.
---
## Setup
### 1. Clone the repo
```bash
git clone https://github.com/yourusername/ainulindale
cd ainulindale
```
### 2. Create your config files
### 1. Clone and configure
```bash
git clone https://github.com/Quinta0/Ainulindale
cd Ainulindale
cp .env.example .env
```
Fill in `.env` with your WireGuard private key, paths, and VPN country.
Everything lives in that one `.env`: VPN key, Soulseek login, Navidrome login, ListenBrainz and Last.fm names, retention. Generate the two internal keys with `openssl rand -hex 24`.
Copy `slskd.yml` to your appdata folder:
```bash
cp slskd.yml /mnt/user/appdata/slskd/slskd.yml
```
Run `docker compose` from this folder. The compose file mounts `./discovery` and `./slskd/healthcheck.sh` by relative path.
Fill in your Soulseek credentials and choose a web UI username/password.
Copy the Explo env block from `.env.example` into a separate file:
```bash
cp .env.example /mnt/user/appdata/explo/.env
```
Fill in your ListenBrainz username, Navidrome URL/credentials, and slskd API key.
### 3. Create required folders
### 2. Folders
```bash
mkdir -p /mnt/user/appdata/{gluetun,slskd,explo/config}
mkdir -p /mnt/user/data/media/music/{explo,incomplete}
APPDATA_PATH=/mnt/user/appdata; MUSIC_PATH=/mnt/user/data/media/music # same as in .env
mkdir -p $APPDATA_PATH/{gluetun,slskd,explo/config}
mkdir -p $MUSIC_PATH/{explo,slskd/incomplete}
touch $MUSIC_PATH/slskd/incomplete/.ndignore # Navidrome skips half-downloaded files
cp slskd/slskd.yml $APPDATA_PATH/slskd/slskd.yml
```
### 4. Get your ProtonVPN WireGuard key
```
$APPDATA_PATH/
├── gluetun/
├── slskd/slskd.yml ← tuning only, no secrets
└── explo/config/ ← Explo cache + ainulindale-state.json (must persist)
1. Log into [account.proton.me](https://account.proton.me) → VPN → Downloads
2. Select **WireGuard** protocol and a **P2P-capable server**
3. Generate the config and copy the `PrivateKey` value into your `.env`
### 5. Start the stack
```bash
# Start VPN first and verify it connects
docker compose up -d gluetun
docker compose logs -f gluetun
# Look for: "Public IP address is x.x.x.x"
# Then start slskd
docker compose up -d slskd
$MUSIC_PATH/
├── slskd/ ← slskd downloads (manual ones show up in Navidrome from here)
│ └── incomplete/
├── explo/
│ ├── Weekly-Exploration-2026-Week38/
│ ├── Custom-Lastfm-2026-38/
│ └── Keepers/ ← rescued favourites
└── (your library)
```
Open `http://your-nas-ip:5030`, log in with your slskd credentials, go to **Options → API Keys**, generate a key and paste it into `/mnt/user/appdata/explo/.env` as `SLSKD_API_KEY`.
### 3. ProtonVPN key
### 6. Install Navidrome via Unraid Community Apps
[account.proton.me](https://account.proton.me) → VPN → Downloads → WireGuard, pick a P2P server, copy `PrivateKey` into `.env`.
Add these environment variables in the Navidrome container template:
### 4. Navidrome
| Variable | Value |
|---|---|
| `ND_LASTFM_APIKEY` | your Last.fm API key |
| `ND_LASTFM_SECRET` | your Last.fm shared secret |
| `ND_LASTFM_ENABLED` | `true` |
| `ND_LASTFM_APIKEY` / `ND_LASTFM_SECRET` | your Last.fm API key and secret |
| `ND_SCANSCHEDULE` | `1m` |
### 7. Start Explo and autoheal
Link both Last.fm and ListenBrainz in Navidrome's user settings so both services learn from what you play.
### 5. Start and verify
```bash
docker compose up -d explo autoheal
docker compose up -d
docker compose ps # gluetun and slskd should turn "healthy"
docker inspect slskd --format '{{range .State.Health.Log}}{{.Output}}{{end}}' | tail -1
# → ok: Connected, LoggedIn
docker exec explo python3 /ainulindale/discover.py status
docker exec explo python3 /ainulindale/discover.py lastfm-preview # what Last.fm would give you
docker exec explo python3 /ainulindale/discover.py run # a full tick now, instead of waiting for 00:15
```
Unlike the rest of the stack, Explo doesn't need any external scheduler (no Unraid User Scripts, no cron). It runs continuously and schedules Weekly Exploration and Daily Jams itself via `WEEKLY_EXPLORATION_SCHEDULE` / `DAILY_JAMS_SCHEDULE` in `docker-compose.yml`, and `EXECUTE_ON_START=false` means restarting or updating the container never triggers an extra, out-of-schedule run. This is what keeps it from downloading the same week twice — see [Troubleshooting](#troubleshooting) below for the full explanation.
## Migrating From the Previous Version
`autoheal` also just runs continuously in the background; it watches slskd and gluetun and restarts either one automatically if Docker marks it unhealthy.
1. `docker compose down`
2. **Replace `slskd.yml`.** slskd lets the YAML file override environment variables, so an old file with `soulseek:`, `web:` or `directories:` blocks silently wins over `.env`. Back it up, copy the new one in.
3. Move your settings from `appdata/explo/.env` into the root `.env` (names differ slightly, see `.env.example`). The old file is no longer read.
4. Set `GLUETUN_API_KEY` and `SLSKD_API_KEY`. Gluetun's control port 8000 is no longer published on the host; nothing outside the stack needs it.
5. New downloads land in `music/slskd/`. Existing files stay where they are. Set `SLSKD_DOWNLOADS_PATH=$MUSIC_PATH` to keep the old behaviour.
6. `docker compose up -d`, then run `prune --dry-run` (above). The first real prune will remove every `Weekly-Exploration-*` week except the newest two, so **star what you want to keep first**. Set `PRUNE_LEGACY_JAMS=true` for one run to clear the old jams playlists and folders.
---
## How Discovery Works
```
You listen in Substreamer
↓
Navidrome scrobbles to Last.fm + ListenBrainz
↓
ListenBrainz builds your taste profile over time
↓
Every Monday night: ListenBrainz generates Weekly Exploration playlist
↓
Tuesday 00:15: Explo pulls recommendations
↓
slskd searches Soulseek network (behind ProtonVPN)
↓
Downloads FLAC/WAV/320kbps+ to /mnt/user/data/media/music
↓
Navidrome scans every 1 minute → appears in Substreamer
↓
You discover new music 🎵
```
---
The first tick re-creates this week's ListenBrainz playlist once (it has no record of the old setup having imported it). Tracks already on disk are recognised, not downloaded again.
## Troubleshooting
### slskd hangs and needs a manual restart every ~24h
**slskd keeps restarting.** `docker inspect slskd --format '{{json .State.Health.Log}}'` shows what the healthcheck saw. "cannot read Soulseek state" means `SLSKD_API_KEY` is not the key slskd is using (old `slskd.yml` still in place?). "logged out of Soulseek" every time, with `docker logs slskd` mentioning another client, means the account is in use elsewhere.
This was almost always one of two things, both addressed in the current compose file:
**slskd is stuck in "Restarting".** It validates its settings on start and exits on the first bad one; `docker logs slskd --tail 40` names it. Usual causes: `PORT_FORWARDING` set to `on` instead of `true`, a missing `music/slskd/incomplete` folder, an `SLSKD_API_KEY` under 16 characters.
- **slskd itself getting stuck** (memory pressure / stuck transfers) — fixed by pinning `slskd/slskd:0.25.1`, which bundles a round of upstream fixes for stuck and failing transfers.
- **gluetun's tunnel silently hanging** — since slskd runs inside gluetun's network namespace, a frozen gluetun looks identical to a frozen slskd. Fixed by pinning `qmcgaw/gluetun:v3.41.1`, which resolves a healthcheck race condition that could make gluetun hang completely.
**slskd never logs in, log says it is waiting for the VPN.** `GLUETUN_API_KEY` is empty or differs between the two containers, or `PORT_FORWARDING=true` on a Proton plan/server without port forwarding.
Both containers already ship a Docker healthcheck, but **Docker does not restart a container just because it's unhealthy** — that gap is why a manual restart was needed. The new `autoheal` service closes it: it watches both containers (via the `autoheal=true` label) and force-restarts whichever one goes unhealthy, with no manual intervention.
**Tick says "slskd did not log in ... Skipping this run".** Working as intended; it retries on the next tick. `REQUIRE_SLSKD=false` lets runs go ahead on YouTube alone.
If slskd still misbehaves after this, check `docker ps` for its health status and `docker logs slskd`/`docker logs gluetun` before restarting manually.
**Last.fm gives 0 tracks.** Open `https://www.last.fm/player/station/user/<you>/recommended` in a browser. Empty JSON means the account has too few scrobbles, or the profile's listening data is private. Set `LASTFM_API_KEY` to enable the API-based path.
### Weekly Exploration downloaded multiple times
**Prune says it could not remove a folder.** Files from the old root-owned setup while Explo now runs with `PUID`/`PGID`. `chown -R` the `music/explo` folder.
The old setup ran two one-shot `explo-weekly` / `explo-daily` containers (`restart: "no"`, `EXECUTE_ON_START=true`) that were triggered by an external Unraid User Scripts cron. That design had two compounding problems:
1. Every container restart (a host reboot, a Docker update, the scheduler firing slightly out of sync) re-ran Explo immediately because of `EXECUTE_ON_START=true`, regardless of whether that week's playlist already existed.
2. Explo's playlist cache had nowhere to persist between runs (no `/opt/explo/config` volume), so each fresh container had no memory of "I already built this week's playlist" and would build it again.
The current compose runs a single, always-on `explo` container with its own internal cron (`WEEKLY_EXPLORATION_SCHEDULE` / `DAILY_JAMS_SCHEDULE`), `EXECUTE_ON_START=false`, and a persistent `/opt/explo/config` volume, plus the newer `v1.1.0` image which includes further upstream fixes for scheduled-job bugs. Together this means Explo only ever runs on its own schedule, and remembers what it already did across restarts.
**One-time cleanup:** this fix prevents *future* duplicates, but it won't retroactively merge the duplicate `Weekly-Exploration-2026-WeekXX` playlists already sitting in Navidrome. Delete the extras once, manually, from the Navidrome UI (or via its API) — new runs going forward should stay to one playlist per period.
---
**Upgrading Explo.** The Last.fm bridge writes Explo's playlist cache (`config/cache/custom-*.json`), an internal format. It is pinned to `v1.2.0`; after bumping, run `discover.py run` once and watch the log. If Explo ever ships Last.fm support itself ([issue #135](https://github.com/LumePart/Explo/issues/135)), the bridge can retire.
## Tips
- **ListenBrainz needs time** — it takes a few weeks of scrobbling before Weekly Exploration playlists are generated. Speed this up by importing your Google Takeout YouTube Music watch history via [ytm-extractor](https://community.metabrainz.org/t/ytm-extractor-import-youtube-music-listens-to-listenbrainz/707619).
- **Manual downloads** — search for any artist or album directly in the slskd web UI at `http://your-nas-ip:5030`. Downloads land straight in your Navidrome library.
- **Playlist import** — use [Soundiiz](https://soundiiz.com) to transfer playlists from YouTube Music/Spotify directly into Navidrome. Tracks already in your library get matched automatically.
- **Quality** — slskd is tried first for every download. YouTube is only used as a last resort fallback for tracks not found on Soulseek.
- **Sharing** — slskd shares your music library back to the Soulseek network. The more you share, the better your download priority from other peers.
- **ProtonVPN** — the WireGuard private key in Gluetun pins you to a specific server. Make sure to use a P2P-capable server when generating your WireGuard config on ProtonVPN's site.
- **ListenBrainz needs a few weeks** of scrobbles before Weekly Exploration appears. Last.fm recommendations work with whatever history your account already has, which makes it a good bridge in the meantime.
- **Manual downloads**: search in the slskd UI at `http://your-nas-ip:5030`; results land in `music/slskd/`.
- **Sharing**: slskd shares your library back. The more you share, the better your queue position with other peers.
- **Port forwarding** noticeably improves Soulseek results. If your Proton plan has it, set `PORT_FORWARDING=true` (one switch for gluetun and slskd; it must be `true`/`false`, slskd does not start on `on`/`off`).
---
## License
MIT — do whatever you want with it.
MIT. Do whatever you want with it.