Adam's blog

My new home data backup strategy

Mon, 23 Mar 2026 00:00:00 +0000

A few years back, I described how my off-site home data backup strategy worked. Since then, it was due for a big overhaul, which I have successfully completed. First, let’s go over how it was before and how I improved upon it.

Previous setup

In the past, my setup was fairly simple. I had one main server in the basement and one RPi in another house. Connected to the main server were two sets of HDDs – one for primary data, and a second for full backups. The RPi also had its own set of HDDs connected, but only for full backups. One crontab job rsynced data from the main HDDs to the backup HDDs, while a second job rsynced data from the main HDDs to the off-site RPi.

This setup worked well enough for multiple years, but it had two main drawbacks:

As the backup was done once a week, changes that happened more quickly would not be backed up anywhere.
The speed on the off-site RPi was so slow that the backup completed successfully only if there was not much new data added, effectively finishing only once in a few months.

It was evident that something needed to change; a week of work is a week of work, after all. You don’t want to lose it.

Current setup

My current setup is much more layered. The first stage is daily snapshots using my backup-prepare tool. This tool is able to create a full snapshot of the current state of a directory either via hardlinks or via BTRFS subvolumes, the latter being instant and the method I use. Every midnight, a read-only snapshot is created and kept for 8 days. Unless there is a hard drive failure, I lose at most a day’s work in case a restore is needed.

The second layer is a weekly data backup using borg. Borg is great at deduplicating data and never backs up anything that was already uploaded. After the first full backup is completed, every subsequent backup is much faster as it is only differential. This solution scales perfectly; it works for my VPSs with 10 GiB of HDD space up to my data disks with tens of TBs. I have created a custom wrapper (+ example config) that works great with the daily snapshots (each snapshot directory has a suffix with the current date, so borg always scans the files as new instead of using the cache). It uses bwrap to make it seem like the latest snapshot always has the same path. The added benefit of using borg over snapshots is that I never get “file changed during backup” warnings.

Because borg can work over SSH connections, the borg repository is not saved on the backed-up device itself, but instead on one of my VPSes. This VPS has my dropbear-backup Docker image running. The main advantage is that I can easily pair SSH keys with different subfolders, making the creation of a new account with restricted access (only for borg, sftp, or rsync) a breeze.

Still, we are talking about backing up tens of terabytes of data, which would cost a lot of money if it was all saved on the VPS disk. Instead, I use an rclone Docker plugin with an rclone-compatible storage provider mounted inside the dropbear backup container. This setup makes my storage virtually infinite, while the VPS itself has only a few GB of storage. A bonus is that I can put a virtual layer of encryption on top of the cloud storage using rclone’s crypt remote.

The cloud storage can be considered warm; if needed, I can recover the files quickly. However, if a ransomware attack hit the VPS, it could very well encrypt the whole warm storage, as it is not backed up itself. To mitigate this, every week the entire warm cloud storage is synced to my local Raspberry Pi with multiple external HDDs attached. Together, they have enough storage to keep the whole cloud storage with 2 months of weekly snapshots. As the RPi is limited by residential network speeds, I consider it cold storage, because recovering a lot of data from there would take several weeks at least.

However, we are still not done. Having only 2 months of history may not be enough for all my use cases, so I back up all the data from the RPi to a freezing cold Backblaze storage, which offers versioning and object locking, preventing a ransomware attack from destroying everything. Now we are done.

If this was a lot to take in, here is a diagram detailing the levels:

Diagram of my backup solution

Closing words

My personal setup may be overkill for many, but it still might be a nice pointer on where to start when trying to establish your own backup procedures. I opted to have more storage capacity and accept the possible downtime in case of a hard drive failure instead of some form of RAID. However, if you would like to sleep better at night, I would suggest RAID 1 with BTRFS, for example. This way you can keep all the other features, such as CoW and (almost) instant snapshots.

Honorable mentions

I also considered other services for a while, but decided they were not the right match for me. That does not mean they won’t be the best match for you, though:

rsync.net - access via standard Linux tools, but a little pricey
Hetzner Storage - EU-based fast storage with snapshots and multiple protocol access

(CZ) Technický průvodce získáním pedagogického osvědčení

Sun, 08 Feb 2026 00:00:00 +0000

Pokud jste jako já, nejspíše znáte pojem pedagogické minimum. Pokud nejste jako já a tento pojem neznáte, může to znamenat dvě věci. Zaprvé buď nevíte, že pedagogické minimum je osvědčení, bez kterého nemůžete učit. A nebo, zadruhé, jste se začali zajímat o učení až po září 2023 a zjistili jste, že nic jako pedagogické minimum neexistuje!

Pedagogické minimum neexistuje!

Ono totiž to je tak, že po chvíli googlení se teda dozvídáte, že “pedagogické minimum však neexistuje” a že “Díky novele zmiňovaného zákona mohou od 1. 9. 2023 začít učit i zájemci bez pedagogické kvalifikace a vzdělání si doplnit během prvních třech let práce ve školství. Do začátku je dobré mít motivaci, možnost si udělat čas na učení a chuť postupně si doplnit pedagogické vzdělání.” (zdroj: zacniucit.cz, archiv).

Snímek obrazovky z zacniucit.cz

Takže bezva, první roky můžete učit i bez jakékohokoliv pedagogického vzdělání! No, takže se do toho učení pustíte, učíte rok, učíte druhý rok a naráz zjistíte, že vás to vlastně baví a chtěli byste pokračovat. Ale co teď? Na třetí rok si už ale něco doplnit musíte. Co je ale to něco?

Hledání správného kurzu

Šikovná stránka zacniucit.cz nám stále pomáhá: “A co se skrývá pod tou pedagogickou kvalifikací? Nemusí to být nutně 5 let studia. Možná jste tento typ studia znali pod už zaniklým názvem „pedagogické minimum“. Nahradilo jej doplňkové pedagogické studium (DPS).” Kde už ale bohužel není tak nápomocná, je vyhledání toho přesného kurzu, který by vám DPS zabezpečil.

Interaktivní průvodce kurzů DPS

No nic, čas ušpinit si ruce a přijít s nějakými kurzy starým dobrým internetovým vyhledáváním. Bydlíte v Brně jako já? Fajn! Máte dvě možnosti: Studium pedagogiky na MENDELU (17 900,- Kč; archív) nebo PdF MUNI (10 900 Kč/1 semestr, 3 semestry, archív) a máte hotovo během roku a půl. Nebo se vám možná nechce studovat rok a půl a nebo se vám možná nechce platit přes 15 000 Kč za celý kurz. Naštěstí existuje ještě jedna možnost – DVPP Petrklíč. Petrklíč nabízí kurz, který je hotový během půl roku, stojí 14 000,- Kč v základu a nebo můžete přímo požádat úřad práce o zaplacení celé částky. To zní ideálně, že? Jediná nevýhoda – od jara 2025 už neposkytují kurzy v Brně, ale jen a pouze v Praze. No co už, pořád to vypadá jako nejlepší možnost, pojďme se tedy pustit do žádosti o proplacení od úřadu práce.

Úřad práce

Petrklíč vypadá jako ideální možnost (až na to, že je pouze v Praze). Nejlevnější, nejkratší a pokud se přihlásíte přes projekt Úřadu práce, bude dokonce úplně zdarma! Jak tedy vypadá přihláška pro úřad práce? V mém případě měla cesta tři kroky. Je ale možné, a také v to doufám a věřím, že zkušenosti jiných se budou lišit. Každopádně moje tři kroky byly:

Krok první – zájemce o práci

Abyste mohli žádat o proplacení (re)kvalifikačního kurzu od úřadu práce, musíte se na úřadu práce zaregistrovat. Existují dvě úrovně registrace – uchazeč o práci a zájemce o práci. Jelikož mám i druhé, hlavní zaměstnání, zajímalo mě, kterou úroveň mám zvolit a jaké pro mne z toho plynou povinnosti. Po několika hodinách googlení a další hodince přímo na úřadu práce vyplynulo, že se chci stát zájemcem o práci. Proč? Protože uchazeč je někdo, kdo žádnou práci zrovna nemá a z toho vyplývají další povinnosti. Na druhou stranu být zájemce o práci registrovaný na úřadu práce je zhruba to stejné, jako založit si účet na Seznamu.

Krok druhý – vyplnit formulář

Protože úřad práce je pokrokový, dovolí vám všechno vyřídit online. Stačí se přihlásit bankovní identitou, rozkliknout správný kurz (na ten je odkaz přímo ze stránek Petrklíče) a vyplnit přihlášku (včetně vašeho vzdělání, zaměstnání, motivační dopis, atp.). Pak stačí jen tento formulář odeslat a čekat, než vám úřad práce pošle mail, abyste poslali všechny tyto údaje znovu.

Krok třetí – zamítnutí

Pokud vám, stejně jako mě, přijde čekání na rozhodnutí úřadu o proplacení kurzu dlouhé, můžete ve volné chvíli vždy nakouknout, jestli už nějaké to rozhodnutí nepřišlo. Rozhodně nečekejte, že vám snad úřad pošle nějaké upozornění, prostě si na to máte pamatovat sami. A tak když tak z dlouhé chvíle koukáte, možná se jednou dozvíte, že vám úřad vaši účast proplatit zamítl. Bohužel prý nejsem ohrožen ztrátou zaměstnání a zaplatit mi kurz a akreditaci na učitele by bylo nehospodárné. No budiž, sice jsem jako důvod ohrožení zaměstnání uváděl zákon, který mi ukládá tento kurz absolvovat, nebo již nebudu moci učit, podložený smlouvami o provedení práce, nicméně to je očividně slabý argument. Rozhodnutí úřadu je konečné a já vám ze celého srdce přeji, abyste měli více úspěchů s vyřízením žádosti než já.

Praha

No ale teď už zase zpátky do pozitiva. Pokud jste tady, nejspíš vám žádost o proplacení vyšla, nebo máte prostě silnou motivaci učit a kurz si zaplatíte tak jako tak. Tím, že jediná nabídka je nyní pouze v Praze, je čas vycestovat do našeho hlavního města. Velmi příjemné je, že kurz se v prezenční výuce vyučuje pouze kousek od Václaváku, takže v pohodě můžete jet vlakem ČD v pátek v 9h z Brna a pak vlakem v 14h zase hezky zpátky. Jen si určitě dejte pozor na místenky, ty vlaky bývají hodně narvané.

Kurz samotný

Kurz samotný je na moc pěkné lokalitě Školská 32, hned kousek za Václavákem. Pokud je ale vedro a nebo se chcete vyhnout Václaváku kvůli počtu turistů, doporučuji vzít šalinu/tramvaj 9 na Národní třídu a hned tam do obchoďáku Quadrio. Nejenže tady mají super fast foody od Hugo, přes Indii až po McDonalds’, je tady i stanice metra a proto je velká šance, že tuto budovu navštívíte častěji. Na druhou stranu, pokud by vám tento McDonalds’ nevyhovoval, ničeho se nebojte. Pěšky se odtud na adresu kurzu dostanete asi za 8 minut a cestou potkáte ještě další 2 pobočky této frančízy. Pokud byste ale chtěli jen někam na kávičku, budova kurzu je pěkně vybavená – mají tu i kuchyňku, kde si sami můžete připravit volně dostupnou rozpustnou kávu.

Týden první

Jak se dá čekat, týden první je poznávací. Uvítá vás nejspíše paní Iva Stratilová, velmi zkušená pedagožka, která vás provede úvodem do pedagogiky, forem a metod. Tento týden se rozhodně vyplatí přijít, protože všechnu probíranou látku také rovnou prakticky zkoušíte – a prakticky si vyzkoušet je prostě super!

Týden druhý

Ve druhém týdnu nás přivítal pan Edgar. Už od prvního pohledu velice veselý a energetický člověk, nicméně v přínosu nových dovedností a kompetencí nastal výrazný propad. Sic s námi za jeden víkend prošel téměř všechny závětečné otázky ke kurzu, všechen výklad byl prakticky a pouze výklad s občasným vtipem. Přečíst si dostupná skripta k předmětu by mělo nejspíše podobný výsledek. Co ale jeho výklad zachraňovalo byly příběhy z jeho života, často s pedagogikou spojené jen velmi vágně. Jestli ale slyšet příběhy ze života stojí za celovíkendovou cestu do Prahy nechám už na posouzení čtenáře.

Online výuka

Během letních prázdnin probíhala výuka téměř výhradně online. Všechny týdny, kterých jsem se zúčastnil vypadaly velmi podobně – v pátek od 14:00 do 18:00, o víkendu 8:30 – 12:30. Na začátku i na konci musí mít všichni povinně zapnutou kameru, aby byla pořízena fotka do evidence. V průběhu pak bylo kameru povoleno vypnout. Na všech přednášejících bylo znát, že je výuka baví a že mají povědomí o tom, co říkají. Obsahově jsme se pohybovali v základech psychologie a výchovy na úrovni střední školy (emoce, počitky apod.), vzdělávací systém v České Republice (školka, základní škola, …), odborná didaktika. Na posledním online víkendu jsem nebyl kvůli dřívějším závazkům, čímž jsem si také vyčerpal všechnu povolenou absenci. Mírnou nevýhodou bylo, že nejspíše nikdo nekoordinoval obsah prezentací jednotlivých přednášejících – každá prezentace, která se byť jen vzdáleně týkala didaktiky, obsahovala znovu stejný úvod do ní a jednotlivých typů, který vždy zabral minimálně půl hodiny.

Týden třetí

Po letních prázdninách zpátky v Praze. Tentokrát s dcerou paní Ivy Stratilové, Štěpánkou Stratilovou. Tématem víkendu byly kompetence, způsoby hodnocení a konflikty. Paní Štěpánka Stratilová působí jako mediátor, takže odbornostně byl celý víkend na vysoké úrovni. Zajímavé bylo, že výsledkem celého víkendu byl pocit, že učitel odborných předmětů je jedinou nadějí na žákovo spasení a že by měl se svým žákem řešit přímo všechny jeho osobní problémy. O tom, že existuje na škole výchovný poradce nebo metodik prevence, který má téměř jistě více kompetencí k vyřešení náročnějších situací, nepadlo ani slovo.

Týden čtvrtý

Chyběl jsem kvůli pracovní cestě, spotřeboval jsem tím všechnu povolenou absenci pro prezenční víkendy.

Týden pátý

Pátý týden byl ve znamení etické výuky. Byl velmi dobře připravený, nicméně, jak sama lektorka podotkla, materiály pro etickou výchovu končí devátou třídou. Přestože se jednalo o zajímavé téma, nebylo pro nikoho z nás zúčastněných příliš aplikovatelné. Navíc jsem se tohoto bloku účastnil nemocný, jelikož jsem si všechnu absenci vybral na pracovní cestu, takže jsem si jen přál, abych se mohl co nejdříve vrátit z Prahy domů.

Závěrečná práce a zkouška

Ukončení celého kurzu se skládalo ze závěrečné práce, její obhajoby a závěrečné zkoušky. Závěrečná práce se píše na jedno z předem zadaných témat, rozsahem se má pohybovat kolem 10 000 znaků (tedy krapet kratší než tento blogový příspěvek). Na závěrečnou práci máte dvě možnosti konzultace v rámci výuky, odevzdání je dva týdny před závěrečnou zkouškou. Obecně hodnotím zkušenost se závěrečnou prací jako pozitivní, byla mi schválena hned po odeslání na první konzultaci.

Závěrečná zkouška probíhala také hladce. Bylo cca 30 otázek, každá rozdělená ještě na podotázky s procentuální hodnotou. Aby byla zkouška úspěšně složena, musel zkoušený odpovědět na minimálně 80 % každé ze tří vylosovaných otázek. Atmosféra po dobu zkoušení byla dobrá, zkoušející se doptávali na doplňující otázky, takže spíše než strohé zkoušení a hledání mezer se jednalo o dialog. Pokud tedy člověku nějaký pojem vypadl, měl možnost si jej v rámci dialogu odvodit či dohledat v paměti.

Horší zkušenosti

Během průběhu celého kurzu se opakovalo pár ne-úplně v pořádku momentů. Předně komunikace ze strany vedoucích kurzu směrem k nám byla poměrně strohá – např. když se změnily termíny prezenčního kurzu, nebyl o tom zaslán žádný email. Jediný způsob, jak to zjistit, bylo přihlásit se do webového portálu a všimnout si upozornění. Protože mi nové termíny nevyhovovaly, podal jsem otázku, jestli se i nové termíny započítávají už i do tak šibeniční povolené absence a jaká témata budou nově probíraná ve který den. Trvalo několik týdnů, než jsem dostal odpověď s novým harmonogramem a nabídku, že pokud mi nové termíny nevyhovují, mohu absolvovat v původních termínech obsah stejný, nicméně s jinou studijní skupinou.

Dalším opakujícím se jevem byla averze k zasílání prezentací. Vysvětlení bylo takové, že vše potřebné bychom měli mít ve vypracovaných skriptech. To ale pokaždé nešlo dohromady s faktem, že některé zajímavé informace se vyskytly pouze v prezentaci a ve skriptech nebyly zmíněny vůbec. Velmi cením práci, kterou vypracování skript muselo zabrat, nicméně bych rád měl možnost mít jednoduchý přístup i k materiálům, které nejsou nezbytně nutné pro závěrečnou zkoušku.

Certifikát

Jako důkaz o absolvování kurzu mi po krátké době po zkoušce dorazil elektronický certifikát. Jedná se o jedinou formu osvědčení, kterou za kurz dostanete. Proto je také potřeba se ujistit, že vydaný certifikát je důvěryhodný – ideálně obsahuje kvalifikovaný elektronický podpis, který je platný v celé EU. Takový certifikát se dá nyní získat už i online. Původní certifikát, který mi dorazil, takový ale nebyl. Byl totiž podepsaný tzv. self-signed certifikátem, který si může vygenerovat kdokoliv a kdykoliv, takže nenesl žádnou právní hodnotu, na což mě můj PDF prohlížeč neopomněl hlasitě upozornit. Příjemně mě potěšilo, když po mé žádosti přišel certifikát s plně ověřitelným podpisem. Věřím, že tento typ podpisu bude využíván i nadále, nicméně přesto doporučuji po obdržení certifikátu provést kontrolu.

Finanční okénko

Zajímá vás, na kolik mne nakonec kurz vyšel? Po sečtení všech částek je cena následující:

Kategorie	Cena
Kurz	14 000, 00 Kč
Ubytování	9 935,00 Kč
Cestování	2 970,00 Kč
Strava	1 896,00 Kč
Ostatní	60,00 Kč
Celkem	28 861,00 Kč

Nutno dodat, že jsem se sice snažil zbytečně nerozhazovat, nicméně taky jsem nešel s cenou tak nízko jak to šlo – mohl bych například celý víkend vždy jíst jen pečivo a vyšlo by to nakonec o pár stovek levněji. Pokud by vás zajímalo opravdu podrobné rozúčtování, můžete si jej stáhnout tady.

Závěr

Přestože tento text může místy působit trochu sarkasticky, mým cílem není očerňovat jednotlivé aktéry. Naopak, rád bych upozornil na situaci, kdy bez velmi silné motivace nejspíše uchazeč o pedagogickou praxi raději celou učitelskou dráhu vzdá. Je mi doopravdy líto, že v momentě, kdy chybí učitelé všech možných předmětů, české prostředí ještě ztěžuje podmínky pro získání pedagogického osvědčení na všech možných frontách. Sám jsem nejednou zvažoval, jestli mi celý tento proces za ty nervy doopravdy stojí. Mohu tedy jen doufat, že do budoucna bude situace jednodušší. Ve výsledku jsem alespoň rád, že zkouška nebyla extra náročná a mám osvědčení o jejím absolvování.

Bonus: Pražský motoráček

Pokud stejně jako já budete bydlet v Apartments V Bloku (archív) v Jinonicích, možná zjistíte, že o víkendu jezdí kolem vás historický vlak Pražský motoráček (archív). A taky, že když v sobotu končíte s kurzem kolem jedné, stihnete v pohodě poslední jízdu motoráčku v 15:07 z Prahy-Jinonice. Přestože se jedná o historický vlak, platí na něj normálně klasická PID jízdenka. Konkrétně, co se vám může podařit s 90 minutovou jízdenkou je následující:

Nastoupit 15:07 v zastávce Praha-Jinonice
Vystoupit 15:12 v zastávce Praha-Stodůlky
Projít se pěšky lesem 4km kolem soch do Praha-Cibulka
Nastoupit 16:18 v zastávce Praha-Cibulka
Vystoupit 16:27 v zastávce Praha-Žvahov
Projít se pěšky 5km na Děvín a kolem výběhu koně převalského zpátky domů

Je to moc hezký výlet plný krásných panoramat. Jen si nezapomeňte dost vody a opalováku!

Historický Pražský motoráček

2025 Recap: Books & Podcasts

Tue, 13 Jan 2026 00:00:00 +0000

As the 2025 has come to an end, let’s recap on some of the books I have read this year and what podcasts did I listen to.

Books

The Consuming Fire (The Interdependency, #2) by John Scalzi
- The second part of the series was more focused on smaller scale of the world, but I really liked the development and further word building. Hooked for the third part.
- Book 1 - very cool, hooked imediatelly
- Book 2 - a lot more graphic, hope to get connection back to the original plot line
- Book 3 - nice finish, I like how it ended
Old Man’s War Series by John Scalzi
- Book 1 - great word building, I hope to see more of the Perry character
- Book 2 - a lot more graphic, hope to get connection back to the original plot line
- Book 3 - this worked out great, I now understand book 1 and 2
- Book 4 - well now I understand book 3 and I love it much more
- Book 5 - happy to see that B-Team is doing its own thing
All of Murderbot by Martha Wells
- Loved it, would want more. However the fact that the release order of books is not chronological storywise got me confused for a few moments.
The Android’s Dream by John Scalzi
- Wanted more android sci-fi, did not get it, but was not dissapointed
Agent to the Stars by John Scalzi
- Second book about aliens communicating by smell in a row? Very predictable, but loved it nontheless
Murder by Other Means (The Dispatcher, #2) by John Scalzi
- Good read, but maybe better if I would read the first book first?
The Dispatcher by John Scalzi
- Well now I know I enjoyed the second book more without knowing the first one first. Still good
Syndikát by Leoš Krysa
- A Czech Sci-Fi with good narration with a toons of Czech references? Count me in! Loved the whole story except for the last quarter, which was a little weaker.
Head-on (Lock In, #2) by John Scalzi
- Again a little confused, loved it in the end, but maybe better if I read the first book first?
Lock-in by John Scalzi
- And again, I liked the second book better. Glad that I have started with it, but still enjoyable.
2001: A Space Odyssey by Arthur C. Clarke
- Never read this classic before, nor seen the movie. A little more confusing story and shorter than I expected, but still very enjoyable
A Very Scalzi Christmas by John Scalzi
- Read this one just before Christmas, laugh out loud, definitely would reccomend

Podcasts

Darknet Diaries
- Discovered recently, large backlog with great content and goosebumps-producing stories
The Layover
- Weekly recap and behind the scenes of what happened during Jet Lag: The Game with some entertaining in-between seasons stuff
It’s Probably (not) Aliens
- Soo many episode when starting now, very great explanations of why Aliens were not the cause, but super-interesting history was it instead

Meeting a scammer in Minecraft

Sat, 18 Jan 2025 00:00:00 +0000

A few days back my friends and I were searching for a new Minecraft server to play the Bedwars gamemode on. Little did we know what rollercoaster it would take us and that we had a new old friend already waiting for us.

Joining a new server

Most of the time when playing Minecraft minigames competitively, since the closing of Mineplex, we are playing on a Hypixel server – the most popular server there is, which comes with a super-scary anti-cheat or anti-scam watchdog. However, this server is running on the pre-1.9 version, which features an old Minecraft combat system, while we prefer the newer one. This is why we decided to venture and find a new server and we landed on Cubecraft.

Immediately after joining my friend was greeted by some “old friend”! Said “old friend” tried to get in touch again after a few years they played together. They even knew they were from Czechia!

Being welcomed immediately after joining a server

Welcoming chat - Scammer (red), my friend (blue)

At this point, we were hooked and wanted to see where this friend went.

Quick Q&A:

Why was my friend approached? A friend of mine has a special cosmetic item in Minecraft and accounts with this special cosmetic item – a cape – are being sold for a lot of money, so stealing them is a good business. He is contacted quite often with offers and regular phishing, however, meeting someone pretending to be his friend was first for us all.
How the scammer knew my friend is from Czechia? They have Czech flag on their NameMC profile.
How was the scammer able to single out my friend seconds after joining a new server? No idea, this was honestly super impressive.

Playing with a new old friend

We have played a few rounds, in which the scammer was super nice to us – they gave us diamonds even when they were on different teams and never attacked us (which is a bannable offence in this game, FYI) and even invited one more of their friends into the group. We all were having fun! You know what is more fun? Calling with friends while we are playing.

After a few rounds, they asked us to join their Discord server for a call - Scammer (red), scammer’s friend (pink), my friend (blue)

And this was the other shoe – an invitation to a Discord server discord[.]gg/furnace. All we need to join this server and verify our Minecraft accounts to be able to join the voice call. Interestingly, when we invited them to our own Discord server they did not join and instead pressured us to join their server. Weird, right?

The verification is nothing sinister, pinky promise - Scammer (red), scammer’s friend (pink), my friend (blue)

By the way, this is where the friend of the scammer came into play – they were supposed to show us how easy is to join the Discord server.

Verifying Minecraft account

As stated above, when you join the public Discord server, you have to “verify” your account before you are able to do anything else. There is a Q&A about why the link is necessary and a big green button with the text Link account.

Verify your Minecraft account by giving a random Discord bot your Microsoft email and Minecraft username

After clicking the button, you find out that the verification itself is very easy. All you need to do is enter your Minecraft username and Microsoft account email (Minecraft username can be anything, the Microsoft account has to exist).

Last step of verification is a 6-digit code from your mail, easy, right?

Then you just fill in the code that gets to your email and you are verified!

My “verification code”, or is it?

Wait a second, do you see the code that arrived? This code is not for some verification, it is for a Microsoft account sign-in!

This is what the scam is all about

So the scam is supposed to get you to join their Discord server through their “verification” process that will steal your Microsoft account. And because your Minecraft account is tied to your Microsoft account, the scammer will gain full access to both accounts for the price of one password reset.

As the list of Discord server members is public, we can see how many people were possibly scammed. Unfortunately, there is no clear indication of how many of these are scammers, but given that the server population is currently approx 500 accounts, we can guess that at least some of them are victims.

Conclusion

This was the first time I have encountered this type of scam – a scammer posing as an old friend of yours asks you to join their Discord server with a slight catch. To join the Discord server, you have to “verify” your account which will lead to your Microsoft account being stolen. The main reason for this type of scam is not only the classic one but also to get high-value accounts that can be sold for a considerable amount of money.

As with many social engineering scams, the scammers are using psychological tactics where they are first pretending to be your friends to incentivise you to do what they want you to do. What is especially sickening is that given the main audience of Minecraft is kids, they may also be the main target of the scammers. They are able to do this with impressive speed as my friend was approached within seconds of joining a new Minecraft server.

We have of course reported the scammer both to Discord and Cubecraft. At least on Cubercraft, we can confirm that both their accounts were banned. This won’t stop them for long but hopefully will make their day a little worse.

2024 Recap: Books & Podcasts

Mon, 13 Jan 2025 00:00:00 +0000

As the 2024 has come to an end, let’s recap on some of the books I have read this year and what podcasts did I listen to.

Books

Earthside (Quantum Earth, #2) by Dennis E. Taylor
- The best setup and payoff I have seen in a book so far
Not Till We Are Lost (Bobiverse #5) by Dennis E. Taylor
- Great as always, although I would rank this as my least favorite Bobiverse book. Compared to others, there were too many story lines with unsatisfying endings
The Apollo Murders (Apollo Murders, #1) by Chris Hadfield
- Appreciated the realism given to it by its author
Redshirts by John Scalzi
- A great fan service with a lot of meta content, at the end I could not get my head around all the layers
The Collapsing Empire (The Interdependency, #1) by John Scalzi
- After reading Redshirts, I wanted to check out some more “serious” book and immediately was hooked on the next one
The Martian and Artemis by Andy Weir
- All of them are great and made my laugh out loud, my ranking is Project Hail Mary > The Martian > Artemis
All of the Expanse by James S.A. Corey
- After finishing the TV series, I wanted to know how the story ends. Discovered soon enough that the books are much more funny and light-hearted than their screen counterparts. Had to go back and read all of them.

Podcasts

Darknet Diaries
- Discovered recently, large backlog with great content and goosebumps-producing stories
The Layover
- Weekly recap and behind the scenes of what happened during Jet Lag: The Game with some entertaining in-between seasons stuff
Jam Mechanics
- Bug and TNC are given prompts for which they have 3 hours to create a song from scratch, love the energy between them two

Songs

Stopped self-hosting everything

Sun, 29 Dec 2024 00:00:00 +0000

Self-hosting everything is fun – at first. After a while and a few days of downtime, cloud starts to seem more shiny than before. This is why I decided to move from self-hosting everything to a more balanced approach.

The big three categories of services that I used to self-host are email, all websites and VPN that connected my home lab with VPSs. As you can probably imagine, there is a different kind of nerves when one of the things enters its downtime. You need to always know that everything is working properly, because if all your mail servers are down for longer period of time, you could miss something important (such as a university assignment, what a horror!). My personal breaking point was when I was hit with a two-week outage on one of my VPS due to the provider having storage issues right at the time I was supposed to give a lecture from a website hosted on it. This does not mean that I have stopped self-hosting everything, just that I now better judge what is worth my time and nerves versus what for what I am willing to give a few bucks to someone to deal with the struggle for me.

For better orientation, I have divided my services into three main categories: Email, VPN and websites. For each of them, I am first going to briefly describe what was my approach in the past and what I am using now:

Email

The first rule of email is that you always have at least two servers. I had a main Linux VPS with Postfix that taken care of receiving and sending email with IMAP access through Dovecot. Then, I had a secondary Linux VPS also with Postfix, but this secondary VPS just saved everything that it received locally and waited for a cron job from the primary server to take it for ingestion. In the unlikely situation that both my VPSes were down at the same time, I had a tertiary MX record pointing at JunkEmailFilter’s MX Backup service. This service promised to keep all received mails and forward them to the primary email once you sort your outage out. Great stuff! Saved me a couple of times during the years. Even though I have nailed the receiving part of email most of the time, there were still hiccups, e.g. when somebody sent me a large file as an attachment. What is worse is sending your email from your own IP address – nowadays, all major mail providers are using block lists, and they are sending new IP addresses to SPAM folder by default (to combat SPAM, which makes sense). For the several years I was running my self-hosted mail provider, sending mail to @gmail.com address was always a roll of a dice.

Where have I moved instead? To Proton mail. I have purchased their 2-year plan during the Black Friday sale and haven’t regretted it since. I really like Proton because of their privacy focus. Plus, it is easy to add custom domain(s) together with catchall addresses and to set up some basic Sieve-like filters (the only thing missing is filters by mail content, which in the context of Proton makes sense). The only inconvenience is that I need to self-host a Proton mail bridge in order to be able to send and read my mail from Thunderbird.

However, I operate more domains than my Proton plan would allow for, so I cannot migrate everything over there. For my other domains, I am using two services:

Cloudflare email routing if I only want to receive the mail and not send it
Purelymail for everything else.

Purelymail offers a pricing based on how many mails you have stored or sent, so when I keep forwarding everything to my Proton mail account and only occasionally send a mail, the overall price per year for multiple domains is far less than $10 per year for me. Unbeatable. Except for that, I am often using DuckDuckGo email protection for throwaway accounts or SimpleLogin for ones I want to use more often, but still keep my real address private.

VPN

As I have mentioned in a past article, there are multiple options how to access applications running on LAN from whole Internet. In the past, I have opted to use a setup with one central VPS that had a public IP address. This VPS had also running WireGuard, to which all my LAN-only servers were connected. Then, by using HAProxy on the VPS, I have decided which domain should be forwarded where. A second layer is that I want my LAN to be accessible from my parent’s LAN and vice versa. Because I am running MikroTik routers everywhere, I have again used the WireGuard on the central VPS to connect all the networks together. Then, when the routers could see each other, I have set up a second WireGuard layer just between them and created relevant forwarding rules for IP ranges. As you can probably already see, the main problem is the VPS being a single point of failure and often also a bottleneck because of it limited 100 Mbit/s bandwidth.

Instead, I have mainly migrated to three different solutions:

web pages see below
instead of connecting routers through central VPS with WireGuard, I use ZeroTier which has a package available for ARM-based MikroTik routers. After the routers can see each other thanks to the ZeroTier, I still use WireGuard to bridge the LAN networks.
to connect to LAN from WAN, I have two very cheap VPS dedicated only to do port forwarding to WireGuard on my local MikroTik router

I opted not to use the ZeroTier directly for connecting networks because I did not want ZeroTier to be able to see all traffic that goes between my LANs. Instead, the additional WireGuard layer provides post-quantum security, while also giving me full access control on every endpoint. However, if I opted to use the ZeroTier directly, I would not have to purchase the two additional VPSes (one primary and one secondary) to connect to my LAN from WAN, I would only need the ZeroTier application.

Web apps

Static pages, such as this blog, are all hosted through Cloudflare Pages. Because the pages can be build through CI/CD directly from GitHub repositories, all I have to provide is the command to build the pages, set the alias and then forget about it. This reduces load on my servers while increasing uptime (no single point of failure on my side!).

Dynamic pages, such as APIs and alike, are running on my VPSes or local servers, which have cloudflared installed. Then, I route relevant subdomains to relevant ports of docker containers on the target machines. This gives me the advantage of not having to configure reverse proxies and also automatically generates TLS certificates for me. Furthermore, I can secure my applications behind SSO through Google or Microsoft or similar if I want to prevent whole Internet reaching my application without proper authentication.

Another option that I use is to use the Cloudflare Access login as a source of truth which I use within my project as an OIDC identity provider, so you can log-in with your Google/Microsoft/w/e account into your locally hosted Nextcloud or Gitea.

Conclusion

In short, even though I still fully support anybody who wants to try to self-host anything and everything, I have decided to move away from it. It was a great learning exercise and gave me much understanding of the inner workings of many of the applications, but in the end it because almost a full-time job to keep everything running and caused problems when the bottlenecks hit. Now, with my mixed solution, I can sleep much better because I know that even when some parts of my network are offline, some services are still running and a good chance is that I won’t need to fix them first thing in the morning.

DNS Bonus

When you are on your LAN you may want to use the LAN-only IP addresses of your servers for faster access, or if you just want to simply block known malicious domains, you may want to overwrite local DNS records. In the past, I have used dnscrypt-proxy as a self-hosted DNS server. Now, I have migrated to setting custom DNS rules directly inside my MikroTik routers. As this task needs to be applied periodically for multiple routers, I have created a simple script that is able to take a hosts file and set the same records inside the MikroTik routers:

#!/usr/bin/env python3
import ipaddress
from pathlib import Path
from sys import argv
from re import sub
def is_valid_ipv4_address(ip_string: str) -> bool:
    try:
        ipaddress.IPv4Address(ip_string)
        return True
    except ipaddress.AddressValueError:
        return False
def conver_hosts_line(line: str) -> str:
    line = line.replace('"', '')
    line = sub('\s+', ' ', line)
    line = line + ' '
    try:
        domain, target, _ = line.split(' ', 3)
    except ValueError:
        return None
    matching_rule = f'name="{domain}"'
    if '*' in domain:
        domain = domain.replace('.', r'\\.').replace('*', '.*')
        matching_rule = f'regex="{domain}"'
    if is_valid_ipv4_address(target):
        return f'/ip/dns/static/add type=A {matching_rule} address="{target}" comment="autogen" ttl=30'
    else:
        return f'/ip/dns/static/add type=CNAME {matching_rule} cname="{target}" comment="autogen" ttl=30'
def main() -> None:
    file_in = Path(argv[1])
    assert file_in.exists()
    print('/ip dns static remove [find comment="autogen"]')
    with file_in.open('r') as f:
        for line in f.read().split('\n'):
            line = conver_hosts_line(line)
            if not line:
                continue
            print(line)
main()

Then you can simply take your hosts file and use the script in bash: ./hosts-to-routeros.py hosts.txt | ssh -T mikrotik-router. It will remove all previously autogenerated rules and replace them with the new content. An example of the hosts.txt file may be:

immich.example.com        10.0.0.2
my-server                 10.0.0.3
nextcloud.example.com     my-server
*.example.org             my-server
homeassistant.example.com external.example.com

As you can see, it supports both wildcards and aliases. Furthermore, you can then use DoH inside your MikroTik router for more private resolving or use some WireGuard-based VPN with in-build malware and adblock such as ProtonVPN as your DNS server.

Cloudflare Access to get to LAN

Sat, 05 Oct 2024 00:00:00 +0000

Self-hosting applications at one’s home provide benefits over using their cloud-native alternatives. These benefits may include privacy (your data is stored in your hardware vs in the cloud with the second party) and overall cost (effectively paying only for electricity and hardware up-front vs paying for a subscription tier every month/annually). On the other hand, as its most prominent disadvantage, self-hosted applications at one’s home are usually only available from inside the local area network. In contrast, cloud-native applications can be accessed from anywhere with an Internet connection. This text will focus on solutions tailored for SOHO (small-office and home-office) settings.

Note: this post was originally submitted as an university subject essay, so that it uses a little more formal language than usual.

Classical solutions

To see the advantages of choosing Cloudflare Access for this task, we first need to discuss what other solutions are already present and often used. Multiple options exist for accessing LAN-only applications from the Internet, each with its custom set of up- and downsides. Please see that almost none of the classical solutions listed below discusses the problematics of IPv4 and IPv6 running simultaneously, as most of the approaches allow for only one at a time. Furthermore, no approach deals with modern security practices such as Zero Trust[s] in any way by itself.

Public IP address

With some of the largest Czech ISPs (internet service providers), it is possible to obtain a public IP address[s][s] that directs all traffic directly to a router in possession of the SOHO administrator. The IP address may be provided as a benefit from an ISP for a fixed price or, much more commonly, for a monthly price. This solution offers the most significant freedom for the administrator, as they can tailor the network stack to their liking by simply setting up port forwardings on their ingress router. On the other hand, this solution may be disadvantageous in terms of security (the administrator is fully responsible for setting correct access controls) and flexibility (if the administrator does use some form of centralized configuration, configuring the rules properly may steadily become infeasible within more complex settings).

VPN service with exposed ports

Some VPN providers like ProtonVPN[s] and ExpressVPN[s] provide an option to forward a single (TCP/UDP) port from the VPN’s public IP address to a local machine or specific LAN address. The main advantage of this approach is the ease of setup and little technical knowledge barrier compared to other solutions when the administrator can install the VPN provider’s official GUI application and enable port forwarding. The drawbacks are evident; with only a single port, the option to host multiple applications is severely limited unless the administrator decides to use additional local solutions such as a reverse proxy for multiple HTTP-based applications or a protocol multiplexer for a specialized list of supported applications. One possibility to overcome this obstacle may be to host a VPN server on the exposed port, allowing client devices to access the LAN upon connection. Nevertheless, the port number and IP address provided by the VPN service may change, which may require additional reconfiguration on client devices.

Creating a custom tunnel with VPS

Another approach may be purchasing a cheap VPS (virtual private server), typically with a public IP address. Then, it is possible to set up a custom VPN server that the services on LAN will be connected to or directly use the remote port forwarding SSH option to forward the traffic from VPS to the self-hosted application. Because the VPS acts as a proxy, this approach may be suitable for both IPv4 and IPv6, even when the ISP is IPv4-only or IPv6-only. On the other hand, using a VPS as a proxy adds a new layer of complexity – the response time is increased by the time that it takes to forward the request from VPS to LAN and back, and the connection may fail, e.g. when the VPS reboots because of applied updates. It also places additional requirements onto the administrator, who now needs to properly configure the VPS so as not to allow unauthorized access, which can be an obstacle for administrators without experience with managing remote servers.

Using a dedicated P2P software

An option that relies solely on the clients talking directly to each other, usually with the help of nodes that act as middlemen exchanging information about connected clients to each other. Examples of such software may be LogMeIn Hamachi, n2n and gsocket. Most prominently, the Hamachi targets regular users, requiring little to no technical knowledge. This makes the entry barrier almost nonexistent. On the other hand, it requires special software to be installed on both endpoints and because of its P2P nature, clients may be blocked on more restricted networks like airports or hotel lobbies.

Cloudflare Access

Cloudflare Access is primarily a web-application product from a company with the same name. Cloudflare is perhaps best known for its word-wide CDN (content delivery network), which offers both hosting static pages and acting as a reverse proxy where the inbound connections are accepted to the Cloudflare network and then internally forwarded to a real server, which will then process the request. The main benefits of using Cloudflare as a reverse proxy are the hiding locations of the real servers, which is helpful in case of DDOS attacks, which Cloudflare can filter out.

Generally, Cloudflare Access masks the actual IP address of the server by overwriting responses to DNS queries with addresses from their IP range.

Cloudflare Access Application types

Cloudflare Access supports three main application types.

SaaS (Software as a Service)

Cloudflare Acess supports integration with SaaS[s] directly, but this solution is more focused on enterprise applications than SoHo, so I will consider it out-of-scope for this project. However, it has a significant advantage for some self-hosted applications in the form of SAML (Security Assert Markup Language[s]) provider.

Private network

With this application type, the administrator can specify a private IP that should be exposed through Cloudflare private network. Because access to this network always requires additional software installed on endpoint devices, I consider this out-of-scope for this project.

Self-hosted

In the self-hosted application type, the administrator is supposed to enter which subdomains (and optionally paths) should be redirected to the LAN. Setting up this application is the basis for further creation of processes, like access policy and the actual tunnels to the LAN.

Example self-hosted application configuration

Cloudflare Zero Trust

Zero trust[s] is a design philosophy in which a user is required to authenticate before accessing resources (e.g., web application), which is mainly in contrast to a more classical approach where a user is granted access to an application based on e. g. being in the correct network. With zero trust implemented, even if the attacker can get access to the network (for example, by compromising a remote code vulnerability within an IoT (Internet of Things) device like IP cam [s]), their impact is limited by still missing credentials that are required to access protected resources.

Whereas generally, when accessing a web application, the Zero Trust is implemented directly as a login system to said web application, the Cloudflare Zero Trust acts as a middleware that will prevent access to the target application until the user complies with the set access policy. Because we cannot expect the average hobbyist who is a self-hosting administrator to follow the latest security-related news, this approach may be beneficial when a vulnerability is discovered within the target web application because the attacker would still need to satisfy the access policy, which in turn gives more time to the administrator to patch the vulnerability.

Access policy

As referenced above, access policy[s] is a set of rules that any party must fulfil to be granted access to the target resource. At least one policy must be configured for each Cloudflare Access application, and each policy needs to be assigned to at least one group of users to which it applies. The policy may be set to approve access or block access. In its most basic form, the access policy allows access to any known user. However, the access policy may contain much more focused controls – e.g., based on the current user’s country, OIDC claim[s] and more. Unfortunately, it is impossible to specify controls based on time (e.g., only allowing access to the application during business hours) directly through policy specifications inside the web UI. If the administrator requires such functionality, it must be implemented directly within each application or by modifying an OIDC proxy, as hinted in the Appendix.

An example of an allow access policy with group and country selection

Identity management

We have already touched on users and their groups inside an Access policy chapter. Cloudflare itself does not provide any identity management (with a tiny exception of one-time mail password), while instead relying on external services using the single sign-on principle (SSO, [s]). The administrator can configure multiple identity sources from predefined providers or specify custom OIDC[s] settings. This approach has several advantages – for individuals/SoHo, most users already have their Google or Microsoft accounts. For enterprises, it is possible to reuse, e.g., their already set up Microsoft AD tenant, making the onboarding process effortless for regular users. Furthermore, Cloudflare provides step-by-step guides on how to set up each provider. Whenever user authenticates to the Cloudflare Zero Trust page with any configured provider for the first time, they are assigned a seat. The free tier has a total of 50 active seats[s]; after a seat is taken, it is possible to free it again.

For this project, I have tested setting up Microsoft Azure AD® and Google as identity providers. The initial setup took about 30 minutes, primarily because of my minimal experience with both environments. Both providers offer modern MFA solutions for securing target user accounts; their only significant difference is that when setting up the providers with personal accounts, with Microsoft Azure AD I was able to select which user accounts are allowed to authenticate to my AD tenant instance. With Google as an identity provider, any valid user can authenticate. To counter this, I have had to set up a particular group only with selected user seat accounts inside the Cloudflare Zero Trust dashboard. Still, it may be possible for a malicious actor to log in with as many Google accounts as possible. While they are still unable to access the resources because of manual group filtering, they can fill up all seats available with the free tier.

Cloudflare Tunnels

After setting up the self-hosted application records, the next step is to launch a local tunnel. The tunnel is a server process or docker container that accepts requests from the Cloudflare network and forwards them to specified local addresses[s]. To initialize a new tunnel on LAN, all the user has to do is to copy a pre-generated command that already contains all required configurations. Afterwards, creating a new forwarding from WAN to LAN is only a matter of selecting a subdomain (same as in configuring a self-hosted application), protocol (by default HTTP, but can be anything TCP-based), and local address with port to which will the tunnel be forwarding remote requests. Optionally, selecting an access policy that must be satisfied before the request can pass through is also possible. The only limitation is that the same subdomain cannot be used for multiple protocols (e.g. sub.example.org cannot be set up to forward both HTTP and SSH; another subdomain like sub-ssh.example.org must be used instead).

Because a single tunnel can forward multiple subdomains, adding new forwarding from WAN to LAN takes tens of seconds after the initial setup is finished, which is a significant simplification for the network administrator.

Preventing double-logging

After the user authenticates themselves against the Cloudflare Zero trust login page, they may still be required to log into the target web application, which leads to several weak points. Firstly, the user’s comfort is reduced as they must go through another login process right after finishing the previous one. Secondly, either the administrator of the web application needs to set up the same SSO options as for the Cloudflare Zero trust login page, which places additional complexity on the application management, or the users need an additional account just for this web application, which in turn may increase risk of password compromise, as users may not follow security best practices for all their accounts.

If the administrator could configure the web application to always accept the same login credentials as the Cloudflare Zero Trust page, it would increase the overall user comfort and application security while decreasing administration costs.

SAML

Security Assert Markup Language[s] can be leveraged to provide an SSO[s], but this standard is more prevalent for enterprise-oriented applications. For NextCloud, an official application can be installed, which adds the possibility of using SAML as a backend for user management. As Cloudflare provides SAML endpoints only for SaaS applications[s], we need to create a new virtual SaaS application within the dashboard and then use the provided details to configure SAML within NextCloud correctly.

Unfortunately, SAML for smaller open-source applications like Gitea is not supported, so it is only an option for some of this paper’s target self-hosted applications.

Every request sent to the target application behind Cloudflare Zero Trust protection includes a CF_Authorization cookie [[@cloudflareAuthorizationCookie]]. This cookie contains a signed JSON web token (JWT [s]) with the user’s email. If the developers of target applications were to implement authentication using this cookie, it would be possible to log in directly to the application. Still, this would require developers to implement a non-standard login mechanism. Another approach may be to verify the provided JWT on a reverse proxy and set a header with a user name, which is supported, for example, on Gitea[s].

The main drawback of using the cookie for authentication is that it requires the user to always log in through WAN through Cloudflare Zero Trust, even when the user may prefer to access the application through LAN for considerably faster response speeds (e.g. by using DNS masking).

Accessing SSH

So far, we have discussed only accessing HTTP-based applications, but Cloudflare tunnels support any TCP-based protocol, with SSH being explicitly stated. If the user only wants to access the LAN-only SSH server from WAN, creating a new record in the Cloudflare tunnel, which will forward all requests, is possible.

Though, if the user would also want to protect the SSH server with Zero Trust (e.g. requiring logging in with MFA-protected Microsoft Account), they first need to install a cloudflared client on their device[s]. Then, by modifying their local SSH config to use cloudflared as a ProxyCommand, it will first launch the cloudflared command whenever they try to assess the SSH server. The command opens a web browser with a Cloudflare Zero Trust page, where users can log in with their accounts. After a successful authentication, an SSH connection proceeds as usual. Furthermore, thanks to the modification of the user’s local SSH config file, this whole procedure needs to be done only once, making the authentication process after initial setup effortless. However, the apparent disadvantage of this approach is that the user needs to install the Cloudflare client on every machine from which they plan to access the SSH server.

To solve this disadvantage, Cloudflare Access offers an option to render the SSH terminal inside the web browser. This setting can be enabled within the Additional Settings section of the self-hosted application. This browser terminal is again protected with Cloudflare Zero Trust, and the rendered terminal supports classical keyboard shortcuts and function keys, making it a reasonably usable alternative. Unfortunately, this approach requires the user to manually input a password or copy a private key unless additional settings like short-lived certificates are used.

Conclusion

In conclusion, Cloudflare Access offers a helpful alternative to access LAN-only resources from WAN. The complexity of the solution and the time spent configuring it for the first time is well-balanced, with easy scalability for adding new resources and following security best practices like Zero Trust. The way in which the Cloudflare Zero Trust is implemented may also increase the overall security of the LAN network, e.g., even when an inexperienced administrator exposes a vulnerable application, a malicious actor is still unable to access it. Furthermore, relying almost solely on external identity management services increases user comfort as they can reuse their already existing accounts. Finally, the Cloudflare Zero Trust is free for up to 50 users/seats, which should be plenty for common SOHO requirements.

Creating safe WiFi abroad, Vol. 3

Sat, 01 Jul 2023 00:00:00 +0000

I have an old small laptop lying under my bed, let’s put some OpenWrt on it! (TL;DR: failure)

Picture of the final setup sent to me

My family went on a vacation to Croatia while I was finishing exams. The only sensible thing to do was to gear them up with a router that will protect them against all bad actors.

Setup & installation

The setup:

unknown old Packard Bell laptop with Intel Celeron CPU
USB WiFi adapter
bootable SD card with OpenWrt for x86 hardware

Pros:

the small size of the whole OS (< 200 MB)
very fast booting speed (< 20s)
easy setup of OpenVPN with kill-switch

Cons:

no interfaces detected from the beginning (missing drivers)
couldn’t get WireGuard routing to work for clients on LAN (maybe with more time)

The overall process of installation was very positive and the only issue I had with the missing drivers was – for Ethernet – solved by downloading the package manually (for my generic x86 I had to download the driver from here – beware that the URL ends with /targets/x86/generic/packages/, otherwise you are on a wrong platform and the installation will fail). As to what driver to install, I have booted up Fedora Live, which showed me the correct driver name by executing lspci -vvv.

The wireless was a little more complicated, as I had to install much more packages. At least now I could use the web UI to install them, because my Ethernet connection was already working.

Wireless trouble

My first thought was to transform this laptop into a slave AP. Unfortunately, the on-board Intel card does not support AP mode. Oh well, I want to connect an external USB WiFi adapter anyway.

My second idea was to create an AP on the external adapter and use the internal card to connect to the host network/WAN. Ultimately, this was the same approach that I used last year. This has worked, but produced unpredictable bandwidth through-output (1-30 Mbit/s) with a threatening syslog message:

rtlwifi: -----hwsec_cam_bitmap: 0x0 entry_idx=X

This, unfortunately, is a known issue with some USB adapters without any sensible solution.

Real-world usage (fail)

Even with all the issues above, I have decided to set up a kill-switch OpenVPN connection and ship it with my family to Croatia. And it failed, miserably. During my testing setup, I have never used more than two connected devices at the same time. But, when everyone connected their devices, the CPU did not make it.

Join to start a Minecraft server

Sat, 10 Jun 2023 00:00:00 +0000

Multiple Minecraft servers are running on my cheap VPS with limited RAM – how to make accessing them as user-friendly as possible?

All of my servers are running almost effortlessly thanks to the amazing Docker Minecraft server. However, in a situation when one of the servers wants to eat at least 6 GiB of memory while the maximal memory available is 8 GiB, all servers can’t be running at the same time. Some of them must be stopped.

AutoStop & AutoPause

The amazing docker image provides the option to pause (to free up the CPU) or to stop the server. Supposedly, when a pause happens, a knock server is fired up on the same port and listens for an incoming TCP connection. When the TCP connection is established, the Minecraft server is started again. Unfortunately, this did not work for me (the server was always running when I checked the RAM usage) and I did not feel like debugging. I wanted something better anyway.

Recreating the behaviour

I liked the idea of a knock server starting the server again. For that, I have written a simple bash script that checks if a docker container is running and if not, creates a NetCat server that waits for a connection. When a TCP connection is established, the Minecraft server is started. To shutdown the server again, I have set AutoStop to shut down the server after one hour without any players:

#!/bin/bash
# Example: the container is named mc_server1 and is using port 25565
# -> run with ./autostart server1 25565

set -u
SERVER="$1"
PORT="$2"

echo "Will check for $SERVER on port $PORT"

CONTAINER="mc_$SERVER"
while true; do
    if docker ps | fgrep --quiet "$CONTAINER"; then
        sleep 10
        continue
    fi

    netcat -lw 1 0.0.0.0 "$PORT" &&
    echo "Starting $SERVER" &&
    docker start "$CONTAINER"
done

Making it user-friendly

This is nice, but the server starts as soon as any player (or bots) checks if the server is online (does not even have to try to establish a player session). As I am using whitelists on the server, I would want the autostart script to respond only to the whitelisted users. For that to work, I asked my brother (as he has much more Minecraft-related experience than me) to create a script that would:

listen on the same port as the Minecraft server
show a message to the users that the server is currently turned off
allow the user to join the server, which will trigger the autostart process

You can download the mock Minecraft server here. The only required change to my autostart script was to replace the netcat with the custom Python script:

#!/bin/bash
# Example: the container is named mc_server1 and is using port 25565
# -> run with ./autostart server1 25565

set -u
SERVER="$1"
PORT="$2"

echo "Will check for $SERVER on port $PORT"

CONTAINER="mc_$SERVER"
while true; do
    if docker ps | fgrep --quiet "$CONTAINER"; then
        sleep 10
        continue
    fi

    python mock-server.py "/ot/minecraft-autostart/$SERVER.json" &&
    echo "Starting $SERVER" &&
    docker start "$CONTAINER"
done

What users see

The final workflow is pretty rewarding to see:

The server is offline - join to start the server

Notification that the server is starting now

Wait few moments for the real server to start

The real server is running

Cat-proofing the Raspberry Pi

Sun, 30 Apr 2023 00:00:00 +0000

At my girlfriend’s place, I have a RPi 4 server with an old external 1.5TB HDD attached. To protect the hardware from her furry pets. I had to place the server inside a big cardboard box filled with plastic air bubbles and hide the box itself under the nightstand. This setup, as time had shown, was not without flaws.

It’s hot & loud

First, imagine a computer in a closed tight space with active USB ports. Such instalment inevitably leads to overheating. Overheating can be solved by adding a passive cooler and an official RPi fan, nice!

Photo of Raspberry Pi with the official fan installed

Not a “fan” of high tempeature

Oops, the fan is really loud with quite a high pitch, which makes sleeping in the same room almost unbearable. To lower the time with the fan on, let’s write a simple script (…because the RPi is running Ubuntu and raspi-settings is not available here…) that will turn the fan only when the CPU temperature reaches a certain threshold:

#!/usr/bin/python3

from os import system
from gpiozero import LED
from time import sleep

FAN_PIN = 14
FILE_TEMP = "/sys/class/thermal/thermal_zone0/temp"
TEMP_MAX = 75
TEMP_COOLDOWN = 65

print(f"The fan will start when the tempeature is over {TEMP_MAX} degrees", flush=True)

fan = LED(FAN_PIN)
cool = False

while True:
    try:
        with open(FILE_TEMP, 'r') as f:
            temp = int(f.read()) // 1000
        if temp >= TEMP_MAX:
            if not cool:
                print(f'Fan ON, {temp}', flush=True)
            cool = True
        if temp <= TEMP_COOLDOWN:
            if cool:
                print(f'FAN OFF, {temp}', flush=True)
                cool = False
        if cool:
            fan.on()
        else:
            fan.off()
    finally:
        sleep(3)

This makes the noise problem almost disappear, but still, when the load would get high enough (e. g. when multiple phones would start backing up their photos at the same time), the temperature could grow over the threshold and start the fan in the middle of the night. I must note, that the setup was active in this form for a few years and I am truly sorry.

Governor!

I figured out that I need to take matters more closely into my hands. From the forums, I was able to deduct that the RPi itself starts to slow down CPU speed, but in, for me unpractically high, temperature. Fortunately, I could switch the CPU governor to manual and force it to run at the lowest possible clock speed:

#!/usr/bin/python3

from os import system
from gpiozero import LED
from time import sleep

FAN_PIN = 14
FILE_TEMP = "/sys/class/thermal/thermal_zone0/temp"
TEMP_MAX = 75
TEMP_CPU_SLOWDOWN = 67
TEMP_COOLDOWN = 65

print(f"The fan will start when the tempeature is over {TEMP_MAX} degrees, the CPU will slow down at {TEMP_CPU_SLOWDOWN} degrees", flush=True)

fan = LED(FAN_PIN)
system('cpufreq-set -g ondemand')
cool = False
slow = False

while True:
    try:
        with open(FILE_TEMP, 'r') as f:
            temp = int(f.read()) // 1000
        if temp >= TEMP_MAX:
            if not cool:
                print(f'Fan ON, {temp}', flush=True)
            cool = True
        if temp >= TEMP_CPU_SLOWDOWN:
            if not slow:
                print(f'CPU SLOW, {temp}', flush=True)
                slow = True
            system('cpufreq-set -g userspace')
            system('cpufreq-set -rf 600M')
        if temp <= TEMP_COOLDOWN:
            if cool:
                print(f'FAN OFF, {temp}', flush=True)
                cool = False
            if slow:
                print(f'CPU normal, {temp}', flush=True)
                system('cpufreq-set -g ondemand')
                slow = False

        if cool:
            fan.on()
        else:
            fan.off()
    finally:
        sleep(3)

Furthermore, I have added maxcpus=2 to the /boot/firmware/cmdline.txt file, so only 2 out of 4 CPU cores were enabled. These two modifications have reduced the frequency of fan starts but still would run if the CPU load stayed high for long enough. This was because the Pi still was placed inside a small box. To resolve this, I would have to place the Pi into a more open space.

Then it clicked

The RPi was now perfectly quiet, but still, one source of the noise remained – during every write, the HDD would give out a (semi-)loud clicking sound, which was utterly annoying (but still lasted several years, I really am sorry). First, I tried some software-based solutions:

set BTRF’s commit interval to 5 minutes, with the idea behind it being that it would write all data only once in a while and would not click that much. As you can imagine, this resulted in a data loss after power loss on more than one occasion
use FUSE-based fs that would cache the data on the internal SD card and after the specified interval would move the data to the HDD. This option resulted in a considerable performance loss and wore off the internal SD card rather quickly

Besides, neither of the options solved the initial clicking problem, only delayed it. The only possible solution was to replace the HDD with an SSD. After a bit of research and setting up an RSS notification for /r/buildapcsales, I have found out that Patriot makes nice and cheap 2TB discs. That, combined with a case, made for a simple and working external SSD.

A new adversary

My SO’s new addition to the pet family got bigger and better (at moving the cardboard box containing the RPi), which poses a new threat to the data integrity, as cats are not known for their carefulness with anything they consider a toy.

Photo of my new adversary, Lucifer

Moreover, even when he did not try to move some boxes while nobody was looking, Lucifer still left quite a large load of furr everywhere he went, which did not seem exactly healthy for any electronic devices. Any form of placing the Pi in a freely accessible spot would now equal hazarding its life. I had to find a space open enough so that the air would circulate properly and that the cat won’t be able to reach it.

New setup

Luckily, my SO has one of the tables with space for cables that was wide enough for the Pi to fit in. Furthermore, because the network in her house is wireless only with some devices running untrustworthy software, I saw this as a great opportunity to create a custom wired network where the Pi would act as a router, safe from all untrusted devices, so I hooked the Pi with a cheap 4-port 100 Mbit/s switch.

Photo of Pi and other electronics hidden inside a table

Even though acceptable in theory, the air circulation was not good enough to keep the Pi from overheating and, in the end, it made the situation more harmful. Even worse, it turned out that the Pi’s plastic case itself was unable to decrease its temperature quickly enough.

After a while, I decided to place the Pi between the supports for the shelf above the table. And, because bright red and white would appear too distracting, I have decided to purchase a new case made of metal. This could act as a supplementary layer of passive cooling. Even though it hadn’t a dedicated spot to hold the fan, I was able to attach the fan with a few wires through the gaps in the upper part of the case. Then, I positioned the SSD on the opposite end of the shelf and arranged all cables inside the table together with the switch (which does not suffer from overheating).

Photo of Pi and SSD under the shelf

Final words

After everything was finished and I sacrificed the USB 3 speed in favour of WiFi, I now have a server setup which provides a safe ethernet network to all devices. Based on logs (and my SO’s experience) there was not a single fan run in a few months (except after a power loss during a thunderstorm). Furthermore, there were no cat-related incidents, because none of the pets is allowed on the table and even if they disregard this rule, they have never paid any attention to the black box attached to a wall.

Buying tickets from Ticketmaster

Sun, 16 Oct 2022 00:00:00 +0000

A few days ago, I wanted to buy a ticket to a concert. A relatively simple task, as you need only to act quickly enough and find your place as soon as the sale opens. Little did I know that since I bought tickets last time, the Ticketmaster introduced some “upgrades”.

Ticketmaster(.cz ≠ .at ≟ .eu)

I believe, that I do not need to introduce you to the Ticketmaster, as it has quite a reputation. The sale was supposed to begin at 10:00, so at about 9:40 I began to look for the event. The link at the artist’s web page took me to the ticketmaster.at (which makes sense as the event takes place in Austria), where I tried to log in with my Ticketmaster account. When filling the username and password through my browser extension, the login has failed, so I opted for copying the credentials from my backup password manager. This resulted in another fail. To check the validity of saved information, I have tried to log in into ticketmaster.cz, which was successful. It would seem that the Ticketmaster does not share the information between its instances in different countries, even if both countries are part of the EU.

Now, at 9:53, there was no time to research what exactly is happening. Instead, I have created a new account at ticketmaster.at using autofill of the same credentials as for ticketmaster.cz. After this, I was met with a reassuring message on a white background.

“You will be automatically placed in the queue when the sale begins”

The interesting this about this was, that now I was redirected to ticketmaster.eu, so the Ticketmaster does share account data between its national instances in the end?

Smart Queue

“The Latest Way to Shop for Ticket to Popular Events”, said Ticketmaster. According to them, it is a form of ticket bots prevention. Very well, I am intrigued how this will go. The 10:00 strikes about … now!

Queue with more than 25 thousands people before me about 20 minutes after the start

As soon as the clock hit the 10:00 mark, the page refreshed and showed me my place in the smart queue - I was nearly 30 thousandth person to be waiting for the tickets. This will take a while. Why not use the time to learn something more about the new queuing system? Let’s see what Ticketmaster recommends me “For a smoother shopping experience…”:

“Sign in to your Ticketmaster account at least 10 minutes in advance of joining the Waiting Room. This will speed up your purchase later.”
“Confirm you have a valid form of payment in your account with current email and billing information. This will make checkout a breeze.”
“If you need to step away, turn up the volume on your device so when it’s your turn, you hear the Queue notification bell.”

Because I created my Austria-based Ticketmaster account only about 7 minutes before the sale has started, this may explain why I received such a high queue position. Well, nothing I can do about it now – but I can prepare for the second point by filling in all my info (which I have already filled in my Czech-based account).

Smart Queue with notifications

It had seemed that the queue will take a few hours, and I certainly did not intend to stare at the screen all that time. But leaving would mean that I may miss a crucial time when to return to the keyboard. If only there was some way to notify me when only a certain number of people is left before me…

The easiest way would surely be to inject a script through a developer tools, which would parse the number of people left and call a HTTP endpoint. But I did not intend to risk being detected as the bot by merely opening the browser’s developer tools, let alone by injecting a new script into the page.

I had to opt for something less invasive – something that would read the text from the screen for me. As it turned out, there exists a package for an OCR software called tesseract, which takes a single image as an input and a single text file as output. Furthermore, because the number would (hopefully) not grow and consequently nor would the size of the text, I could be able to manually record the dimensions and coordinates of the text for cropping unnecessary information out of the picture.

After cropping, what I was left with was a black text on a white background, which almost seemed like it was made with people using OCR on screenshotted web browser in mind! After a bit of fiddling with Bash, I came up with a script that will:

send me periodical updates twice a minute
send me a priority message once less than 500 people is in front of me

#!/bin/bash
set -Eexuo pipefail

trap "while true; do bash -c 'notif --important FAIL' && break; sleep 2; done" ERR

wd="$(mktemp -d)"
cd "$wd"
touch last.txt

(
    while true; do
        gnome-screenshot -f screen.png
        convert screen.png -crop 420x38+2090+568 screen.crop.png
        tesseract -l eng screen.crop.png screen.txt
        count="$(egrep 'ahead of you: ([0-9]+)' screen.txt.txt | sed -Ee 's/^.*:\s([0-9]+).*?$/\1/')"
        lastCount="$(cat last.txt)"

        if [ "$count" != "$lastCount" ]; then
            if [ "$count" -lt 500 ]; then
                important="--important"
            else
                important=""
            fi
            notif $important "Queue: $count"
            echo "$count" > last.txt
        fi
        sleep 35 || break
    done
) && true

rm -r "$wd"

And how did it turned out? I think pretty great, as evidenced by my message history.

Messages sent by the notification script

Here comes the bell

After about 2 and half hours of waiting, the time to act was now. It was my turn to buy the tickets – all I had to do was to select how many of what type of tickets I wanted, press the “Search for tickets” button and … “There was an error while processing your request”. Wait, what? I have waited more than 150 minutes just to fail? What about a single ticket of any other category? Also fail. Why? What was happening?

At this point I had nothing to loose, so I opened the browser developer tools and looked at the network section. Whenever I pressed the button, the response for the subsequent HTTP request would be simply blocked with a status code of 403. The system things that I am a bot. Why does it think that? I really don’t know, as nothing I have done should differ from a normal user behavior. Maybe because I have registered a new account, as they suggested in their 2nd point, only 5 minutes before the sale has started? By the way – the bell mentioned in the 3rd point never rang.

Desperately, I have refreshed the page and was greeted with a new place in the queue – more than 40 thousands.

Do I have the tickets?

Yes! With a great foresight, my girlfriend has joined the queue at about 10:05 from her iPhone without any prior registration and has successfully completed the whole process. Needless to say, if there was any alternative to the Ticketmaster, I would gladly use it. But with the current state, if I want to get to the event, I have no other option.

Saving photos on an unencrypted smartphone SD card

Mon, 12 Sep 2022 00:00:00 +0000

When I have bought my Sony Xperia XA2, my main goal was to buy a phone that can run a (mostly) open-source OS – LineageOS. My last smartphone, the Xiaomi Redmi 4X, has supported a feature in which an SD card could be encrypted and extend the internal phone storage. It has newer occurred to me that this newer phone may not support this. Now I am stuck with my 32 GB of internal storage, where 29 GB is already occupied by the system and apps. Where can I keep my photos?

Internal storage

By default, all photos taken with the preinstalled LineageOS camera are saved to a fixed location inside the internal storage, and there is no option to change that behavior. Other applications, like OpenCamera, do not work as well with the device’s camera. As stated before, the internal storage is already quite occupied and would not stand a chance against a longer 4K video. I have to store it somewhere else.

External storage

My phone supports only an unencrypted ex-FAT formatted SD card, which limits my options by a large margin. On my previous phone, I had an option to format the SD card as a part of the encrypted internal storage. Unfortunately, as I have learned perhaps too late, my phone does not support this method. There are many applications on the Google Play Store, that claim to encrypt and protect your photos, but I see four main deal-breakers with this approach:

they are not open-source
they use an unspecified encryption algorithm
they mostly do not allow you to select where to store the encrypted photos
storing photos cannot be automatized

DroidFS

On the other hand, on the F-Droid, I have found a neat application called DroidFS. This application internally creates a gocryptfs volume(s) with a custom location and passphrases (with an optional fingerprint unlock). This application is open source, uses a known encryption algorithm, and lets you select where to store the photos. The only problem is with the automation of the encryption process – if you would want to add photos through an external app, you would have to first mount the gocryptfs volume, which depends on /dev/fuse, which is not available without rooting the phone, which is something I do not want to do, as this could break my SafetyNet.

The only implementation without the requirement for fuse device that I have found is gocryptfs-inspect, but this works only for a decryption of files, not encryption.

I am certain that it would be possible to create a custom project in Python that would also work for encrypting files, but I would rather find an easier solution.

RCX - Rclone for Android

Rclone is an application that I have already mentioned on this blog at least once. Its center point are so called “Storage systems”, which can be a real type of storage – e.g. local drive, SFTP, Google Drive – or a virtual one – e.g. a crypt storage that takes another storage and performs a transparent encryption upon it.

RCX - Rclone for Android is an open-source Android application that can manage and connect to the rclone “Storage systems”. Furthermore, it also integrates well with the Android’s SAF, so that I can access the content from the Total Commander application.

Even better, when saved into a shared directory on a SD card, I can gain a full control with Termux bash shell! The shared directory on the SD card, to which all applications can both read and write, is located at /storage/<sd-id>/Android/media/. With the RCX we can create a new directory – /storage/<sd-id>/Android/media/photos.crypt – and a corresponding crypt remote. Then we can export the configuration from the RCX and import it into Termux.

Automating the process

Now I have a rclone remote called Photos both in RCX and Termux. This remote is a crypt wrapper over a physical /storage/<sd-id>/Android/media/photos.crypt directory. The Photos remote has disabled filename obfuscation.

The thing that I do want to do now is for the photos and videos from my internal storage to be encrypted and copied to the SD card, where they will be sorted by their year and month for easier browsing. This task can be achieved with a following script:

#!/bin/bash

# https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/
set -euo pipefail

# Set the initial directory to the scripts location
cd "$(dirname "$(realpath "$0")")"

# All picures and videos start with IMG_ or VID_, followed by year, month and day
template='^((IMG_)|(VID_))([0-9]{4})([0-9]{2})([0-9]{2})_.*?$'

# Where to look for original files
directories=( "$HOME/storage/pictures/" "$HOME/storage/movies/" )

# Encrypt all files
for directory in "${directories[@]}"; do
    echo "Encrypting $directory"
    cd "$directory"
    for f in *; do
        if [ ! -f "$f" ]; then
            continue
        fi
        echo "$f" | grep -qE "$template" || continue

        echo "... $f"
        rclone moveto --progress "$f" "Photos:/$f"
    done
done

# Move the encrypted files into coorrect sub-directories by year and month
echo "Moving"
cd /storage/sdcard/Android/media/photos.crypt

for f in *; do
    if [ ! -f "$f" ]; then
        continue
    fi
    echo "$f" | grep -qE "$template" || continue

    echo "... $f"
    year="$(echo "$f" | sed -Ee "s/$template/\\4/")"
    month="$(echo "$f" | sed -Ee "s/$template/\\5/")"
    day="$(echo "$f" | sed -Ee "s/$template/\\6/")"

    directory="$year/$month"
    mkdir -p "$directory"
    mv "$f" "$directory/"
    echo "$f -> $directory"
done

Optionally, if I would want to synchronize the content of Photos remote with my cloud storage, the process would be fairly easy – I just have to add a new rclone remote and then sync them by adding the following lines at the end of the script:

# Optionally synchronize with cloud
set +u
if [ -n "$1" ] && [ "$1" == "--sync" ]; then
    echo "Syncing"
    rclone sync --progress Photos:/ cloud:/Photos/
fi

Final words

I have tried several options on how to securely store photos on an unencrypted SD card. In the end, I have settled for a script that takes care of the encryption for me and also can synchronize my files with my cloud storage. Now I can set up a cron job inside the Termux or use something like Tasker or Automate to have it run in periodical intervals. My internal storage space is saved!

Creating safe WiFi abroad, Vol. 2

Sun, 28 Aug 2022 00:00:00 +0000

Last time, I have mentioned that my first setup could be improved with an USB WiFi adapter, that I have forgotten home. So I have gone back to Croatia once again to test my hypothesis in action. How did in turn out?

The state of apartment complex WiFi

This time, the apartment complex had two WiFi APs - let’s call them complex and complex_5G. As you may have already guessed, the second one was the same as the first, only their frequencies had differed. Furthermore, both my RPi Zero and the USB WiFi adapter are capable of receiving only 2.4GHz WiFi APs, so we will ignore the 5GHz one for now. The complex WiFi had a good signal strength and its DHCP server was working, so we already were in a better situation in comparison to the last time. When measuring the connection speed on the AP, I have been able to get speeds up to 4Mbit/s for both upload and download. Though, during rush hours, when most of the guests were present in the building, the top speed I have gained was about 0.5Mbit/s for download and 3Mbit/s for upload. Not good, not terrible.

Setup #1

Directly improving upon the last time, I have prepared a following setup:

the core of the setup was once again my Raspberry Pi Zero W
the RPi was connected to the complex AP through an external (and more powerful) USB WiFi adapter
through an USB HUB was also connected an USB Ethernet adapter
on the other side of the Ethernet Adapter was a TP-Link WiFi router
the WiFi router has acted as my AP and DHCP server for my network
the RPi has accepted all traffic from my AP and routed it through a secure VPN tunnel

The first setup, nicely packed under the night table

Even though this setup was a direct upgrade upon the last one, when it was used in a real environment, it still shared some flaws with the original one. Most notably, the degradation of the TP-Link WiFI router speed has occurred – when measured with iperf3 from my laptop to the RPi zero, the overall throughput sometimes was only about 0.3 Mbit/s, which is clearly not even close to the advertised speed of 300 Mbit/s. It was clear that the WiFi router is the weak link, even when used only as an AP. It had to be removed.

Setup #2

Hardware

A great thing about RPi Zero W is that it has its own WiFi antenna, which can connect to or host an access point (or both at the same time, but that option is unreliable for real-world usage). Its antenna is fairly small and its signal weak, but since we have rented only one room, it should provide us with enough coverage for all of it. So, in this setup:

the core of the setup is still Raspberry Pi Zero W
the RPi is connected to the complex AP through an external (and more powerful) USB WiFi adapter
my AP is transmitted from RPi’s internal antenna
the RPi accepts all traffic from my AP and is routing it through a secure VPN tunnel

After the TP-Link router is removed, I can also remove the USB hub and Ethernet adapter, making the night-stand setup even nicer-looking:

The second setup, nicely packed under the night table without bloat

The second setup from the side with easily distinguishable components

Software

Of course, hosting the whole AP by myself has some implications – if I do want something more than static IP addressing, I have to run a DHCP server. (And a DNS server, but I can just reuse the already working dnscrypt-proxy from the original setup). I already have experience with isc-dhcp-server, which surely would be a reliable option, but what if there was a better way?

The answer is dnsmasq – a combination of both DHCP and DNS server, made for low-end machines (which RPi Zero certainly is). By default, it forwards all DNS requests to the system resolver, but, with a slight change of configuration, I can make it forward all requests to my DoH DNS proxy.

For creating the hotspot itself, I like to use hostapd, which provides a super easy way to create one. Just point it to an interface, add SSID, password, channel & version, and you are good to go.

Or, let’s go even easier! One could use something like RaspAP, which handles all these actions by itself. You can just use their single-command install script curl -sL https://install.raspap.com | bash and afterwards connect to a web control interface. From there, you can adjust your hotspot, DNS and DHCP settings, restart services or access additional features like VPN, ad-blocking and more. Under the hood, RaspAP uses previously mentioned hostapd and dnsmasq. Just beware, the version of RaspAP that I have used has set the default forward policy to ACCEPT, meaning that everything could be forwarded everywhere, even packets coming from the interface connected directly to the complex WiFi, which is not ideal. To mitigate this security risk, I have used iptables to set the default forward policy to DROP and allowed only forwarding from my AP to the VPN interface (effectively creating a VPN kill-switch for all clients connected to the AP).

Efficiency

The size of the rented room together with the bathroom and the balcony was about 6x8 meters, so that even the small and weak RPi internal antenna was able to cover the whole area with a good-enough signal. When testing the throughput with iperf3 command from the most distant place, I have been able to measure stable speeds of about 6Mbit/s, which is not great, but given that the complex’s AP provided me with the maximum speed of 4Mbit/s, it is good enough.

Getting more speed

I have figured out that the complex’s AP distributes its available bandwidth evenly between all TCP streams. Fortunately, a few days before, I have written a Node.js application that allows me to split my UDP VPN connection into multiple TCP streams. Even though I have written this application for another purpose (and will possibly write another post about it), it seemed just perfect for this use case, when the single VPN connection gave me less bandwidth than browsing without any VPN tunnel.

When testing on my multicore smartphone, I was able to gain up to 4 times more bandwidth when using 8 TCP streams, which seemed promising. Unfortunately, when I deployed this Node.js application on the single-core RPi Zero, it took about 60% of the CPU, leaving almost nothing for other services. In the end, the throughput was worse than using a single UDP connection directly to the VPN provider.

Final words

Overall, the second setup was a success – the AP was working reliably for a number of days until we left, there was almost no decrease of my AP bandwidth when compared to the complex’s AP and the traffic of all connected clients was protected from nosy neighbours lurking through the complex’s AP (from UFW logs it was evident that someone has tried to port-scan the RPi). Even though this was most probably my last vacation abroad this summer, I may reincarnate this series in the future. Maybe I will purchase a more powerful device? Or the rented room will be too large for the RPi’s internal antenna and I will have to use the TP-Link WiFI router as a signal repeater? Whatever the future brings, only one thing is certain - I will keep you all updated with my latest setup. So let me end this the same way as the original post – with the view at the sea.

The view from the room’s balcony

Replacing more than two occurences with Python RegEx

Sun, 14 Aug 2022 00:00:00 +0000

Recently, I was faced with a fairly easy task – to use Python’s re.sub to replace all occurrences of one text with another content. The only thing was that I was able to replace only the first two of them. But why?

Of course, the situation was a little more complex – the project that I was modifying takes a Markdown text as an input and, with the help of Pandoc, returns its content as HTML code. After a few hours of debugging the whole transformation process, I have tracked the issue to a single function. Can you spot the mistake?

def format_custom_tags(source: str) -> str:
    """
    Replaces all custom-defined tags as divs
    e.g. <ksi-tip> is replaced with <div class="ksi-custom ksi-tip">
    :param source: HTML to adjust
    :return: adjusted HTML
    """
    tags = ('ksi-tip',)
    for tag in tags:
        tag_escaped = re.escape(tag)
        source = re.sub(fr'<{tag_escaped}(.*?)>', fr'<div class="ksi-custom {tag}"\1>', source, re.IGNORECASE)
        source = re.sub(fr'</{tag_escaped}>', r"</div>", source, re.IGNORECASE)
    return source

Side note: Why is this function required? During the conversion, I want to replace the occurrences of a custom tag <ksi-tip> replace to <div class="ksi-custom ksi-tip">. If I did not perform this action, the Pandoc would sometimes take my custom tag as a plain text, which broke the formatting in certain cases. When the tag is specified as a class of div, everything works as expected.

Solution

Funnily enough, after search for Why does Python re.sub replace only two occurences I have found out that I am not the only one who was loosing mind about this issue - there already was an exactly same question. In Python, re.IGNORECASE equals to 2 and the fourth parameter of re.sub is count, not flags. That’s it. The whole issue was caused by a wrong order of parameters passed to the function, so, in the end, the fix was quite straightforward.

Storing luggage in Prague

Mon, 25 Jul 2022 00:00:00 +0000

Today, I am going to a concert in Prague. Naturally, while travelling by train, I am taking my laptop so that I can get some work done while traveling. The only problem is that the area, where the concert is going to be, does not allow notebooks, so I have to store it somewhere. The first part of this post is written while traveling to the destination, the second part will be amended after I successfully return home from the concert and regain my laptop.

Before Prague

Before travelling, I have made sure to look for options where to store my laptop, so that I make sure that I will see it again. In this section, I describe the information I found on the web, but beware that this information (as evidenced by the post-Prague section) was not very accurate.

What do I need?

The train is going to arrive to Prague at about 12:00, and I am planing on going back at about 4:00. I do not plan to use the laptop while I am not traveling by train, so I want to store it from 12:00 to about 3:00. The best option would be some kind of self-serving storage box, possibly with a PIN set, so that I can access it multiple times throughout the day if I need to store something more. Also, the closer to the train station, the better.

Where do I store it?

When searching for storage options on the Internet, I have come across a few different solutions. Let’s examine them now one by one:

Travel Box

CubeSave/Travel Box are self-serving boxes located directly at the main train station. All these boxes are supposed to be protected with a PIN, with options to book the boxes for 6 or 24 hours, while the 24 hours option is supposed to cost 125 CZK for a small box. This seems like the perfect solution, but it may be unusable due to all boxes being possibly occupied at the moment of arrival.

This storage service also provides a mobile application. With this application, the boxes can presumably be reserved after registration.

Side note: when verifying a registration email for the application, the verification link points to app.cubesave.cz. The login information provided in the app also work on this webpage, but upon login the only thing that is shown is a “permission denied” error, while the app seems to work just fine. Maybe this webpage is only for CubeSave admins and should not be presented to the regular users at all?

Screenshot after logging into app.cubesave.cz

One thing, that does not encourage my trust in this service, is that when accessing a secure version of Travel Box, the user is greeted with an invalid certificate made for the domain *.vshosting.cz.

Luggage Storage Prague

According to their website, Luggage Storage Prague has an open branch at Masaryk railway station, which is about 10 minutes of walk from the main train station. This branch is supposed to be open from 03:00 to 23:45 and contain self-serving boxes. According to the current prices, storing a single luggage should cost 175 CZK when stored for more than 3 hours and less than 2 days. Even though this location is a little further away, it still can be my second choice in case that Travel Box will fail me.

Walking route from Masaryk to the main railway station, OpenStreetMap

Honorable mentions

While looking for the perfect solution, I have come across a few that did not suite my needs, but my serve someone else’s, so I am including them nonetheless.

EasyLocker

Self-serving boxes near the historic center of Prague, but open only from 09:00 to 21:00.

České Dráhy’s storage

Main train station Prague offers its own storage option, operated manually. Open from 06:00 to 23:00.

Mysterious self-serving boxes

As I remember, inside the main train station, apart from the TravelBox’s, there is another set of self-serving boxes. I have used them a few years ago, and I certainly do not remember their opening hours, they are there nevertheless. They are supposed to be located in the north-most section of the station, but I fail to find any mention of them on the Internet.

Backup option

In case everything would fail, I have though about – a little complicated – backup option. In the case that I would fail to find a solution that would meet my needs, I would go to a paper store and purchase a box. Then, I would put my laptop inside the box and send it through Packeta to a pick-up point near my home. Sending a packet through Packeta can be done from almost any smaller shop. Typical delivery time of the packet is about 2 days and with a price of 79 CZK it is the cheapest option. Though, the package could possibly get damaged during transit, which is why this is only a backup option.

Final pre-Prague words

With all this preparation, I believe that I will be able to regain my laptop after the concert. I believe that the research I made will be enough for me to see my laptop once gain, yet I am still a little concerned, that something will turn wrong. Wish me luck.

Post-Prague update

Cube Save

Right after arriving at the main railway station in Prague, I wanted to rent one of the Cube Save boxes. On the station, there are two sets of boxes, both right next to the main entrance from the Opletalova street. However, there were two surprises waiting for me – the first set of boxes that I have encountered was out of order.

The out-of-order notice on the first set of boxes

This may be a problem, because now there is twice a demand for the second set of boxes, meaning that finding an empty box has become twice as hard. I have checked their mobile app, that indicated, that every box is now occupied. Nevertheless, I wanted to take a look at a working interface on the second set of boxes. Here I have found the second surprise:

Working Cube Save box interface

The box itself is technically opened 24/7, but the hall, where the boxes are located, is closed between midnight and 04:00. This information makes Cube Save completely unusable, because my train leaves at 04:12 and if more people will want to check out right after hall’s opening time, I will have a hard time getting on the train in time. With that, let’s move on to the second option.

Side note 1: I have also tried to take a look at the mysterious self-serving boxes, but, due to a large crowd and the hall being closed up until 04:00, I have decided that they are not a worthy option.

Side note 2: I have checked the app again at about 15:15, and it showed at lease one medium and one large boxes available.

Luggage Storage Prague

My second option was the “most trusted Luggage Storage in Prague” (according to their About us section), so I knew I was in good hands. As previously stated, their boxes are not on the main railway station, but rather on the Masaryk railway station, though both stations are withing easily walk-able distance from each another. It is supposed to be open from 03:00 to 23:45, which is roughly exactly what I need. Surely, there won’t be any more surprises…

The opening hours as shown on site

The first surprise right when I have entered the location was that the opening hours differ from the ones listed on their website. They are opening a half an hour later than advertised. Though a little later than I would like, I have decided to continue, because I still could catch the train easily.

When operating with the interface, I was asked to enter my email address. After returning home, I have found out that the box has sent me a message with my pickup code (which has fallen into my spam folder). The same pickup code was also printed on a receipt that I have received after a payment, so the email is more like a backup option for when you misplace your receipt.

Still, there is one surprise left – the price. Their web says 170 CZK when storing for more than 3 hours. So what price options did the interface show?

Prices for a small box

Because I needed more than 5 hours of storage time, I would have to opt for a 24 hours storage, which would cost me a total of 249 CZK, roughly 50% more than I have expected and twice as much as the Cube Save prices! Even after the trip, I am still unable to find any indication of this price on their website. After considering the possible issues of executing the backup plan of sending my laptop back home through Packeta, I have reluctantly accepted.

The pickup

At 03:33 I have entered again the Masaryk railway station and entered my pickup code. The box has asked me if I am only visiting the box or already checking out, which is neat and this feature should be advertised more beforehand. Because I had no intention of returning, I have chosen to check out. The box has opened, and I have seen my laptop once again.

My laptop, umbrella and a new belt at 03:35

From the Masaryk railway station I have taken a bus that departed at 03:49 from the station and arrived at 03:51 at the main railway station, to which I have entered at about 03:55. The main hall has already been opened, and no crowd was to be seen near Cube Save boxes, so I probably could safely leave my laptop there.

Final words

Even though I have successfully regained my laptop, after considering all the pitfalls I have endured when trying to find a place to store it and the overall price versus me using the laptop only for about 20 minutes during the whole trip in total, next time I will probably just take a book instead. On the other hand, it was a fun little adventure and I could visit the Masaryk railway station, which looks really nice, so (at least for me) it was worth it.

Creating safe WiFi abroad, Vol. 1

Fri, 15 Jul 2022 00:00:00 +0000

After successfully completing my bachelor’s finals, I have set for a vacation near a sea in Croatia. The apartment that I was to stay in has advertised to have WiFi connection, even though some reviews have stated that it is quite unstable. Nonetheless, this seemed like a perfect opportunity to test my secure WiFi AP setup.

The state of apartment complex WiFi

The whole apartment complex has two active WiFi AP - <place>123 and <place>123-EXT. My room is in a second floor, where the receiving signal of <place>123-EXT is much stronger than the one of <place>123. Unfortunately, as it seems, the <place>123-EXT is only a D-Link repeater for the base network, and no a good one – it has frequently lost connection to the base AP for an extended period of time, which makes it practically unsuitable. I was unable to pinpoint the precise location of the base AP and I suspect that it may be hidden inside the complex owner’s room. The only thing that I can say about it is that it is located on the ground floor and is manufactured by Huawei. As of writing of this post, I was unable to obtain any dynamic IP from <place>123-EXT DHCP server for two days and the connection to the base AP is not possible from any of my standard devices, so I am unable to measure the precise network speed. The best measurement I have is the 8Mbit down/4M bit up from the day one of the vacation.

Secure Wifi AP setup

The setup itself consists of the parts - a TP-Link WiFi router and a Raspberry Pi Zero. Fortunately, the TP-Link WiFi router (when placed on the right spot) is strong enough to connect to the <place>123 AP directly, so I can skip the misbehaving repeater. The TP-Link router is also set to work in a WIPS mode – it connects to one AP and simultaneously emits another one. It also acts as an authoritative DHCP server. This DHCP server informs clients to use the IP address of the Raspberry Pi Zero both as a gateway and a DNS server.

The Raspberry Pi is connected to the TP-Link router with and Ethernet cable and has a static default gateway set as the IP of the TP-Link router. It also accepts all incoming packets and forwards them through an VPN back to my home country, so I can both keep my streaming services running and protect any other guests in the complex from sniffing my traffic. To provide the DNS functionality, an dnscrypt-proxy server is running on the RPi.

The working setup with the TP-Link router and RPi zero

Throughput of the setup

When I have tested this setup at home, I have been able to achieve a stable connection of about 24 Mbit/s, which is good enough for general usage. Somewhat mysteriously, the performance of the TP-Link router has started to degrade from the day one. Noted, it had shown marks of misbehaving in the past, but powering the device on and off has always solved the problem (rebooting it through the configuration interface had never any effect). Right now, when measured with iperf3 with server on my laptop and client on the Ethernet-connected RPi zero, the results are only around 1.5Mbit/s.

What to improve

Evidently, these results are not enough for most of everyday usage as they are barely sustainable for reading news. So what can I do in the future to improve upon this setup? The most obvious answer is to buy a device that has a firmware update since 2020, but I don’t want to do that – it is still a (mostly) functional device and I believe, that I can find a more elegant solution. I have also tough about flashing another open source firmware, but neither OpenWRT not DD-WRT seem to support this device, while OpenWRT actively discourages from using it. As a third option, I want to try a setup in which I use my USB WiFi adapter, which I have unfortunately forgotten home. It could act as a strong client/AP directly for the RPi. So, until the next time, all I can do is to enjoy the view and not YouTube clips.

The view from the apparment balcony

Post-publish updates

#1 The setup is working again

After publishing this post, the TP-Link router has stared to behave properly for an extended period of time. I have no idea what has changed and so I continue to search for any causes.

#2 Why the complex’s repeater does not work

I have been able to capture the precise model of the <place>123-EXT AP – the D-Link DAP-1330. The device shows only one amber indication LED, which (according to the manual) means that it has a very poor connection to the base network. This could provide a hint as to why the extender AP has close-to-nonexistent connectivity to the Internet.

Manual showing that one amber LED means a very poor connection

A photo of the complex’s repeater AP

Backing up the router configuration

Fri, 01 Jul 2022 00:00:00 +0000

Recently, I have been fond of with Mikrotik routers and their enterprise-grade RouterOS system. There is plenty of possible configuration, which gives me great freedom of choice in how I want my home network to operate. Because setting everything, including VPN, guest network and more may take quite a lot of time, it is a good idea to backup the configuration.

The idea

My idea is to have a scheduled script that will save the router configuration to a directory inside my self-hosted NextCloud instance. This will also save me the hassle with versioning the backups, as NextCloud itself takes care of this.

The backup on the router is saved to its disc, which is accessible trough SFTP protocol, while the NextCloud can be accessed through WebDAV. RClone is an open-source solution that can manage both.

The implementation

Create a new account and group on the router – config-backup. When creating the group, I have to make sure to check ssh, ftp, read, write, policy, test and sensitive permissions.

Add router entry to a ssh config on the machine on which will the scheduled script run:

Host router
 HostName router
 User config-backup
 IdentityFile /secret/ssh_id_rsa
 IdentitiesOnly yes
 StrictHostKeyChecking yes

Create new RClone remote for the router using SFTP

[router]
type = sftp
host = router
user = config-backup
disable_hashcheck = true
key_file = /secret/ssh_id_rsa

Create new RClone remote for the NextCloud using WebDAV

[adam-cloud]
type = webdav
url = https://cloud/remote.php/dav/files/Adam/
vendor = nextcloud
user = Adam
pass = *****

Finally, combine all these setting inside a single Bash script

#!/bin/bash
ssh router -- /system/backup/save name=config.backup dont-encrypt=no password=123456 &&
rclone copy router:/config.backup adam-cloud:/Archive/router-config/ &&
ssh router -- /file/ remove config.backup

And that’s it. After running the Bash script, the router configuration backup is saved into my NextCloud.

More off-site backup storage is needed

Wed, 15 Jun 2022 00:00:00 +0000

Backing data up is vital if we do not want to lose it. My NextCloud server saves all data to a BTRFS partition with daily snapshots, which are then weekly synced with a local secondary disc in case of a primary disc failure. But what to do in case of a lightning strike that would overload surge protection? For that reason, I also keep an off-site Raspberry PI, which works as a tertiary form of backup. The only problem is that it had just run out of storage.

How is the RPi storage connected?

Because I want to run the things as cost-effective as possible, the RPi runs with a single USB connected 4.5TiB external harddrive. It is also attached to an UniPi extension, which allows me to control eight relays with REST API. This gives me an opportunity to power on and off the connected external hard drive whenever needed.

What is the off-site backup process?

Every week, the RPi powers on the connected hard drive with a single LUKS-encrypted BTRFS partition. Because the data inside the backup are write-mostly, I am using BTRFS’s transparent compression to maximalize available virtual space. After a successful mount, the RPi syncs data from the primary NextCloud drive using rsync over SSH (with options --inplace and --no-whole-file to keep the advantage of BTRFS’s CoW feature). The privileges separation is designed so that at no point in time does any of the servers have write access to the other’s data to minimize the possibility of human error deleting all of it. After a successful sync, a new BTRFS snapshot is created.

How to extend the storage?

Thanks to the fact that the disc is formatted to use BTRFS, all I need to do is to call a btrfs device add command on a second device. This seems like a job for my old 640GB Fujitsu-SIEMENS Storagebird 35EV821 (sale ended before 2012). Surely, before using it, I have to make sure that it still works properly, and because this old drive does not support SMART, I have to stick with a good-old baddblocks command (BEWARE: this command will rewrite the whole drive, do not copy it blindly):

> badblocks -o badblocks.log -svw -b 512 -c 65536 /dev/sde
Checking for bad blocks in read-write mode
From block 0 to 1250263727
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff:done
Reading and comparing: done
Testing with pattern 0x00: 60.57% done, 42:04:35 elapsed. (0/0/0 errors)
Interrupted at block 758382592

I caused the interruption because the badblocks was unlikely to find new damaged blocks after multiple successful passes, so it was expected. Furthermore, the backup will be tested monthly with BTRFS’s scrub to ensure data consistency so that any potential hidden problems will be soon flagged.

The disc seems OK, so it is time to connect it to the RPi. First - I need to cut the power wire and insert it into an UniPi relay so that I can power it on and off as needed, like the first disc.

Finishing the software connection

First, the disc has to be encrypted. Because it is now empty, it has no ID except for the standard /dev/sda path:

> cryptsetup luksFormat /dev/sda
WARNING!
=====
This will overwrite data on /dev/sda irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sda: ********
Verify passphrase: ********

Next, because the /dev/sda is not permanent and can change based on the time the kernel detects a new drive, I want to use the drive UUID:

> ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 9 Jun 4 18:06  1234UUID -> ../../sda

Now that we have the UUID of the disc, we can use it to open the LUKS encrypted content (which is currently empty):

> cryptsetup luksOpen /dev/disk/by-uuid/12134UUID cryptBackup2

And now comes the magic moment I have been waiting on – adding the storage:

> df -h /media/backup/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cryptBackup1 4.6T 4.6T 33G 100% /media/backup
> btrfs device add -f /dev/mapper/cryptBackup2 /media/backup/
> df -h /media/backup/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cryptBackup1 5.2T 4.6T 629G 89% /media/backup

Done! My off-site backup server has almost 600GB of new storage (effectively more because of the transparent BTRFS compression).

Conclusion

My off-site backup server has run out of storage, but thanks to the magic of BTRFS, extending it was a piece of cake! The process, of course, involved more steps like connecting the disc power to a controlled relay and using transparent encryption.

How I found (and fixed) a critical bug a day before official launch day

Wed, 01 Jun 2022 00:00:00 +0000

A day before launching the new online seminar Jump-start the FI (jump@fi for short), I have been doing some final testing that everything is OK and clear to launch. This new online seminar is expected to be officially advertised to the hundreds of newly accepted students of the Faculty of informatics at Masaryk University, so the stakes for making it behave correctly from the start are pretty high. Fortunately, about forty people have already tested the entirely new web frontend, so the assurance that nothing can go wrong was also relatively high. Unfortunately, everyone (me including) has managed to miss a critical bug in the application.

How does jump@fi work?

Jump@fi is an online seminar whose main components are tasks with tests that its participants can solve. These tasks are presented in the form of a directed acyclic graph, where the participant has first to solve all tasks with an edge pointing to the task before it is unlocked. So you would guess that this part is critical-bug-free as it was tested the most. You would be wrong.They are going to be stuck if someone does not fix it, which is certainly not the best start for an activity officially advertised by the Faculty of Informatics.

The bug

To take a look at the bug, let’s visit our test environment. As seen in the picture below, we have a greyed-out egg-like task in the second row with an edge pointing to it from the turtle task.

This arrangement means that the participant must first solve the turtle task before the egg-like task is unlocked (and is not greyed out anymore). Easy, let me just solve the turtle task, and we can move to the egg-like one.

Oh no. I have solved the turtle task (as indicated by the green check), and now I want to move to the egg-like one, but it is still locked! This is an action that most participants will perform and end up with a locked task, even though it should be opened by now. They are going to be stuck if someone does not fix it, which is certainly not the best start for an activity officially advertised by the Faculty of Informatics. We have found our critical bug, just a day before offical launchday.

The cause

After investigating the issue, I have discovered that the source of this issue is a logical mistake made when designing the task icon component. This is the code responsible for updating the task icon:

@Input()
set task(val: TaskWithIcon) {
  // subscribe to the task so that task info is updated on its solve, etc.
  this.task$ = this.tasks.getTask(val.id);
}

As you can see, this code takes care of updating the task icon whenever the task changes – this is how the green checkmark appeared in the turtle task icon. Do you see where the logical mistake is? The task itself is not updated; only its prerequisite is, and nobody asked the task icon to watch for changes of that.

The solution

The solution is simple – just watch for the changes in prerequisites of the task. The only problem is that when a task is solved, the user is on another page, so the icon of the connected task does not exist at the moment. How can something watch for anything when it does not exist? To our advantage, the Angular with RxJS comes to the rescue because it internally remembers the component state when we come back to the graph page after solving the turtle task. Let’s edit the task-watching observable to account for the prerequisites changes:

@Input()
set task(val: TaskWithIcon) {
  // watch tasks requirements for state change
  const requirementsIDs = Utils.flatArray(val.prerequisities);
  const requirementsState: { [requirementId: number]: string } = {};

  const requirementsChange$ = combineLatest(requirementsIDs.map((watchedTaskId) => this.tasks.getTask(watchedTaskId))).pipe(
    filter((tasks: TaskWithIcon[]) => {
      const changed: number[] = [];

      for (const requirement of tasks) {
        // compare states for changes
        const savedState = requirementsState[requirement.id];
        environment.logger.debug(`[TASK] requirement ${requirement.id} of task ${task.id} was ${savedState} and now is ${requirement.state}`);
        if (savedState !== requirement.state) {
          requirementsState[requirement.id] = requirement.state;
          if (savedState !== undefined) {
            changed.push(requirement.id);
          }
        }
      }

      const isChange = changed.length > 0;
      if (isChange) {
        environment.logger.debug(`[TASK] requirement of task ${val.id} have changed their state: ${changed}`);
      }
      return isChange;
    })
  );

  // refresh task's data when its requirements change
  const refreshTaskCache$: Observable<boolean> = merge(
    requirementsChange$.pipe(mapTo(true)),
    this.tasks.getTask(val.id).pipe(mapTo(false))
  );

  // subscribe to the task and so that task info is updated on its solve, etc.
  this.task$ = refreshTaskCache$.pipe(
    mergeMap((refreshCache) => this.tasks.getTaskOnce(val.id, refreshCache))
  );
}

Wow. What an ugly piece of code. It works, but it could use a little separation into multiple smaller functions. Let me move the requirementsChange$ Observable to a separate function, away from the task icon component to the Task service, which is already responsible for watching for the changes in the task itself.

// TASK SERVICE
/**
 * Watches for changes in the state of the requirements of the task
 * @param task task to which requirements watch for
 * @return Observable<true> emits whenever the state the requirements has changed from the initial state
 */
public watchTaskRequirementsStateChange(task: TaskWithIcon): Observable<true> {
  // watch tasks requirements for state change
  const requirementsIDs = Utils.flatArray(task.prerequisities);
  const requirementsState: { [requirementId: number]: string } = {};

  return combineLatest(requirementsIDs.map((watchedTaskId) => this.getTask(watchedTaskId))).pipe(
    filter((tasks: TaskWithIcon[]) => {
        /* ... same as before ... */
    }),
    mapTo(true)
  );
}

// TASK ICON COMPONENT
@Input()
set task(val: TaskWithIcon) {
  // refresh task's data when its requirements change
  const refreshTaskCache$: Observable<boolean> = merge(
    this.tasks.watchTaskRequirementsStateChange(val).pipe(mapTo(true)),
    this.tasks.getTask(val.id).pipe(mapTo(false))
  );

  // subscribe to the task and so that task info is updated on its solve, etc.
  this.task$ = refreshTaskCache$.pipe(
    mergeMap((refreshCache) => this.tasks.getTaskOnce(val.id, refreshCache))
  );
}

This is much better. Now we can call the service function from the component and have a nicer-looking code. Except for one small thing - it does not work anymore. As I found out, for some reason, the requirementsState dictionary empties itself when used inside the service, while it keeps its content when inside the component. Also, when I tried passing the requirementsState from the component as a parameter to the service function, still it emptied itself every time. Curious.

The final solution

At this point, I have concluded that I have to store the requirementsState in some static cache. But when to decide what and when to store, to prevent unwanted memory consumption? Fortunately, the task service already has a caching mechanism for tasks. We can just expend that for also caching task prerequisites state! It also makes sense that we will not need to watch for changes in prerequisites of tasks that were not seen long enough for it to be exempt from the cache. So does that work?

Yes! It does! Finally!

Why has nobody discovered this bug before?

The only thing that remains is to answer why has not even one of the forty people (myself included) that have tested this web application noticed this critical bug? These forty people can be divided into three groups:

developer (me) - 1 person
organizers of the Online seminar of Informatics (KSI for short) ~ 16 people
KSI participants ~ 22 people

Note: KSI is another online seminar which uses a slightly modified version of the web application (e. g. different colours)

As a developer of this web application, I have missed this bug, possibly because it would vanish when the page reloads itself, which automatically happens after a source code change. The KSI organizers see all tasks already unlocked, so they were unaffected by this bug. Nevertheless, the last group, KSI participants, are affected by the bug only if they have not already solved all tasks, which is possible because they were given access to the web application in the last part of the KSI year. Oh well.

Conclusion

Just a single day before the official launch day of the new online seminar, I discovered a critical bug in the web application that would make all its participants stuck without the possibility to progress. After four hours of debugging what exactly went wrong and why sometimes a dictionary kept its state and sometimes it did not, I have settled for a solution. The most fun part is that this application was quite well tested, but everyone has missed this bug due to bad timing.

Adam's blog

My new home data backup strategy

Previous setup

Current setup

Closing words

Honorable mentions

(CZ) Technický průvodce získáním pedagogického osvědčení

Pedagogické minimum neexistuje!

Hledání správného kurzu

Úřad práce

Krok první – zájemce o práci

Krok druhý – vyplnit formulář

Krok třetí – zamítnutí

Praha

Kurz samotný

Týden první

Týden druhý

Online výuka

Týden třetí

Týden čtvrtý

Týden pátý

Závěrečná práce a zkouška

Horší zkušenosti

Certifikát

Finanční okénko

Závěr

Bonus: Pražský motoráček

2025 Recap: Books & Podcasts

Books

Podcasts

Meeting a scammer in Minecraft

Joining a new server

Playing with a new old friend

Verifying Minecraft account

This is what the scam is all about

Conclusion

2024 Recap: Books & Podcasts

Books

Podcasts

Songs

Stopped self-hosting everything

Email

VPN

Web apps

Conclusion

DNS Bonus

Cloudflare Access to get to LAN

Classical solutions

Public IP address

VPN service with exposed ports

Creating a custom tunnel with VPS

Using a dedicated P2P software

Cloudflare Access

Cloudflare Access Application types

SaaS (Software as a Service)

Private network

Self-hosted

Cloudflare Zero Trust

Access policy

Identity management

Cloudflare Tunnels

Preventing double-logging

SAML

Cloudflare Access cookie

Accessing SSH

Conclusion

Creating safe WiFi abroad, Vol. 3

Setup & installation

Wireless trouble

Real-world usage (fail)

Join to start a Minecraft server

AutoStop & AutoPause

Recreating the behaviour

Making it user-friendly

What users see

Cat-proofing the Raspberry Pi

It’s hot & loud

Not a “fan” of high tempeature

Governor!

Then it clicked