Meetups/Infra/2026-03-02

outages + kinetic effects, backups, password + secrets management, AOSAbook.org, hetzner, uptime kuma, nix, macaroons

Join Zulip Markup/markdown comparisons

Introductions

• [name] - [background]. [goals for meetup, or interests to explore]

Loren - Cloud engineering and data scraping, wnat to explore b
Ciara - play with k8s, helm, network VyOS, did network work previously, trying to do less of that
Jet - been using a lot of nix, self hosting a bunch, helping noisebridge with that
Zaccae - A lisp addict, would be a real infra guy if not so stubburn about lisp, just can't stop
Lucifer - just observing
Chinese Man - Believes in local models, local weights. Host your own, never pay for AI!! (great cheers)
Renaud - recent grad, likes operating systems, excited to learn about what we're talking about
Doug - likes computers, self hosting, doesn't like paying for AI, local models for local people, here to learn.
Erik - making a bunch of things with local models here at the space, made a docker ontainer for AI projects, recently grabbed latest qwen model 4.5, running locally as coding environment, but needs better hardware.
Robert - interesting DIY, GUI, operating systems, getting into hacking different devices, like IOT 
Derek - doing Spec-Driven Development, building a library around that. Developing a locker tracking system
Alex - claude gave me a hat, in the mail today, lot of vibe coding, doing a lot with agents, interested in backup architectures.

Topics, Lessons, or Demos

• Post Mortem
  • Amazon's data center in UAE
    • hit by debris, out for several hours

not all data loss, just liveness 
after service degredation, just reboot and backups happen automatically.
Claude's relationship with DoD
OpenAI's killbots are inevitable?

• Alex asks:

move login auth into claude.
operating a hundred claw'd bots, collaborating with each other
Agent walled gardens.
proxy network?
met a dude who would flash routers with firmware,

• Residential proxies

people allowing home connections
scandal: set top boxes with hidden residential proxies.
A residential proxy SDK for android

• Architecture
  • Backups

- "X is not a backup"
3-2-1: 3 copies, 2 types of media, 1+ offsite

Elan, NB locally, docker registry on cluster
practice recovering your backups, if you haven't checked them, they might not exist

people say snapshots are as good as backups--it is not so, they're on the same machine. Until they're copied elsewhere, they're not
There are now PCI reqts to set up too
Issues where — you can afford 1 copy of your data but not 2. 
When you buy or rent a server--should plan to be able to restore the same workload twice over.
Had a user, using Synology, with TBs of data, would regularly delete 300GB virtual file system

• • Glacier: why take 12 hours?

Tape archives
optimized for internet network
used for medical data -- for millisecond-responsive responses, backed all the way to tape
lowest cost guarentee retreival 
can ship drives to amazon for backup
For really large data, they'll drive trucks (AWS Snowmobile) to carry spinning rust.
egress is expensive
Jet wants to start a business trucking around drives for ppl. Brilliant!

• • How much data is the cost threshold for trucking around drives?

bandwidth of a station wagon?

Who's following 3-2-1 backups?

a few people, 1/3.
Why different mediums?
 If there's a bug in software related to medium storage
Ebay has good deals on tape drives.

https://girl.surgery/graph

• • Secret Management

What are some secrets people keep?
 passwords
 (relationships)
 credit cards
 routing & account numbers
 API keys. sk_, 
 
 personal, business
 
notebook
memory palaces
storage units
YubiKeys
password managers

complicated -- nix system locally

Jet's custom email server on a nix box.
 managing many secrets
 nix used to configure a server with some text files...
 secrets.nix imported into a server config.
  good for deploying
  how many of these secrets are needed elsewhere simultaneously -- not just ephemera of infrastructure?
   "changeme" for more robust password management
    when nix builds the server deployment, the secrets are generated.
     macaroons, uses hashes to subspecialize cookies.
     

ways of sharing secrets, or delegating 
 Kubernetes
 Hashicorp
 Platform as a service
  Heroku
  Render
  Vercel

note the risk of using an email from a self-hosted domain, when registering accounts with a dns registrar. You may need to do recovery, and self-bootstrapping may not be possible if your domain is expired and not pointing to your own expected mail servers.
Google is a great at account recovery.
Self hosters won't be typical users, not everyone will do it.

• • MediaWiki architecture

https://aosabook.org/en/v2/mediawiki.html
https://workingwithmediawiki.com/book/
mediawiki built with PHP, 


https://aosabook.org/en/v1/asterisk.html

• jet demo -

Kuma - uptime monitor

Outros

Loren 
Jet - 

• W-ez - running a backup!
• Robbie - learned about uptime kuma
• Ciara - learned about the AOSAbook.org site -- for erlang/otp, mediawiki
• Renaud- learned about AOSAbook site
• Will - learned how cheap it is to host on hetzner
• Zachae - uptime kuma and grafana (self hosters paradise)
• Doug - can host uptime kuma, publicly, host

with nix

• Erik - like to do some more monitoring, dashboards.
• Robert -
• Alex - will think about key management more.
• Derek - will be looking at nix+macaroon