Reliability practices, infrastructure, automation
Every organization has a graveyard of bloated Terraform modules. We catalog principles that separate modules teams want to use from modules that become technical debt — parameter discipline, output design, state boundaries, and when not to write a module at all.
TerraformIaCArchitectureThree years of running Istio and a brief Linkerd experiment taught us that the default answer to "should we adopt a mesh" is no. Here are the signals that mean you actually need one, and the operational costs nobody mentions.
Service MeshIstioKubernetesFrom the KRaft migration to partition count discipline, ISR tradeoffs, and the compaction trap. Four years of operating Kafka at growing scale, distilled into the configurations and habits that prevent the incidents we have already debugged.
KafkaStreamingOperationsWe cut annual cloud spend by 31% while improving p99 latency. No single optimization moved the needle by more than 6% — but the combination of attribution, right-sizing, spot tiers, and non-prod discipline compounds.
FinOpsCloudCost OptimizationDNS is the protocol everyone assumes works. It also causes more subtle Kubernetes incidents than any other component in our experience. CoreDNS tuning, ndots:5, the conntrack race, and the observability gap that hides it all.
DNSKubernetesNetworkingPlatform teams often find themselves trapped: they build infrastructure but cannot answer "how reliable is it?" We break down how to define SLIs for internal platforms and negotiate SLOs with product teams.
SRESLOPlatform EngineeringWe ran both tools on clusters ranging from 50 to 400 nodes. Here is where each one excels, where it struggles, and why the choice is about team culture, not technology.
GitOpsKubernetesArgoCDBlameless postmortems are not just a document template. They require systematic changes to processes, metrics, and even the language teams use to describe failures. Here is a framework that works.
Incident ManagementSRECultureeBPF is no longer exotic — tools like Cilium, Pixie, and Tetragon make it accessible. We show how to get traces without code instrumentation and profile CPU with zero overhead in production.
eBPFObservabilityLinuxToil is work that scales linearly with service growth. We introduced a toil budget for every team and automated everything that exceeded 30%. Here is what happened.
SREAutomationToilThe OTel Collector is the backbone of modern observability stacks. We share our battle-tested pipeline configuration for processing 2M spans/sec with filtering, sampling, and multi-backend export.
OpenTelemetryObservabilityArchitectureResource requests and limits are only the beginning. Real capacity planning involves understanding burst patterns, node failure domains, and scheduling headroom. A practical guide to right-sizing clusters.
KubernetesCapacity PlanningInfrastructureOn-call rotations burn people out when done wrong. We document our approach to on-call that keeps engineers sane: escalation policies, runbooks, alert fatigue reduction, and compensation models.
On-CallSRETeam HealthRunning PostgreSQL at scale requires more than replication. We cover our HA stack with Patroni, PgBouncer connection pooling, WAL archiving, and the failover scenarios that caught us off guard.
PostgreSQLHigh AvailabilityDatabasesYou do not need a Netflix-scale platform to benefit from chaos engineering. We walk through lightweight fault injection using LitmusChaos and simple scripts that exposed real production blind spots.
Chaos EngineeringResilienceTesting