Observability Migration - A new approach

https://www.cloudraft.io/blog/influxdb-to-grafana-mimir-migration

Discussions: https://discu.eu/q/https://www.cloudraft.io/blog/influxdb-to-grafana-mimir-migration

Job search journey as a DevOps/SRE/Platform engineer in Netherlands/Amsterdam(Dec '24 - Apr '25)

http://cargo.one/

Discussions: https://discu.eu/q/http://cargo.one/

Hello, hachyderm! we've been working hard on building up our ansible runbooks and improving hachyderm's overall resilience. Recently, we've been focusing on is database resilience.

We're getting close to retiring our original database server (finally!) and preparing to move to a fully ansible-managed set of databases servers, primary and replica on new hardware. We'll send another announcement when we do the cut over. The team has done excellent work to make this highly automated, quick, and painless!

Done:

author ansible roles for managing postgresql, pgbackrest (backups), pgbouncer, and primary/replica failover
decide to continue with pgbouncer and *not* use pgcat
rotate database passwords
order new replica database hardware
order new future primary database hardware

To do soon:

rebuild replica database with ansible scripts
prepare primary database with ansible scripts
start replicating to new database replica
cut over to new database server

We're also planning on open-sourcing our ansible roles in the coming weeks - just a little housekeeping & tidying up before we do!

hey, fediverse friends - i'm excited that we're finally announcing our Fediverse Security Fund over at @nivenly to help make fedi software more secure.

we're starting off super small to see if the Fund is a thing that can help. along the way we'll learn and improve our intake/payout process. and if there's solid interest and we see good impact, we'll hold a member vote near the end of the experiment to decide if we'll renew/expand the program.

thanks to @thisismissem for her contributions and being the first disclosure to validate the process.

let's close some vulns!

If you're at #KubeCon today (or the rest of the week?) do go say hi to the #CUE folks I help write their open-source docs (https://cuelang.org/docs) 'cos I *really* want the awesome tech to succeed!

If you're a #SysAdmin or #DevOps in #YAML config hell go and have a chat with them - next to the CNCF corner store at stall S761

And that concludes my job in infosec.
Tomorrow, I will be in IT working on Linux.
#infosec #cybersecurity #SRE #linux #foss

Pushing core workout lately and being rewarded with more mornings free of migraine.

I played deeply into my music the past few nights, awaking the next morning scrubbed of a migraine.

Having those who listen and witness allows me to let go of emotions when I am having them, not carry them around. Less migraine activity ensues.

This week I learned that my anxiety about others is entwined with a particularly evil symptom of religious trauma, I saw both but never saw hiw they were connected.

I can recognize it now. And the feeling of not needing to "save" someone is a really powerful emotion - or lack of one - that, today, I am thankful for contributing to a clear head and no migraine.

Also feeling self-assured that fixing failures in our systems look a lot more like treating a migraine than using quick-fixes and low-hanging-fruit.

howdy, #hachyderm!

over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.

since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.

to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.

then, after we've flipped the switch and while DNS is propagating -- -- we can watch in real-time as different servers begin flipping over to the new provider.

we named the tool hachyboop and it's available publicly --> https://github.com/hachyderm/hachyboop

please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code.

attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.

hello, #SRE friends. looking for good resources for SRE-related content, specifically for #observability please share your fave resources, TY

Ugh slept like shit. The drama and stress from work is giving me insomnia. The boss who quit didn't give leave ANYTHING for my new boss to understand what I do with DevEx and incidents.

I had to bust my ass yesterday to basically hold the ground that I had worked extremely hard over the past four months to make it so that I could manage incidents. New boss wanted to yank me out of it and put me back on "SRE" infrastructure.

CTO says incidents are staying where they are, with Engineering. So I told him I want to transfer to Engineering.

And then I get the question "do want to be an SRE?"

Ridiculous. Makes me want to scream.

I really liked this informal community poll and thematic analysis on SLO usage. It does a better job at highlighting the hurdles to adopting them at a Company Who Is Not Google than a lot of "Here's how to do SLOs" pieces that just don't cover it.

If there is ever a "Seeking SLOs" book, this should be the first chapter.

https://ericmustin.substack.com/p/notes-on-service-level-objectives

Google has open-sourced a visual timeline tool for Kubernetes that helps developers trace cluster issues

#Kubernetes #GoogleCloud #DevOpsTools #Observability #OpenSourceTools #CloudNative #SRE #InfraOps #ClusterDebugging #K8s

https://winbuzzer.com/2025/03/24/googles-kubernetes-history-inspector-reinvents-cluster-debugging-with-timeline-based-log-visualization-xcxwbn/

What a friggin wild first quarter I tell you what. As of today lost my boss and his boss so no director of SRE and no VP of TechOps.

I am ... I am in the midst of patterns I have experienced in the past, but in much more high stakes positions. We are under a reorg and I will be honest ... the thing probably saving me from cuts is that I took a low salary. No clue what's to come... except I kinda do have an intuition about it.

I see something happen in the way a CTO handles a shift in power that I have begun to find intuitive because I have witnessed all sides of it so many fucking times. Like I know what they're doing, and I know where they're going next. When things happen, like heads of departments are just gone one day, I have seen this battle before. Ain't my first rodeo, try ain't my tenth.

Hard to tell how things will shake down. Ultimately I think the work we're accomplishing in DevEx is carving out a recognized need that makes my little team's center for excellence work indispensable. I have worked damn hard to forge some bonds there that I could not do elsewhere.

I guess one feeling I have is that I can only go up from here as long as I can hang on. Now that I can get over freaking out about it, I can enjoy the ride instead!