Unpublished Musings about software
The Release Pipeline you can't have
21 Mar 2021
Let me tell you about the Release Pipeline we nearly built at one of my ex-job. It partially cannot be done today because the technologies does not exist or are not composable enought to be used this way for now. But we are really close and I really really want this to become a thing. Because the benefits for everyone would be too big.
No One Like You
Let think about what we want from a high level standpoint. The first thing is that I want the team that own the service to own the whole pipeline. If there is infra done for them, it is at best a set of VM and some networking. That way, the pains are theirs and they can follow what happens. And adapt as needed.
The second thing we want is for the Build system to know nothing about the deployment side. All it does is produce an artefact, Content Adressed by the hash of its content, and push it to the Content Adressed Store used. At best it is allowed to set a flag on the content to tell if it is allowed to be deployed yet. In the Store.
That store need to be hostable in a variety of ways. Artefactory, S3, a filesystem,… This generate a third constraint. On top of that, the store need to be able to communicate in a way with the service, to tell them a new version is available. Or the service need to query it regularly.
On fourth constraint, the service needs to be able to self orchestrate a deploy of the new version, between all the machines that make its deployment set. It needs to also be able to coordinate with external services that needs it. At the very least, to not disturb in flight work too much.
Too Drunk To Drive Drunk
If the Solarwind hack told us anything, it is that CI/CD are RCE-as-a-service. So to mitigate this kind of supply chain attack, we want our Build system to follow a few rules. It always start with a blank env. The inputs and dependencies need to come from a CAS. And verified on their hash. The output will be a CAS artifact too. Everything is rebuilt from source, the only exception being that you can use the CAS as a cache. Every build end with the machine being scrubbed clean.
The recipe itself is then CAS stored and is the package you note as deployable. You are not allowed to know more. The Build system live outside of the rest and never talk to anything, except strictly controlled by its contents hash.
The typical examples here are Bazel, Nix, Guix, Habitat, …
That means that the team is responsible for updating their dependencies. To mitigate the cost, it may make sense to use a central team to update “low level” dependencies. In particular, things like libc, compilers, classic GNU utilities, …
This is the part you can have these days. Nix and Guix are well advanced and have a nice set of packages as a base to build on top. Habitat have a nice set of packages.
Takin’ Up Space
On the service side, this get a bit harder.
On one hand, swapping the service atomically on the machine is doable. NixOS, Guix, Habitat even, they all can do it. A pretty good start.
Publishing the recipe and using that as a base ? Same thing, we can do it.
Getting the information an update is needed, that gets harder. There are mechanisms, but outside of Habitat
Otp25 Float To String
01 Mar 2022
In OTP 25, you will have access to a new option for
float_to_binary/2. This was the result of work i contributed with the support of the EEF. Why did we do that and when should you use it ?
What Diana have done, H1-2022 Edition
14 Sep 2022
So we are now past the first half of the year and i have not published a lot. I feel like i should at least update a bit on what i have been doing in the first half of 2022.
New Company, New Job Title
First of all, i am now a SRE at Indeed. I started looking for a new job at the start of 2022. Community was great, but the pace and risk of a startup in this kind of economy was not worth the salary i was getting. The offers i found as an Elixir engineer at the time were… underwhelming. Far too many red flags, far too many company that seems to not understand remote work.
I then got approached by Indeed. They were launching a new SRE team, centered around Incident Management. This fit right in the kind of stuff i have been learning about and exploring in the Learning for Incident and Resilience Engineering communities, so it felt like an offer i could not refuse.
The Interview process was the usual load of bullshit. But i am happy to say that since April 2022, i am now a Senior SRE in the Incident Management team at Indeed.
Shipping Ryu in OTP 25
OTP 25 ship with the
float_to_list/2. It gives significant boost in performance and significantly reduce the memory impact of these operations. It may sound niche, but it is quite impressive on JSON encoding, as Michal Muskala showed on the Jason project. Please use it if you are doing erlang. And for Elixir, you can use it already but your library will slowly upgrade to it as we keep dropping support for older versions of OTP.
I have spent a significant amount of time in 2021 to get that in. I had financial support from the EEF for part of it, which i am thankful for. To know more about what is entailed by this change and how it works, you can go watch the talk i gave at elixirconf.
That said it bring us to the elephant in the room that i think deserve its own point.
The BEAM Situation
I am no longer being paid to work in the BEAM community. I am still bullish on Elixir, Erlang and the BEAM, and use it for all my personal projects. As you can see, i am still in there on my free time. I would have preferred to stay in the BEAM engineering community as someone paid to do the work.
But i am getting really scared of the direction of the community right now. Don’t misunderstand me, José, Chris and all the maintainer are doing amazing work. The OTP team does great thing. But the organizations that use the BEAM seems totally disjointed from the ecosystem.
As of today, there are still only one or two person paid to work on the BEAM itself outside of the OTP team at Ericsson. No-one is paid to work on rebar3. Hex does not make enough money to pay the maintainers. The EEF does great job, but is nowhere near the amount of donations that would allow for engineers to see it as a force to bet on to put food on the table. Contributing to OTP is still a massive pain in the ass, with tooling that simply do not work and take hours to setup. Cowboy and its dependencies are not actively being worked on because Loic need to put food on the table. Fred Hébert, maintainer of tons of resources essential to the community, had to move out to SRE to find a job that fit his needs.
And the orgs keep saying they have trouble recruiting, while the foundation are crumbling due to chronic underinvestment. Let’s be clear. I am not saying that other stacks have a far better prospect. This is a reality that you could find, at one level or another, in every language and runtime. But the BEAM community is supposed to be about robustness and anticipating the future failures. And in this domain, what i see is willful blindness.
And i understand why WASM or ML are areas to go for in order to attract new domains and new people to come in. But consider that to build on top of foundations, you need them to be solid. And the BEAM one may not be as solid as you think. If you want to know more, feel free to contact me, i have receipts and ideas.
Ok, Rant Over.
I spend a good part of my free time the past 6 months working on my fork of the
hyperreference library in erlang.
One of my personal project needed a good distributed way to do count distinct queries over massive sharded datasets. So i went looking at the erlang reference implementation. It is used in Riak, it should be good right ?
- It use rebar2 to compile, which is definitely not supported anymore
- When porting it to rebar3, it fails to run its tests due to a dependency on
:randomwhich was deprecated in favor of
:randa few years ago in erlang.
- After porting the test suite to
:randit became obvious that the library was… wrong. The test suite would not pass, purely based on it having a different set of random input, despite the invariant being tested not depending of the input in theory.
- It was not even on Hex.
At this point, the library was simply unmaintained for years. And bugged. So time to fork! And publish on Hex. In order, i ported it to rebar3. Ported the tests. Added
ex_docfor documentation. Published it on Hex. And then started the work of making the test suite make sense (no, Proper is not a test runner for unit test), fixing the bugs and trying to make the codebase a bit more maintainable.
In the process, i managed to find a new, more precise, way to estimate from a paper from Omar Ertl. I added a few things i needed for safety like versionning, in order to not merge two HLL that are not compatible, which kinda matter in distributed systems. I have bigger plans for this, and it is definitely not at version 1.0 yet. At the very least i need better documentations.
But i think at this point we are quickly coming to the point where it would be the most advanced and complete hyperloglog implementation in all languages i have been able to find. Not yet mind you, but eh.
And not a lot more.
And that is mostly the extend to my FOSS work on the past 6 months. It may not sound a lot, but the reality is that i probably can only find 2 to 4 hours a months to work on “personal” projects that do not pay me. And that limits what i can do. That is also why i have published so few things. A post take easily 2h to write, check for language errors and publish. It competes with all the rest for this limited time.
This is also why i am quite scared for the BEAM ecosystem. This situation is not specific to me. If we want things to evolve in the lower levels of the runtime, we are going to need the community to step up to support it. We cannot depend on Ericsson for everything. They have been great steward, but at a point, they have their own priorities.
To the next time, i hope you are all having fun in your own projects in the meantime.
The Missing Middle of Funding Tech Companies
05 Dec 2022
Tech companies that make money for their investors seem to mostly… not exist. And the reason for that is that we have no financial instrument to fund them.
In this post i will review the existing financial instruments usually offered to finance the creation of a new firm. Try to show you these instruments do not work for tech company that want to make money for their investors. And leave you at that.
A SSR dev Browser API wishlist
08 Mar 2023
Last time, I shared my view on the current state of the browser APIs for SSR developers. This time, to not be seen as only criticizing, let’s see at what is in the pipeline, or could be in the pipeline, from the browsers, to make our life easier.