gRPC, Haskell, Nix & love-hate
Discuss the state of the gRPC ecosystem in Haskell in the context of Nix, presenting some open-source fixes, and making sense of the current (reproducibly) half-functioning solutions.
Contents
Open source TL;DR #
- https://github.com/purplenoodlesoop/http2-grpc-haskell
- https://github.com/purplenoodlesoop/gRPC-haskell
Structure of a gRPC toolkit #
A toolkit/suite for a proto/gRPC for a specific language is a concept that this article is talking about. A toolkit consists of two mandatory public-facing parts that enable its functioning – (1) a language-specific implementation of the API with core types and functions, and (2) a code-generator from .proto files.
service VoiceService {
rpc Call(stream InputPacket) returns (stream OutputPacket);
}
Each implementation would differ, as every part can be represented very differently – arguably, some better than others.
Ecosystem Overview #
On Hackage, complete implementations are http2-grpc-haskell and gRPC-haskell. Both packages are heavily undermaintained, but mostly not outdated and fully usable.
This results in some of the toolkits' packages being broken both in vanilla Haskell and Nix ecosystems or requiring obscure, outdated dependencies themselves. Given that the implementations are very much useful, it makes sense to provide the fixes.
On mu #
mu-haskell was an interesting project! Searching for gRPC lands on it as one of the solutions, but the project was archived on Oct 19, 2024. The project received its last commit almost two years ago, and the proto/grpc part is especially outdated. Their Examples, and source code are of a very much value, though.
mu-haskell uses WAI, so does http2-grpc-haskell
http2-grpc-haskell #
A full, full-Haskell implementation of the protocol. Does not rely on external C, and implements everything on top of native abstractions. Flexible, correct, and idiomatic.
Implementation #
A higher-level implementation of warp-grpc is the most likely part of the library for an interaction. The toolkit uses wai and warp to implement the rpc part, and the .protos are generated using an idiomatic gRPC compilation plugin that generates proto-lens message and service definitions.
''
protoc \
--plugin=protoc-gen-haskell-protolens=${getExe pkgs.haskellPackages.proto-lens-protoc} \
--haskell-protolens_out=packages/backend/gen \
packages/proto/core.proto
''
The API is somewhat of a trap, however. It poses a similar flaw to grpc-web in missing an important part of functionality – asynchronous, bidirectional stream communications are impossible, as each event can be met with a 0-1 response in the borders of the same message.
handleIndex :: UnaryHandler GRPCBin "index"
handleIndex _ input = do
print ("index"::[Char], input)
return $ defMessage & description .~ "desc"
& endpoints .~ [ defMessage & path .~ "/path1"
& description .~ "ill-supported"
]
nixpkgs #
The toolkit is mostly broken in the Nix ecosystem, but it is easily fixed in the form of an overlay. After bumping dependencies everywhere to:
- bytestring >= 0.10.8 && < 0.13
- http2 >= 3.0 && < 5.3
- warp >=3.3.15 && <3.5
- text >= 1.2 && < 2.2
- tls >= 1.4 && < 2.1
- zlib >=0.6.2 && <0.8
Running cabal2nix on updated .cabal files generates valid .nix files that can be used in an overlay drop-in.
{ lib }:
self: super:
let
packages = [
"warp-grpc"
"http2-grpc-types"
"http2-grpc-proto3-wire"
"http2-grpc-proto-lens"
"http2-client-grpc"
];
genName = "package.gen.nix";
foldl = f: lib.foldl' (packages: name: packages // { ${name} = f name; }) { } packages;
in
{
haskellPackages = super.haskellPackages.override {
overrides = hSelf: hSuper: with data; foldl (name: hSelf.callPackage ./${name}/${genName} { });
};
}
The fix can be found here and applied as a default overlay. The fork is based on another fork with dependency bumps.
gRPC-haskell #
Implementation #
Shared C usage. This implementation is really a wrapper around a mostly canonical implementation, which at the same time makes it a much less flexible implementation that does not allow room for using idiomatic underlying implementations AND provides a full API.
This solution allows for all of the standard protocol functionality.
addHandler ::
ServerRequest 'Normal TwoInts OneInt ->
IO (ServerResponse 'Normal OneInt)
addHandler (ServerNormalRequest _metadata (TwoInts x y)) = do
let answer = OneInt (x + y)
return
( ServerNormalResponse
answer
[("metadata_key_one", "metadata_value")]
StatusOk
"addition is easy!"
)
The API departs from idiomatic context integrations in one more way – .hs code is generated by a separate program and not as a plugin in the protoc compiler.
{
env = with pkgs; [ haskellPackages.proto3-suite ];
text = ''
compile-proto-file \
--includeDir packages/proto \
--proto core.proto \
--out packages/backend/gen
'';
}
The resulting code is somewhat more verbose, but it is a matter of utility functions and types, which are absent in the core implementation.
nixpkgs #
Again, broken, again, fixable with an overlay. The project does not use flakes, and using the latest unstable channel would result in compilation errors. The fix is two-fold.
Bumping bytestring:
- bytestring >= 0.10 && <=0.12
And utilizing an older version of gRPC
nixpkgs-grpc.url = "github:NixOS/nixpkgs?rev=d59a6c12647f8a31dda38599c2fde734ade198a8"; # gRPC 1.45.2
Running cabal2nix and building nixpkgs with an overlay that overrides Haskell packages and grpc the project is built.
Favoring over other solutions #
The differences might seem subtle, but they are quite important and I see a choice as somewhat lacking.
Guarantees of implementation #
gRPC-haskell expects a record with a service implementation, which allows the compiler to check its completeness of it, and http2-grpc-haskell would allow the compilation to run, even on incomplete, patchy service implementations.
voiceService state =
voiceServiceServer VoiceService {..} defaultServiceOptions
where
voiceServiceCall :: RPC 'BiDiStreaming InputPacket OutputPacket
voiceServiceCall (ServerBiDiRequest _ req resp) = do
-- ...
Type inference #
Not using fully abstract, lens-powered records allows for a neater type-completion, with the compiler being able to interact with the records closer.
Bi-directional streaming #
Already been mentioned, but this is one of the reasons that pushed me to fix a second library – the imperative, persistent interface for receiving and requesting events asynchronously in independent directions.