Tweag Blog

Tweag [0] has great content supported by a high contrast, brutalist(?) website that I find quite attractive. [0] https://tweag.io/

I didn't find any architectural details on QPX after they've been acquired by Google. For Haskell I remember reading about Cloud Haskell ( https://haskell-distributed.github.io/ ) which allows you to write Erlang-style systems and Sparkle ( https://github.com/tweag/sparkle ) by Tweag ( http://www.tweag.io/ ) which allows you to use Haskell with Apache Spark. But those are tools and libraries and not complete production systems. I'd love to read about Haskell or Common Lisp distributed system implementations if you have any pointers!

Tweag.io | Paris, France | REMOTE + ONSITE | Software Engineer | Formal methods, PL design http://tweag.io We are a research and development laboratory at the heart of Europe, applying functional programming techniques to tame the complexity of distributed systems and scale predictably. Most of our existing folks have a PL research and/or formal methods background and enjoy demonstrating the correctness of their solutions with gusto: randomized test case generation, model checking in SPIN/Promela or interactive theorem proving using Coq. We're very active in the Haskell community - contributing new language extenions (linear types, static pointers) to GHC, publishing and maintaining HaskellR, inline-java, sparkle, rules_haskell and many others. We hit the HN front page just yesterday with our Haskell support in Bazel. Tweag.io are organized as a distributed team of experts around three main areas of focus: distributed systems engineering, functional programming based devops (Nix, Haskell etc) and mathematical modelling / machine learning. We are looking for formal engineers and PL designers to join our distributed systems team, to help us apply interactive theorem proving techniques to industrial scale projects and work on tools to make this cost effective for the masses. If you'd love the opportunity and the space to solve the hard problems of science's large dataset infrastructure, by systematically decomposing them into simple, orthogonal solutions that compose and…

> Yup, it's mostly just the story and tools we used to get ourselves out of a mess that was made harder by some decisions made earlier -- the tests were running in a container with stripped symbols (we're going to ship symbols after this, no reason to over-optimize) It's worth noting here that you can also build your binaries and keep debug symbols separately. You don't need to ship them with the binary (although it will make many scenarios a bit simpler if you do, since you'll always have the right ones available). Some info that might help: https://www.tweag.io/blog/2023-11-23-debug-fission/ https://undo.io/resources/gdb-watchpoint/reduce-binary-size-compile-ti…

That makes a lot of sense, thanks for all the answers! It's definitely a very interesting project and I think that once more programs can work without changes, it is something that could perhaps be used for building packages by distros that have displayed a special interest in (and can benefit significantly from) reproducible builds -- like Debian for instance, but especially NixOS due to the existing work on content-addressed paths (explained here: https://www.tweag.io/blog/2020-09-10-nix-cas/ ). The serialization of threads (and I assume processes?) would probably be an issue for large packages, but at least it would be possible to build many packages in parallel once the base dependencies have been built. I had thought about starting such a project myself (for reproducible builds), but of course, the amount of work is very significant and for me it was not worth the benefits :)

As an aisde, i love Nix, but i hate the lack of types. It's such a chore to figure out what fields are on a value, is the value a func, what are the docs for that func, etc. I have high hopes for Nickel[1] [1]: https://www.tweag.io/blog/2022-03-11-nickel-first-release/

> Ideologically, I love nix. The syntax/language itself is a massive barrier IMO. Really rooting for the Nickel project which is a new configuration language aims to allow for gradual typing and contracts for Nix expressions: https://www.tweag.io/blog/2020-10-22-nickel-open-sourcing/

The following blog post is referenced in the proposal: https://www.tweag.io/blog/2017-08-03-linear-typestates/ . This is maybe a different aspect than what others have been trying to explain, but I found the socket-based example enlightening. Type states allow the state of the socket (unbound, bound, listening, closed) to be expressed as a part of the type (a type annotation, if you will) so the compiler can verify that you're not reusing a closed socket, or try to send data over an unbound socket. Linear types then, allow to express in code how a type (annotation) changes over time (e.g., "bind() takes an unbound socket and returns it an a bound state") so the compiler can statically check that it's in a valid state for every operation.

Is this related to https://www.reddit.com/r/haskell/comments/7cmppn/a_great_thread_in_cas… or https://www.tweag.io/posts/2018-09-12-asterius-ffi.html ?

This company uses Nix & recently funded certain features. If you really want some help with Nix, they might have expertiese for you: http://www.tweag.io

> Besides, config languages never have the level of polish (re: tab completion and static analysis) that the main language does. They don't have to! > Why deny your users the niceties that you set up for yourself? Don't do this, of course. :) For instance, Nickel gained[1] an LSP implementation years before it hit version 1.0. When editing Nickel configs,byou get all the stuff you'd expect for a programming language: syntax highlighting, type hints, autocompletion, etc. But beyond static analysis, Nickel can also validate your configurations dynamically, according to arbitrary predicates-- and that too, integrates with your editor through the language server. Nickel ships with a command-line evaluator, REPL, pretty printer for data structures, code formatter, documentation generator, and CLI tab completions for Bash, zsh, Fish, PowerShell, and Elvish. Upstream distributes the tooling (which is, Go-style, a single executable with subcommands) as a Nix flake, a Docker image, and static executables for Linux aarch64 and x86_64 (< 30 MB, for now at least). Nickel doesn't yet have a package manager[3], but other than that it honestly has a better tooling story than many general-purpose programming languages currently in widespread use. CUE, an older (but not old) configuration language (very cool in its own right), likewise has great tooling. It ships with a command-line evaluator, a pretty printer, a formatter, package management based on Go modules (i.e., it has both a…

I don't follow it. This is the crux of the article: > and there is nothing you can do in Haskell that is not data. No logic can be written in Haskell outside of data. Is he talking about Lisp? > the often dreaded Haskell monads are really just side effects represented as data. Nah > Typeclasses are data ... Really? > ... that define certain behaviour, yes, but this behaviour, like all behaviour in Haskell, is merely a rule, a declaration, a law, which states how one piece of data is transformed into another. This is also what functions are. Yes. Functions. Not data. Also some bad code in the example: getNumOfReplacement :: String -> Maybe Char getNumOfReplacement word | first_char =='{' && third_char == '}' = Just second_char | otherwise = Nothing where first_char = head word second_char = word !! 1 third_char = word !! 2 Verbose and crashy (for inputs "", "{", "{x", ...). Here's better: getNumOfReplacement' :: String -> Maybe Char getNumOfReplacement' ('{':x:'}':_) = Just x getNumOfReplacement' _ = Nothing > The above algorithm could also be written in a pretty similar way in many other languages, even some non-functional ones. What makes Haskell different however is that this is the only way in which you can do it. Weird claim! There's loads of ways to do it. > It forces you to yes, learn some mathematical concepts, but on a more fundamental level to…

Just to add to what others said regarding question 2. After you write a derivation that you can build, you need to create a module [1]. The module defines configuration options, then takes the donation and does other things such as creating service users (if needed), services and configurations. Then you simply import it and use it. Regarding question 1 what was suggested by others is what is the current way of doing it, but I will recommend to go through Nix Pills[2]. The author starts with making a simple derivation and then starts introducing conventions. The conventions make things more complex, but they make things more flexible. I can't wait for Flakes[3], they will standardize a lot of this, removing extra cruft and allowing things to be more composable. [1] https://nixos.org/nixos/manual/index.html#sec-writing-modules [2] https://nixos.org/nixos/nix-pills/index.html [3] https://www.tweag.io/blog/2020-05-25-flakes/

I’m a bit conflicted about your point here, because they seem cherry-picked to highlight disparate aspects of Haskell that don’t make it less interesting or important. Haskell is more often than not sued as a general purpose language, not as a systems language. Its performance is actually outstanding, usually equal to Java and often close to C. This means that Haskell can be safely used in any domain were java would be fast enough. I find also somewhat arrogant to claim that there are no complex systems built in it. Just take a look at the webpage publishing this article. They mention several projects of great complexity. Here’s an example: https://www.tweag.io/posts/2017-03-13-linear-types.html They are using Haskell as a prototyping language that lets the refactor and radically change the direction of their solutions at a low development cost. Haskell clearly is helping solve complexity here. Also, where did the claim that GHC is badly documented come from, and how is it relevant to Haskell being important? Just looking at randomly selected file in the repo, it looks actually quite well documented: https://gitlab.haskell.org/ghc/ghc/blob/master/compiler/simplCore/Floa…

The Nix team (in particular grahamc with https://github.com/NixOS/ofborg) is working hard on making CI better . But it's true that the on-boarding experience and CLI could be much improved. It's quite frustrating to me, because after I've been using Nix & NixOS for about 3 years now, there's no way I'd go back to the insanity out there. I never had the amount of stability, control, and predictability on any other platform, from configuring my OS, editors, WM, shell, random ricing, or building VMs, Servers, Docker containers, production deploys, development environments, there's really not much Nix can't handle. But getting familiar with Nix took some serious effort I'm sure not many people can afford. I begin to understand how frustrated people in the Lisp and Smalltalk communities must feel while people slowly reinvent everything they've enjoyed for decades (and Nix is still in its teens). Teaching people about the value of having just a single package manager, no matter what OS or language they're using, is mostly futile because there's so much value attributed to being "mainstream" that now we have hundreds of mainstream package managers with varying degrees of sophistication, security, predictability, and ability to _not_ try and take over your system. I use it to build and develop Go, Ruby, JS, Crystal, Elm, Mint, Haskell, Perl, Bash, VimL, Elisp, Guile, and whatever else comes my way. All I have to do to get a working and isolated dev env is go into that directory…

For our last 2 clients (with both, offering a "data team as a service"): the reporting tools were written in Haskell, using Servant [1] which got started trying to do the same at Zalora to set up a customer service management tool. The motivation was that it made painful stuff much simpler and faster to write (performance was a bonus). It's also used for a bunch of tricky data manipulation problems that were harder to solve using pure SQL. Example: identify a unique customer based on attributes (email, phone, address, etc.) one or more of which can change when a customer tries to create a "new" profile to grab the $10 new customer signup voucher. This ran recursively on the entire customer dataset (12 countries, some customers had created as many as 250 profiles) in about 4 seconds on a m3.medium instance. We don't/didn't write blog posts or tweets about it though. I think the set of people who write blog posts (i.e. both have free time to do so, the talent, and the motivation) is relatively small, and the number of blog posts and other visible stuff being published is proportional to the size of the community, so you'll see many more Node.js/RoR posts than you will on using set theory to reduce a 5,000-table flawed data model systematically or the kind of everyday stuff Haskellers are doing everywhere. Also, personally at least, I don't feel like I know enough to have much to offer by writing a blog, so I stick to making these semi-anonymous HN comments. Interviewing…

Tweag has many interesting entries with good technical depth: https://www.tweag.io/blog

See also the Haskell library monad-bayes https://monad-bayes.netlify.app/tutorials/ https://www.tweag.io/blog/2019-09-20-monad-bayes-1/

The Choice Gradient Sampling algorithm shown in the paper could be used to integrate/exploit feedback (coverage, etc.). Hypothesis can use a fuzzer as its input source, but it's marked as experimental. It doesn't seem like "proper" fuzzers are used very often with Haskell (e.g. there's an example at https://www.tweag.io/blog/2023-06-15-ghc-libfuzzer/ ), though many people seem to have hooked up the HPC code coverage tool to property checkers, e.g. https://www.cse.iitk.ac.in/users/karkare/MTP/2011-12/subhash2012quicke… https://www.doc.ic.ac.uk/~tora/irulan/issta.pdf https://github.com/shapr/kudzu

LLVM_USE_SPLIT_DWARF may help with this, some recent measurements: https://www.tweag.io/blog/2023-11-23-debug-fission/#an-example-at-scal…

Another note, NixOS containers with systemd-nspawn cpmpatibility: https://nixos.wiki/wiki/NixOS_Containers There is a good blog post here about how these work: https://www.tweag.io/blog/2020-07-31-nixos-flakes/ And of course, it is just a tarball, so it works on every systemd based distro.

Since this uses cosmopolitan and the build script already downloads portable binaries from https://cosmo.zip has there any thought been given to wrap other portable binaries in scrapscript / download them? Small, pure, functional, content-addressable and network-first sounds a lot like a mini Nix+ca-derivations [1] [1] https://www.tweag.io/blog/2021-12-02-nix-cas-4/

The contact page https://www.tweag.io/contact has a contact form and an email address.

This is offtopic, sorry, but does anyone have a contact at this site? They should change their blog posts so they don't include this: rel="canonical" href="https://tweag.io/blog" HN's software uses canonical URLs when it finds them, so most of the submissions at https://news.ycombinator.com/from?site=tweag.io are getting replaced with https://tweag.io/blog. I've fixed this one manually.

Link should be https://www.tweag.io/blog/2023-11-16-announcing-organist/

Correct URL: https://www.tweag.io/blog/2023-10-19-json-schema-to-nickel/

Aw nice; I think haskell-wasm uses FFI too https://www.tweag.io/blog/2018-09-12-asterius-ffi/ super cool to see an nginx version of it!

Reminds me of this more abstract perspective: you can draw an analogy between working the filesystem, and programming languages working with memory. https://www.tweag.io/blog/2022-07-14-taming-unix-with-nix/

> I really wished there was a GC-ed but strictly-evaluating Haskell. To answer this question literally, there are PureScript (transpile to JavaScript), Idris (dependently typed), OCaml, and Standard ML (as discarded1023 pointed out). But I think the wish for a strictly-evaluating Haskell is sometimes in fact a wish for a Haskell with more predictable performance (especially in memory usage). If so, the Linear Type/Arrow of recent GHC may fit the bill [1]. My wish is for Haskell to have a better runtime [1] with optimal reductions (without GC) [2], which could have Rust-like performance without lifetimes. [1]: https://www.tweag.io/blog/2017-03-13-linear-types/ [2]: https://discourse.haskell.org/t/high-order-virtual-machine-hvm-an-opti… [3]: https://github.com/HigherOrderCO/HVM

Another advantage of the gomod2nix approach concerns long term build reproducibility [1]. Having each source dependency in its own derivation would make easier source code mirroring. Since Nix knows the hash of each sources, we could imagine to teach Nix how to get the source code from some other places than GitHub. For instance, Nix could use the hash of the dependency to query Software Heritage [2]. Note there is also an opened issue to restrict Nix fixed output derivation which would make impossible the buildGoModule "hack". [1] https://github.com/NixOS/nixpkgs/issues/84826 [2] https://www.tweag.io/blog/2020-06-18-software-heritage/