Armstrong distributed systems

How do we build reliable, scalable and maintainable computer systems?

This repository contains notes on how, I think, we can improve on the state of development, documentation, testing, deployment, observability, debugging, and upgrading of distributed systems. Most of the ideas are stolen from others, many from Erlang and Joe Armstrong. Over time I hope to turn this into a more coherent text, for now think of it a crude blog or some basic scaffolding for me to hang my thoughts on.

If any of this interests you as well, please do get in touch -- one of the reasons I'm writing this down is to find people to collaborate with.

Development

• Erlang's not about lightweight processes and message passing...

• Implementing behaviours

• Modular state machines

• Implementing gen_event using the LMAX disruptor

  • https://github.com/stevana/pipelined-state-machines
  • Shard on: "people, stuff or deals", says Martin Thompson.

• State machines with of async I/O

• Persisted/mmapped lock-free concurrent data structures

  • Working with binary data
  • bytebuffer
  • journal
  • hashmap
  • arena allocator
  • buffered actions
  • Efficient Tree-Traversals: Reconciling Parallelism and Dense

• On the role of practice in programming

Documentation

• Specification language

  • Interfaces
  • Messages
  • Compression
  • Protocols
  • Usage model or operational profile

• Joe's idea of "the bigger picture" in particular the "research" part, not just the end result (code) but how you got to the result.

Testing

• Simulation testing using state machines

• Usage model or operational profile

• Performance testing

  • https://ocw.mit.edu/courses/6-172-performance-engineering-of-software-systems-fall-2018/resources/lecture-10-measurement-and-timing/

• Load/soak testing

Deployment

• Deploying and restarting state machines using supervisor trees

• Elastically scalable thread pools

Observability

• Visualising datastructures over time using SVG

Debugging

• Domains-specific debuggers

Upgrading

• Hot-code swapping à la Erlang with Arrow-based state machines

• smarrow-lang an experimental programming language where programs are state machines expressed in arrow notation to allow easy hot-code swapping

• Version everything