title: "[dapps made easy] chapter 1: replication" data: 2019-08-01 draft: true featured: true tags: technology dapps-made-easy

there are a lot of very interesting peer-to-peer protocols out there these days. "hashing the world" hasn't quite played out how some people might have hoped, but these are still very exciting times. we will start by framing the discussion around a system archtecture that will be familiar to web developers: a ui client that talks to an application server which talks to a database. the application server will often have the ability to defer work to another time and/or send messages to external services. often a local or external object storage system will be used to persist binary blobs. eventually you will need to have multiple replicas of this system and data.

lets start with database replication. we want to make sure this part is easy to do first, because it is extremely difficult to add later. first we will compare and contrast the replication strategies of secure scuttlebutt and bit torrent.

when it comes to modeling data, ssb is very much in the event sourcing camp. so you end up needing a lot of special constructs to manage and query these non-compactible replicated logs. out-of-order gossip (ooo) helps, but you are still stuck with the message stream as the fundamental primitive. contrast that with a DHT where everything lives in a key/value store. if you look at any database architecture you’ll find that a key/value store is always the basic storage unit. starting here doesn’t prevent you from creating other abstractions that are a better fit for your application.

lets try sketching out a basic feed construct using a key-value and some basic crypto primitives:

a modern DHT will use a SHA 2 hash to address values. normally this means that everything in the DHT is immutable. however, we can use some simple cryptography to give us enough mutability to do something useful like posting images of your waifu.

start with an asymmetric key pair. i would recommend an ed25519 key for reasons i can get into later. take the public key and take the sha256


sharing a tiny key like this on a DHT isn’t very useful by itself, but suppose your client knows how to check signatures. an ed25519 key is small enough that you can include it on the message body along with a signature asserting that you have a write capability for this key. the key can be validated by checking the hash on this key. if you have too much data to store in one DHT record you can chunk it and point the same signed object at the merkle DAG that represents your data. now we can represent arbitrary mutable data on an immutable key/value store!

using this construct, we can build ssb-style feeds. the goal isn't to get away from this construct specifically. append-only logs are very useful for a lot of applications. a key/value store is a very flexible primitive that can be used to create many useful abstractions. search is a fairly complex problem, for now we will consider it a black box.

the key/value store is a sort of "narrow waist" which enables efficient replication. in the future we may be able to drastically simplify the replication stack using a future internet protocol called [Named Data Networking].