Merge pull request #18 from valentinewallace/update-overview

Update overview

Author: valentinewallace

docs/build_node.md

@@ -9,7 +9,9 @@ This document covers everything you need to make a node using LDK.
 
 * [Setup](#setup) covers everything you need to do to set up LDK on startup.
 * [Running LDK](#running-ldk) covers everything you need to do while LDK is running to keep it operational.
-* [Using LDK](#using-ldk) covers most lightning operations you'll want to use, such as opening a channel. Sending and receiving payments are supported but not yet a part of this guide.
+* [Using LDK](#using-ldk) covers most lightning operations you'll want to use,
+  such as opening a channel. Sending and receiving payments are supported but
+  not yet a part of this guide.
 
 Note that LDK does not assume that safe shutdown is available, so there is no 
 shutdown checklist.
@@ -25,7 +27,9 @@ shutdown checklist.
 final fee_estimator = FeeEstimator.new_impl((confirmation_target -> 253));
 ```
 
-**Implementation notes:** Rather than using static fees, you'll want to fill in the lambda with fetching up-to-date fees from a source like bitcoin core or your own API endpoint.
+**Implementation notes:** Rather than using static fees, you'll want to fill in
+the lambda with fetching up-to-date fees from a source like bitcoin core or your
+own API endpoint.
 
 **Dependencies:** *none*
 

docs/getting_started.md

@@ -12,7 +12,7 @@ can be supplied by the user or by one of LDK's sample modules.
 
 ## How To Use These Docs
 "Getting Started" + "Overview" provide an introduction to the architecture and
-design philosophy of LDK. 
+design philosophy of LDK.
 
 "Build a Node: Checklist" walks through how to specifically integrate LDK into
 your application, as well as documentation for what features are currently available

docs/open_channel.md

@@ -4,7 +4,7 @@ title: "Opening a Channel with LDK"
 ---
 
 ## Prerequisites
-See "Build a Node: Checklist" for preparing LDK to open a channel. This guide 
+See [Building a Node with LDK](build_node.md) for preparing LDK to open a channel. This guide 
 is a follow-up.
 
 ## Overview

docs/overview.md

@@ -4,7 +4,118 @@ title: Overview
 slug: /
 ---
 
+import useBaseUrl from '@docusaurus/useBaseUrl';
+
 LDK is a flexible lightning implementation with supporting batteries (or modules).
-The core lightning implementation is Rust-Lightning.
 
-See the Rust-Lightning `README` for more overview: https://github.com/rust-bitcoin/rust-lightning
+## To jump into integrating LDK with your application, click [here](build_node.md)
+
+## Introduction
+LDK/Rust-Lightning is a generic library which allows you to build a lightning
+node without needing to worry about getting all of the lightning state machine,
+routing, and on-chain punishment code (and other chain interactions) exactly
+correct. Note that Rust-Lightning isn't, in itself, a node. There are various
+working/in progress demos which could be used as a node today, but if you "just"
+want a generic lightning node, you're almost certainly better off with
+`c-lightning`/`lnd` - if, on the other hand, you want to integrate lightning
+with custom features such as your own chain sync, your own key management, your
+own data storage/backup logic, etc., LDK is likely your only option.
+
+We are currently working on a demo node which fetches blockchain data and
+on-chain funds via Bitcoin Core RPC/REST. The individual pieces of that demo
+are/will be composable, so you can pick the off-the-shelf parts you want and
+replace the rest.
+
+## LDK Batteries
+While LDK provides all the core lightning state machine logic, other
+batteries/modules are needed to run a node. LDK interacts with these modules
+through generic interfaces, meaning the user can choose the implementation that
+best suits their needs. LDK provides sample implementations for many of these
+batteries, which are enumerated below.
+
+* On-disk storage 
+  * You can store the channel state any way you want - whether Google
+  Drive/iCloud, a local disk, any key-value store/database/a remote server, or
+  any combination of them - we provide a clean API that provides objects which
+  can be serialized into simple binary blobs, and stored in any way you wish.
+  * [**Sample module in Rust**](https://github.com/rust-bitcoin/rust-lightning/tree/main/lightning-persister)
+* Blockchain data
+  * We provide a simple `block_connected`/`block_disconnected`
+  API which you provide block headers and transaction information to. We also
+  provide an API for getting information about transactions we wish to be
+  informed of, which is compatible with Electrum server requests/neutrino
+  filtering/etc.
+  * [**WIP sample module in Rust**](https://github.com/rust-bitcoin/rust-lightning/pull/791)
+* On-chain funds wallet/UTXO management
+  * Rust-Lightning/LDK owns on-chain funds as long as they are claimable as
+  a part of a lightning output which can be contested - once a channel is closed
+  and all on-chain outputs are spendable only by the user, we provide users
+  notifications that a UTXO is "theirs" again and it is up to them to spend it
+  as they wish. Additionally, channel funding is accomplished with a generic API
+  which notifies users of the output which needs to appear on-chain, which they
+  can then create a transaction for. Once a transaction is created, we handle
+  the rest. This is a large part of our API's goals - making it easier to
+  integrate lightning into existing on-chain wallets which have their own
+  on-chain logic - without needing to move funds in and out of a separate
+  lightning wallet with on-chain transactions and a separate private key system.
+  * LDK does not currently provide a sample wallet module, but its sample node
+    implementation uses Bitcoin Core's wallet for UTXO management e.g. [here](https://github.com/TheBlueMatt/rust-lightning-bitcoinrpc/blob/139a653eeba313284c6d9d2eb2776d30dbb0ca3d/src/main.rs#L219)
+* Networking 
+  * To enable a user to run a full lightning node on an embedded
+  machine, we don't specify exactly how to connect to another node at all! We
+  provide a default implementation which uses TCP sockets, but, e.g., if you
+  wanted to run your full lightning node on a hardware wallet, you could, by
+  piping the lightning network messages over USB/serial and then sending them in
+  a TCP socket from another machine.
+  * [**Sample module in Rust**](https://github.com/rust-bitcoin/rust-lightning/tree/main/lightning-net-tokio)
+  * [**Sample module in Java**](https://github.com/lightningdevkit/ldk-garbagecollected/tree/main/src/main/java/org/ldk/batteries)
+* Private keys
+  * LDK has "default implementations", but users can choose to provide private
+  keys to RL/LDK in any way they wish following a simple API. We even support a
+  generic API for signing transactions, allowing users to run RL/LDK without any
+  private keys in memory and/or putting private keys only on hardware wallets.
+  * [LDK's `KeyManager` docs](https://docs.rs/lightning/0.0.12/lightning/chain/keysinterface/struct.KeysManager.html). While LDK's default implementation is currently within Rust-Lightning, it still is considered a "sample module."
+* Transaction filtering
+  * Clients running a light client may wish to filter for transactions on a separate server, in which case LDK will tell them about transactions to filter for. More information is available in the [Blockchain Data guide](blockdata.md).
+* Fee estimation
+  * LDK's sample node implementation uses Bitcoin Core's fee estimation API [here](https://github.com/TheBlueMatt/rust-lightning-bitcoinrpc/blob/139a653eeba313284c6d9d2eb2776d30dbb0ca3d/src/chain_monitor.rs#L31).
+* Transaction broadcasting
+  * LDK's sample node implementation uses Bitcoin Core's transaction broadcasting API [here](https://github.com/TheBlueMatt/rust-lightning-bitcoinrpc/blob/139a653eeba313284c6d9d2eb2776d30dbb0ca3d/src/chain_monitor.rs#L132).
+* Random number generation
+  * Because Rust-Lightning aims to make no system calls, it is restricted from generating its own randomness.
+  * LDK's sample node implementation uses Rust's `rand` crate [here](https://github.com/TheBlueMatt/rust-lightning-bitcoinrpc/blob/139a653eeba313284c6d9d2eb2776d30dbb0ca3d/src/main.rs#L512) and elsewhere.
+
+
+## LDK Architecture
+![LDK Architecture](assets/ldk-architecture.svg)
+
+LDK's core components are shown in the center box labeled `lightning`. Boxes
+with dotted borders are LDK's batteries — these must be configured with either
+off-the-shelf or custom implementations that you provide.
+ 
+EventHandler in the diagram is not so much a necessary LDK battery, but instead
+refers to the fact that LDK generates events that the user should handle (e.g.
+the `PaymentReceived` event).
+
+## References
+
+### [Rust Documentation](https://docs.rs/lightning)
+
+These provide the most searchable and comprehensive documentation on LDK.
+If you're using Java and want more information on any method/struct/etc., searching
+the Rust docs for the Rust version of that struct/method is your best bet.
+
+### [Rust Sample Node](https://github.com/TheBlueMatt/rust-lightning-bitcoinrpc)
+
+While this node is a little outdated, it's still a very useful reference for how to construct
+a lightning node using LDK.
+
+### [Swift LDK Documentation](https://github.com/arik-so/SwiftLightning/tree/master/Documentation)
+
+These docs are mainly geared towards how Swift could call LDK C bindings directly, but still may
+provide a useful overview of Rust Lightning in the context of language bindings.
+
+### [LDK Architecture](https://docs.google.com/drawings/d/1Ql-q5gyrPnJhi7z_D39jayG0HEEVh6UEY1eULXb03Eg/edit?usp=sharing)
+
+Gives a high-level organization of LDK and how the pieces fit together. Variations of this diagram
+are used throughout the site. This is the primary source and is still a work in progress.

docs/use_cases.md

@@ -3,4 +3,41 @@ id: use_cases
 title: Use Cases for LDK
 ---
 
-LDK is designed to provide incredible flexibility in integrating a lightning node into your application. It is not designed to be a stand-alone application and those seeking a full lightning node to run may wish to look elsewhere. LDK focuses on ensuring tight integration with existing on-chain wallets is easy, allowing reuse of existing blockchain download, key management, UTXO management, and on-disk/cloud storage. See the Getting Started page for more details on the interfaces LDK provides for integration.
+The standard lightning use case is running a standalone node on one's laptop.
+Here's some other use cases that LDK supports.
+
+## Mobile Devices
+Mobile devices with lightning have unique requirements often not well served by
+today's lightning ecosystem. Not only do they need to operate with minimal
+footprint, they also have intermittent data access and cannot shutdown safely.
+More importantly, many existing wallets already have business logic to handle
+blockchain data, keys, and storage, and do not wish to duplicate much of that
+logic to integrate lightning (at worst fetching the blockchain twice). LDK
+offers a flexible API to allow users to integrate lightning with their own keys,
+blockchain data, and storage. To allow full flexibility in footprint, the API
+supports routing data being fetched via the Lightning P2P protocol, an external
+service, or routes can be calculated off-device. It also provides cross-platform
+compatibility for free, allowing synchronization of lightning state across
+devices and, as long as there is protection from simultaneous-updates, users to
+access their wallet on any device. See the [Overview](overview.md) page for more
+details on the interfaces LDK provides for integration.
+
+## HSMs (Hardware Security Modules)
+
+LDK Supports various HSM configurations. In conjunction with the [Lightning
+Signer project](https://github.com/lightning-signer/) , an external HSM can be
+used to verify most protocol details about states before signing, ensuring host
+compromise cannot steal funds by broadcasting revoked states. For nodes seeking
+a higher level of assurance, the entire Rust-Lightning channel state machine can
+be run on an offline device, communicating with the outside world via a proxy
+host which maintains TCP connections with peers. Such a configuration ensures
+all details of the lightning protocol are enforced without consideration of host
+compromise.
+
+## Production Lightning Nodes
+Many large Bitcoin transactors have large amounts of custom-built infrastructure
+for interacting with the Bitcoin blockchain. Such tight integration with
+business logic may be difficult with existing lightning implementations focusing
+on standalone operation. For such transactors, LDK offers the possibility of
+integrating a library in their native runtime, storing and handling lightning
+data and events in the same way they do blockchain events.

docusaurus.config.js

@@ -9,6 +9,7 @@ module.exports = {
   organizationName: 'lightningdevkit', // Usually your GitHub org/user name.
   projectName: 'lightningdevkit.org', // Usually your repo name.
   themeConfig: {
+    sidebarCollapsible: false,
     navbar: {
       title: 'LDK',
       logo: {

sidebars.js

@@ -1,6 +1,6 @@
 module.exports = {
   someSidebar: {
-      'Lightning Development Kit': ['overview', 'getting_started', 'use_cases', 'references'],
+      'Lightning Development Kit': ['overview', 'use_cases'],
       Guides: ['build_node', 'open_channel', 'key_mgmt', 'blockdata'],
   },
 };