Async I/O in Depth: Futures and Implementing an Async Runtime

In this presentation I discuss Futures in Rust and implementing our very own async/await runtime. We start out by reviewing our current implementations, discussing state machines, generators, threads, forking architectures and various approaches we have taken thus far. We take a look at how Futures and async/await code is compiled by the Rust compiler. Looking at HIR and MIR compiled rust code, we can see how generators play a part in managing the state machine of async await code. We understand that generators are converted back into a Future for the async runtime to poll. We take a look at the Reactor pattern, review the waker/context syntax and the Future trait itself. By building on these concepts we are then ready to create our very own async runtime by building an executor event loop, the reactor for polling, waker functions to reschedule tasks, and async networking with the TcpListener and TcpStream. We build on the previous examples of the HTTP Web Server by implementing the Future trait for read and write operations. Finally, we are able to run our HTTP Web Server with fully async/await code with non-blocking I/O polling apis. ↠ References: Rust Compiler Expressions: https://github.com/rust-lang/rust/blob/master/compiler/rustc_ast_lowering/src/expr.rs Future: https://doc.rust-lang.org/std/future/trait.Future.html make_async_expr: https://github.com/rust-lang/rust/blob/master/compiler/rustc_ast_lowering/src/expr.rs#L548 lower_expr_await: https://github.com/rust-lang/rust/blob/master/compiler/rustc_ast_lowering/src/expr.rs#L647 Async Await Book: https://rust-lang.github.io/async-book/01_getting_started/01_chapter.html Waker Functions: https://github.com/smol-rs/waker-fn Generators in Rust: https://cfsamson.github.io/books-futures-explained/4_generators_async_await.html Reactor Pattern: https://en.wikipedia.org/wiki/Reactor_pattern epoll: https://man7.org/linux/man-pages/man7/epoll.7.html epoll/kqueue/iocp: https://cfsamson.github.io/book-exploring-async-basics/6_epoll_kqueue_iocp.html poll: https://man7.org/linux/man-pages/man2/poll.2.html as_raw_fd: https://doc.rust-lang.org/std/os/unix/io/trait.AsRawFd.html#tymethod.as_raw_fd Poller: https://docs.rs/polling/latest/polling/struct.Poller.html RwLock: https://doc.rust-lang.org/std/sync/struct.RwLock.html Arc: https://doc.rust-lang.org/std/sync/struct.Arc.html Mutex: https://doc.rust-lang.org/std/sync/struct.Mutex.html Box: https://doc.rust-lang.org/std/boxed/struct.Box.html dyn Trait keyword: https://doc.rust-lang.org/std/keyword.dyn.html std::process: https://doc.rust-lang.org/std/process/index.html fs::File: https://doc.rust-lang.org/std/fs/struct.File.html BufWriter: https://doc.rust-lang.org/std/io/struct.BufWriter.html BufReader: https://doc.rust-lang.org/std/io/struct.BufReader.html HTTP Protocol: https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview Radix Tree: https://en.wikipedia.org/wiki/Radix_tree Interrupts: https://en.wikipedia.org/wiki/Interrupt Virtual Memory: https://en.wikipedia.org/wiki/User_space_and_kernel_space JoinHandle: https://doc.rust-lang.org/std/thread/struct.JoinHandle.html thread::spawn: https://doc.rust-lang.org/std/thread/fn.spawn.html std::net::TcpListener: https://doc.rust-lang.org/std/net/struct.TcpListener.html std::net::SocketAddr: https://doc.rust-lang.org/std/net/enum.SocketAddr.html std::net::TcpStream: https://doc.rust-lang.org/std/net/struct.TcpStream.html std::io::Stdin: https://doc.rust-lang.org/std/io/struct.Stdin.html std::io::Stdout: https://doc.rust-lang.org/stable/std/io/fn.stdout.html socket: https://man7.org/linux/man-pages/man2/socket.2.html setsocketopt: https://linux.die.net/man/2/setsockopt bind: https://man7.org/linux/man-pages/man2/bind.2.html listen: https://man7.org/linux/man-pages/man2/listen.2.html connect: https://man7.org/linux/man-pages/man2/connect.2.html write: https://man7.org/linux/man-pages/man2/write.2.html read: https://man7.org/linux/man-pages/man2/read.2.html libc crate: https://github.com/rust-lang/libc ↠ Dotfiles: https://github.com/nyxtom/dotfiles ↠ Twitter: @nyxtom https://twitter.com/nyxtom ↠ Channel: https://www.youtube.com/c/nyxtom Cheers! 🍻 🦀 #rust #rustlang #rustprogramming #exercises #programming