Efficient Reliable Database Migration: A Legacy Innovation Story • Dave Thomas • GOTO 2024

This presentation was recorded at GOTO Chicago 2024. #GOTOcon #GOTOchgo https://gotochgo.com Dave Thomas - Founder, Keynote Speaker & Software Industry Expert RESOURCES http://www.davethomas.net https://twitter.com/daveathomas https://linkedin.com/in/davidathomas ABSTRACT Data migrations often fail due to underestimated complexity. This talk explores a critical, large-scale migration at a major ERP company that demanded high performance, reliability, and aggressive timelines. We'll describe novel techniques including efficient bulk data transfer, table-driven approaches, functional prototyping, and a unique application of IV&V for robust testing. Learn how algorithms developed in distributed systems research solved real-world problems in legacy system modernization. Discover strategies to implement reliable migration techniques and meet tight deadlines. This talk offers valuable insights for data engineers, architects, and development leaders facing migrations of any scale. [...] TIMECODES 00:00 Intro 01:53 Reality of data migration 02:21 Case study: Critical legacy problem 05:38 Reporting problem 06:42 Reporting solution 07:45 Near-term solution 08:22 Long-term migration challenge 09:37 Conventional migration approach 10:52 Innovative solution 12:15 Logical & physical model 13:34 Essence of approach 15:37 Binary is better 16:09 Batch is better 16:43 Efficient serialization 17:49 Conservative implementation strategy to reduce risk 20:28 Verification & validation approach 29:53 Readers & writers? 32:53 Recap 37:39 Outro Download slides and read the full abstract here: https://gotochgo.com/2024/sessions/3362 RECOMMENDED BOOKS Kevlin Henney • 97 Things Every Programmer Should Know • https://amzn.to/2Yahf9U Henney & Monson-Haefel • 97 Things Every Software Architect Should Know • https://amzn.to/3pZuHsQ Matthew Skelton & Manuel Pais • Team Topologies • http://amzn.to/3sVLyLQ Michael Jackson • Software Requirements and Specifications • https://amzn.to/3ql2T14 Geoffrey West • Scale • http://amzn.to/3eKMbpc Donald G. Reinertsen • The Principles of Product Development Flow • https://amzn.to/3hJ2Ye https://bsky.app/profile/gotocon.com https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #DatabaseMigration #DataMigration #Legacy #TechnicalLeadership #DeveloperProductivity #Productivity #SoftwareEngineering #Programming #Teams #DaveThomas CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1