Code Documentation with Omer Rosenbaum and Tim Post

172: Transformers and Large Language Models

Intro topic: Is WFH actually WFC?

News/Links:

• Falsehoods Junior Developers Believe about Becoming Senior
  • https://vadimkravcenko.com/shorts/falsehoods-junior-developers-believe-about-becoming-senior/
• Pure Pursuit
  • Tutorial with python code: https://wiki.purduesigbots.com/software/control-algorithms/basic-pure-pursuit 
  • Video example: https://www.youtube.com/watch?v=qYR7mmcwT2w 
• PID without a PHD
  • https://www.wescottdesign.com/articles/pid/pidWithoutAPhd.pdf
• Google releases Gemma
  • https://blog.google/technology/developers/gemma-open-models/
    
    

Book of the Show

• Patrick: The Eye of the World by Robert Jordan (Wheel of Time)
  • https://amzn.to/3uEhg6v
• Jason: How to Make a Video Game All By Yourself
  • https://amzn.to/3UZtP7b

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick: Stadia Controller Wifi to Bluetooth Unlock
  • https://stadia.google.com/controller/index_en_US.html
• Jason: FUSE and SSHFS
  • https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh
    
    

Topic: Transformers and Large Language Models

• How neural networks store information
  • Latent variables
• Transformers
  • Encoders & Decoders
• Attention Layers
  • History
    • RNN
      • Vanishing Gradient Problem
    • LSTM
      • Short term (gradient explodes), Long term (gradient vanishes)
  • Differentiable algebra
  • Key-Query-Value
  • Self Attention
• Self-Supervised Learning & Forward Models
• Human Feedback
  • Reinforcement Learning from Human Feedback
  • Direct Policy Optimization (Pairwise Ranking)

Intro topic: Monitor setups

News/Links:

• BlueScuti, Willis, beats Tetris
  • https://www.youtube.com/watch?v=GuJ5UuknsHU
• PalWorld accused of being an AI Product
  • https://www.forbes.com/sites/paultassi/2024/01/22/palworld-accused-of-using-genai-with-no-evidence-so-far/?sh=26a9651b4239
• 4 Billion if-statements to determine if a number is even or odd
  • https://andreasjhkarlsson.github.io/jekyll/update/2023/12/27/4-billion-if-statements.html
• Seamless M4T
  • https://ai.meta.com/blog/seamless-m4t/
    
    

Book of the Show

• Patrick:
  • Foundation by Isaac Asimov
    • https://amzn.to/3SrmgnP
• Jason: 
  • Propaganda by Edward Bernays
    • https://amzn.to/47JUCXJ

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick: 
  • The Room Game
    • https://www.fireproofgames.com/games/the-room
• Jason:
  • Incredibuild
    • https://www.incredibuild.com/
      
      

Topic: Compilers and Interpreters (Request by Jessica W.)

• Machine Code
  • Architecture Specific
• Assembly
  • Single vs Two Pass Compiler
• High level Languages
  • Intermediate Representation
  • JVM ByteCode vs Machine Code for portability
• Scripting/Interpreters
• JIT
• Profile Guided Optimization
• Resources
  • https://www.craftinginterpreters.com/
  • https://nandgame.com/
  • Turing Complete

Predictions:
Jason 

• VR for Work
• Lowering AI training cost/ improved efficiency
• RISC-V takeoff

Patrick

• Ai claim of AGI
• Ai peer reviewer
• Ai Video Generator
• More space vehicles reaching orbit

Early career, finding role at FAANG, liaising vs shipping code. Startup?

3 part. 1. How and when current hype for AI will end? 2. Shape of the show 3. Upcoming in tech

What are essential programmer knowledge items?

CS Student, how to organize life and goals? What purpose life should serve?

What kind of programmer were you in college?

Happy Holidays!

Intro topic: Testing your car battery

News/Links:

• Tech Layoffs still going on
  • https://www.sfchronicle.com/tech/article/google-layoffs-california-companies-18465600.php 
• Real-time dreamy Cloudscapes with Volumetric Raymarching
  • https://blog.maximeheckel.com/posts/real-time-cloudscapes-with-volumetric-raymarching/
• Robot Rascals
  • https://en.wikipedia.org/wiki/Robot_Rascals 
• Meta Quest 3 
  • https://www.theverge.com/23906313/meta-quest-3-review-vr-mixed-reality-headset
    
    

Book of the Show

• Patrick:
  • HyperLogLog Paper
    • https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf 
• Jason: 
  • Eureka! NVIDIA Research Breakthrough Puts New Spin on Robot Learning 
    • https://blogs.nvidia.com/blog/2023/10/20/eureka-robotics-research/ 

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick: 
  • Techtonica: https://store.steampowered.com/app/1457320/Techtonica/ 
• Jason:
  • ESP32 development board: https://amzn.to/3Qpmb20 
  • WEMOS

Topic: HyperLogLog

• Motivation
  • Cardinality Counting
• LinearCounting
  • Hash + expectation of collision based on how full
  • Bloom Filter
• LogLog
  • Use first N bits as bucket
  • Use max sequential 0s in each bucket
  • Average
• HyperLogLog
  • Handle empty buckets
  • Use correction factor like linear counting for low counts (number of empty buckets) and high counts
• Distributing
  • Transfer bucket counts

Intro topic: What are expectations on developers that have nothing to do with programming

News/Links:

• Pushing for a lower dev estimate is like negotiating weather with a Meteorologist
  • https://smartguess.is/blog/your-estimate-is-less-than-that/
• Announcing python in excel
  • https://techcommunity.microsoft.com/t5/excel-blog/announcing-python-in-excel-combining-the-power-of-python-and-the/ba-p/3893439
• 7 Habits of Highly Effective Software Engineers
  • https://makingsmallercircles.com/articles/7-habits-of-highly-effective-software-engineers/
• Raspberry pi 5 begins shipping
  • https://www.phoronix.com/news/Raspberry-Pi-5-Shipping
    
    

Book of the Show

• Patrick:
  • Harry Potter and the Sorcerer's Stone Illustrated Edition
    • https://amzn.to/3St3L35
• Jason: 
  • The Pete and Sebastian Show
    • https://podcasts.apple.com/us/podcast/the-pete-and-sebastian-show/id570256898

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick: 
  • Obsidian
    • obsidian.md
• Jason:
  • Ink by Inkle
    • https://github.com/inkle/ink
      
      

Topic: Godot

• What is a game engine?
  • Graphics, animation, particle effects
  • Tilemaps
  • Entity component systems
  • Physics
  • Sound
  • I/O
  • Input handling (touchscreen, joystick)
  • GDScript
• Why use a game engine?
  • Portability
  • Allow for testing individual components of the game
  • Libraries for ads, in-app-purchases
• Godot
  • Open source
  • Focused on 2-D but now many 3-D features
  • Great development environment
• AI Hero
  • Started in phaser (development rut)
  • Moved to Godot, rapid prototyping

Intro topic: Jogging Metrics

News/Links:

• Unholy LLM
  • https://huggingface.co/Undi95/Unholy-v1-12L-13B 
• The reverse red herring  
  • https://www.blameless.com/blog/the-reverse-red-herring
• The "ens–tification" of TikTok
  • https://www.wired.com/story/tiktok-platforms-cory-doctorow/ 
• Response letter to Godot is not the new Unity
  • https://sampruden.github.io/posts/godot-is-not-the-new-unity/
  • https://gist.github.com/reduz/cb05fe96079e46785f08a79ec3b0ef21
    
    

Book of the Show

• Patrick:
  • Math Games with Bad Drawings by Ben Orlin
  • https://amzn.to/48qlg9A
• Jason: 
  • The Invisible Hook: The Hidden Economics of Pirates
  • https://amzn.to/3LChBff

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick: 
  • Factorio (Desktop Game)
• Jason:
  • AI Hero (iOS and Android)

Topic: Desktop user interfaces

• What is a user interface?
• Web and Mobile UI toolkits
• Desktop options
  • Qt
    • Cross platform, custom UI elements
    • Qt creator, code generator
  • WxWidgets
    • Cross platform, uses native UI elements 
    • WxFormDesigner, code generation
    • Can look different on different operating systems
  • Electron
    • Local nodejs webserver
    • Html/JavaScript technology
    • Requires interprocess communication to use other languages  
  • Jupyter notebooks
    • Mathematica-like notebook 
    • Not for distribution 
  • Streamlit
    • Python to web compiler
  • Game Engines
    • Unity, Godot, Unreal
• Tips for building desktop UI
  • UI is slow (startup time, interaction time)
  • Separate the UI from the engine & business logic

- pganalyze: https://pganalyze.com/

- Weekly series "5mins of Postgres": https://www.youtube.com/channel/UCDV_1Dz2Ixgl1nT_3DUZVFw

- How Postgres chooses which index to use: https://pganalyze.com/blog/how-postgres-chooses-index

- CMU databases courses: https://db.cs.cmu.edu/courses/

- Postgres community: https://www.postgresql.org/community/

As well as social links:

- Mastodon: https://hachyderm.io/@lukas

- Twitter/X: @pganalyze, @LukasFittl

- GitHub: @pganalyze, @lfittl

- LinkedIn: https://www.linkedin.com/in/lfittl/ , https://www.linkedin.com/company/pganalyze/

Intro topic: Revisiting the power of Spreadsheets

News/Links:

• LK-99 Isn’t a Superconductor
  • https://www.nature.com/articles/d41586-023-02585-7
• Normalizing Flows
  • https://pyro.ai/examples/normalizing_flows_i.html
• How is llama.cpp possible?
  • https://finbarr.ca/how-is-llama-cpp-possible/
• Chat with open source large language models
  • https://chat.lmsys.org/
    
    

Book of the Show

• Patrick: Math with Bad Drawings by Ben Orlin
  • https://amzn.to/44dsgDz
• Jason: 
  • Overboard! https://play.google.com/store/apps/details?id=com.InkleLtd.Overboard Nhl=en_US&gl=US 

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick: 
  • ffmprovisr https://amiaopensource.github.io/ffmprovisr/
• Jason:
  • Pandas read_ods() read_excel()

Topic: Differential Equations

• Why should programmers learn about DiffEq
  • Law of Large Numbers
• What are differential equations?
  • When you know the rate of change
  • EigenVectors & EigenValues
  • What is Jacobian What is Jacobian? | The right way of thinking derivatives and integrals
    
    
• Special cases
  • Partial Differential Equations
  • Ordinary Differential Equations
• Why solvers are important
  • Numerical Stability at larger step sizes
  • Example: https://medium.com/@pukumarathe/eulers-method-and-runge-kutta-4th-order-method-in-python-b4a0068a8ebe 
• Fun Examples
  • Predator-Prey relationships in scipy
    • https://scientific-python.readthedocs.io/en/latest/notebooks_rst/3_Ordinary_Differential_Equations/02_Examples/Lotka_Volterra_model.html 
  • Physics Engines for games
    • https://youtu.be/52n2qKgwW_Q 
  • PageRank
    • https://arxiv.org/pdf/2001.08973.pdf 

Things to consider when choosing a database

• Speed & Latency
• Consistency, ACID Compliance
• Scalability
• Language support & Developer Experience
• Relational vs. Non-relational (SQL vs. NoSQL)
• Data types
• Security
• Database environment
  • Client vs Server access

Info on Kris & Harper:

• Website: harperdb.io
• Twitter: @harperdbio, @kriszyp
• Github: @HarperDB, @kriszyp

Episode 163 - Recursion

Intro topic: Electric Cars

News/Links:

• Snake Game in 101 Bytes in a QR Code
  • https://www.reddit.com/r/programming/comments/15ab4ct/my_qr_code_snake_game_is_now_only_101_bytes/
• Superconductor Rumors abound
  • https://arstechnica.com/science/2023/08/whats-going-on-with-the-reports-of-a-room-temperature-superconductor/
• OpenWorm
  • https://github.com/openworm/OpenWorm
• Creator of vim passes away
  • https://news.itsfoss.com/vim-creator-passed-away/
    
    

Book of the Show

• Patrick:
  • Little Book of Common Sense Investing by Jack Bogle https://amzn.to/43YqANR
• Jason: 
  • Mistborn Saga: https://amzn.to/3DJkUN8

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Jason:
  • reMarkable https://remarkable.com/
• Patrick: 
  • Stellarium (iOS and Android)

Topic: Recursion

• What is it
  • Divide-And-Conquer
  • Fibonacci numbers
• How to (not) teach recursion
• Practical Applications
  • Graph operations
    • Tree retrieval, balancing
    • Graph Search
  • Spatial partitioning
• Pitfalls
  • Stack size
• How to solve problems with recursion
  • (1) Consider the base cases
  • (2) Build the recursive step
  • (3) Look for ways the recursion will not terminate and fix
  • (4) (rest are optional) Remove global contexts
  • (5) Add memoization
  • (6) Build solutions incrementally