The PyTorch Developer Podcast is a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch.
…
continue reading
Minhaaj Podcast are Candid Conversations with Some of the Most Intelligent People. From Forbes and WSJ contributors, inventors, wall street bankers, Fintech experts, memory champions, neuroscientists, psychology veterans, FAANG employees and Youtube Educators, i have had the distinct pleasure to learn from these luminaries, for which i shall remain thankful, forever.
…
continue reading
The AI Buzz is a conversation about what’s happening in AI between Willam Falcon, CEO at Lightning AI and the creator of PyTorch Lightning, and Luca Antiga, CTO at Lightning AI. We make the firehose of information accessible to individuals, journalists, executives and investors. Whether you’re an indie developer or seasoned VC, we cover the latest in AI/ML and explore what has the potential to change everything. Feel free to reach out to us on Twitter @_willfalcon @lantiga. Want to meet with ...
…
continue reading
Compiler collectives are a PT2 feature where by compiler instances across multiple ranks use NCCL collectives to communicate information to other instances. This is used to ensure we consistently decide if inputs or static or dynamic across all ranks. See also PR at https://github.com/pytorch/pytorch/pull/130935…
…
continue reading
TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.Από τον PyTorch
…
continue reading
Higher order operators are a special form of operators in torch.ops which have relaxed input argument requirements: in particular, they can accept any form of argument, including Python callables. Their name is based off of their most common use case, which is to represent higher order functions like control flow operators. However, they are also u…
…
continue reading
The post-grad FX passes in Inductor run after AOTAutograd has functionalized and normalized the input program into separate forward/backward graphs. As such, they generally can assume that the graph in question is functionalized, except for some mutations to inputs at the end of the graph. At the end of post-grad passes, there are special passes th…
…
continue reading
CUDA graph trees are the internal implementation of CUDA graphs used in PT2 when you say mode="reduce-overhead". Their primary innovation is that they allow the reuse of memory across multiple CUDA graphs, as long as they form a tree structure of potential paths you can go down with the CUDA graph. This greatly reduced the memory usage of CUDA grap…
…
continue reading
The min-cut partitioner makes decisions about what to save for backwards when splitting the forward and backwards graph from the joint graph traced by AOTAutograd. Crucially, it doesn't actually do a "split"; instead, it is deciding how much of the joint graph should be used for backwards. I also talk about the backward retracing problem.…
…
continue reading
AOTInductor is a feature in PyTorch that lets you export an inference model into a self-contained dynamic library, which can subsequently be loaded and used to run optimized inference. It is aimed primarily at CUDA and CPU inference applications, for situations when your model export once to be exported once while your runtime may still get continu…
…
continue reading
Tensor subclasses allow you to add extend PyTorch with new types of tensors without having to write any C++. They have been used to implement DTensor, FP8, Nested Jagged Tensor and Complex Tensor. Recent work by Brian Hirsh means that we can compile tensor subclasses in PT2, eliminating their overhead. The basic mechanism by which this compilation …
…
continue reading
Compiled autograd is an extension to PT2 that permits compiling the entirety of a backward() call in PyTorch. This allows us to fuse accumulate grad nodes as well as trace through arbitrarily complicated Python backward hooks. Compiled autograd is an important part of our plans for compiled DDP/FSDP as well as for whole-graph compilation.…
…
continue reading
We discuss some extension points for customizing PT2 behavior across Dynamo, AOTAutograd and Inductor.Από τον PyTorch
…
continue reading
Define-by-run IR is how Inductor defines the internal compute of a pointwise/reduction operation. It is characterized by a function that calls a number of functions in the 'ops' namespace, where these ops can be overridden by different handlers depending on what kind of semantic analysis you need to do. The ops Inductor supports include regular ari…
…
continue reading
Traditionally, unsigned integer support in PyTorch was not great; we only support uint8. Recently, we added support for uint16, uint32 and uint64. Bare bones functionality works, but I'm entreating the community to help us build out the rest. In particular, for most operations, we plan to use PT2 to build anything else. But if you have an eager ker…
…
continue reading
Inductor IR is an intermediate representation that lives between ATen FX graphs and the final Triton code generated by Inductor. It was designed to faithfully represent PyTorch semantics and accordingly models views, mutation and striding. When you write a lowering from ATen operators to Inductor IR, you get a TensorBox for each Tensor argument whi…
…
continue reading
I talk about VariableTracker in Dynamo. VariableTracker is Dynamo's representation of the Python. I talk about some recent changes, namely eager guards and mutable VT. I also tell you how to find the functionality you care about in VariableTracker (https://docs.google.com/document/d/1XDPNK3iNNShg07jRXDOrMk2V_i66u1hEbPltcsxE-3E/edit#heading=h.i6v7gq…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Working at DeepMind with Aleksa Gordic
1:49:49
1:49:49
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:49:49
Aleksa Gordic is an ex-software/ML engineer at Microsoft & DeepMind with a broad background across the "whole stack" - maths, electronics, software engineering, algorithms, ML & deep learning (computer vision, natural language processing (NLP), geometric DL, reinforcement learning (RL)...), web, mobile, etc. He is a Top Linkedin Voice in AI for 202…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Julia and Data Science with Bogumil Kaminski
2:07:59
2:07:59
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
2:07:59
Season 2 episode 2 of The Minhaaj Podcast this week brings on the child prodigy and genius co-creator of dataframes.jl package for Julia, Dr Bogumił Kamiński. Bogumil learned C language without owning a computer from library books at the age of 16 in a small Polish town. In post-communist Poland he went on to study applied problems in management an…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
AI powered self serving BI with Ryan Janssen & Paul Blankely - Zenlytic
1:59:07
1:59:07
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:59:07
Ryan is an entrepreneur, data scientist, engineer, and former VC. He is the co-founder and CEO of Zenlytic, a SaaS business that makes a next-generation AI-powered BI tool that uses LLMs and Semantic layers. He previously co-founded Ex Quanta AI Studio, a full-service data consultancy.Ryan started his career as a software developer in his native Ca…
…
continue reading
In this episode we discuss licensing and commercial use for large language models. How does open source licensing affect you as an entrepreneur or decision maker? How do you navigate the line that separates commercial vs non-commercial use? Join our Discord to participate in the discussion: https://discord.gg/94xbKyTX…
…
continue reading
Should AI already be transforming your business? And what does all of this mean for the future of applications? In this episode, we discuss GPT-4 and how the emergence of an increasingly closed-source AI ecosystem can be damaging for the entire industry.Από τον Lightning AI
…
continue reading
In this episode, Luca and I talk about Sarah Guo's advice to AI Entrepreneurs, Aligning models to customer needs, Luca's predictions about the future of AI and Programing without Programming, or Automation for Everyone. Also, if you want to learn more, check out our Read Log: https://lightningai.notion.site/The-AI-Buzz-with-Luca-and-Josh-Episode-5-…
…
continue reading
In this episode, Luca and I talk about ChatGPT + Bing, Google vs Microsoft, Artificially learning high order logic, and how to start an AI company in 3 easy steps. Also, if you want to learn more, check out some of our sources here: https://lightningai.notion.site/Readlog-21-Feb-2023-eb0f44e895ce4c81b5777e6360f1324e…
…
continue reading
This podcast goes over the basics of unbacked SymInts. You might want to listen to this one before listening to https://pytorch-dev-podcast.simplecast.com/episodes/zero-one-specialization Some questions we answer (h/t from Gregory Chanan): - Are unbacked symints only for export? Because otherwise I could just break / wait for the actual size. But m…
…
continue reading
Mikey Dagistes joins me to ask some questions about the recent recent composability sync https://www.youtube.com/watch?v=NJV7YFbtoR4 where we discussed 0/1 specialization and its implications on export in PT2. What's the fuss all about? What do I need to understand about PT2 to understand why 0/1 specialization is a thing?…
…
continue reading
In the first episode of The AI Buzz with Luca and Josh we talk about ChatGPT, Transformers and Attention.Από τον Lightning AI
…
continue reading
In this episode, Luca and Josh chat about Big Data and The Pile, an open source resource for training models, Reinforcement Learning and Aligning Models.Από τον Lightning AI
…
continue reading
In this episode, Luca and Josh talk about Constitutional AI, a way to get models to train themselves not to say bad words, Emergent Abilities and Foundation Models.Από τον Lightning AI
…
continue reading
What is torchdynamo? From a bird's eye view, what exactly does it do? What are some important things to know about it? How does it differ from other graph capture mechanisms? For more reading, check out https://docs.google.com/document/d/13K03JN4gkbr40UMiW4nbZYtsw8NngQwrTRnL3knetGM/edit#Από τον PyTorch
…
continue reading
Soumith's keynote on PT2.0: https://youtu.be/vbtGZL7IrAw?t=1037 PT2 Manifesto: https://docs.google.com/document/d/1tlgPcR2YmC3PcQuYDPUORFmEaBPQEmo8dsh4eUjnlyI/edit# PT2 Architecture: https://docs.google.com/document/d/1wpv8D2iwGkKjWyKof9gFdTf8ISszKbq1tsMVm-3hSuU/edit#Από τον PyTorch
…
continue reading
Join me with Richard Zou to talk about the history of functorch. What was the thought process behind the creation of functorch? How did it get started? JAX’s API and model is fairly different from PyTorch’s, how did we validate that it would work in PyTorch? Where did functorch go after the early user studies? Where is it going next?…
…
continue reading
What’s a learning rate? Why might you want to schedule it? How does the LR scheduler API in PyTorch work? What the heck is up with the formula implementation? Why is everything terrible?Από τον PyTorch
…
continue reading
What are they good for? (Caches. Private fields.) C++ side support, how it’s implemented / release resources. Python side support, how it’s implemented. Weak ref tensor hazard due to resurrection. Downsides of weak references in C++. Scott Wolchok’s release resources optimization. Other episodes to listen to first: https://pytorch-dev-podcast.simpl…
…
continue reading
Mike Ruberry has an RFC about stride-agnostic operator semantics (https://github.com/pytorch/pytorch/issues/78050), so let's talk about strides. What are they? How are they used to implement views and memory format? How do you handle them properly when writing kernels? In what sense are strides overspecified, and therefore, not worth slavishly reim…
…
continue reading
AOTAutograd is a cool new feature in functorch for capturing both forward and backward traces of PyTorch operators, letting you run them through a compiler and then drop the compiled kernels back into a normal PyTorch eager program. Today, Horace joins me to tell me how it works, what it is good to use for, and what our future plans for it are.…
…
continue reading
Sherlock recently joined the PyTorch team, having previously worked on ONNX Runtime at Microsoft, and Sherlock’s going to ask me some questions about the dispatcher, and I’m going to answer them. We talked about the history of the dispatcher, how to override dispatching order, multiple dispatch, how to organize various dispatch keys and torch funct…
…
continue reading
PyTorch recently moved all of its CI from CircleCI to GitHub Actions. There were a lot of improvements in the process, making my old podcast about CI obsolete! Today, Eli Uriegas joins me to talk about why we moved to GitHub Actions, how the new CI system is put together, and what some cool features about our new CI.…
…
continue reading
C++ has exceptions, Python has exceptions. But they’re not the same thing! How do exceptions work in CPython, how do we translate exceptions from C++ to Python (hint: it’s different for direct bindings versus pybind11), and what do warnings (which we also translate from C++ to Python) have in common with this infrastructure?…
…
continue reading
PyTorch’s torch API is the Python API everyone knows and loves, but there’s also another API, the ATen API, which most of PyTorch’s internal subsystems are built on. How to tell them apart? What implications do these have on our graph mode IR design? Also, a plug for PrimTorch, a new set of operators, not designed for eager mode, that is supposed t…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Prosthetic Hands with Aadeel Akhtar
1:45:34
1:45:34
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:45:34
Dr. Akhtar received his Ph.D. in Neuroscience and M.S. in Electrical & Computer Engineering from the University of Illinois at Urbana-Champaign in 2016. He received a B.S. in Biology in 2007 and M.S. in Computer Science in 2008 at Loyola University Chicago. His research is on motor control and sensory feedback for upper limb prostheses, and he has …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Warehouse with Bill Inmon
1:52:13
1:52:13
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:52:13
William H. Inmon (born 1945) is an American computer scientist, recognized by many as the father of the data warehouse. Inmon wrote the first book, held the first conference (with Arnie Barnett), wrote the first column in a magazine and was the first to offer classes in data warehousing. Inmon created the accepted definition of what a data warehous…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Building Data Teams & Culture - Lisa Cohen
1:34:30
1:34:30
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:34:30
Lisa Cohen is the Director of Data Science at Twitter and Formerly at Microsoft for 20 years. He holds a bachelor and a master in Applied Mathematics from Harvard and is one of the most influential women in Data Science and AI.00:00 Intro02:52 Harvard, Microsoft, and Twitter. From SE to Data Science03:40 Work Culture at Microsoft, Bigger Picture & …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Science Careers with Dhaval Patel - Codebasics Youtube
1:52:42
1:52:42
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:52:42
Dhaval Patel is a software & data engineer with more than 17 years of experience. He has been working as a data engineer for a Fintech giant Bloomberg LP (New York) as well as NVidia in the past. He teaches programming, machine learning, data science through YouTube channel CodeBasics which has 428K subscribers worldwide. 00:00 Intro 01:34 Autoimmu…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
BCIs, Neuro Modulation and NeuroEthics with Harrison Canning
1:49:42
1:49:42
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:49:42
Harrison Canning is a student at the Rochester Institute of Technology in the School of Individualized Studies, Founder of The BCI Guys & Neurotechnology Exploration Team. He makes videos on his Youtube channel The BCI Guys and has designed his own degree centered around brain-computer interface technology (BA in Neurotechnology). The BCI Guys is a…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Pytorch Geometric with Matthias Fey
1:31:46
1:31:46
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:31:46
Matthias Fey is the creator of the Pytorch Geometric library and a postdoctoral researcher in deep learning at TU Dortmund Germany. He is a core contributor to the Open Graph Benchmark dataset initiative in collaboration with Stanford University Professor Jure Leskovec. 00:00 Intro 00:50 Pytorch Geometric Inception 02:57 Graph NNs vs CNNs, Transfor…
…
continue reading
PyTorch is in the business of shipping numerical software that can run fast on your CUDA-enabled NVIDIA GPU, but it turns out there is a lot of heterogeneity in NVIDIA’s physical GPU offering and when it comes to what is fast and what is slow, what specific GPU you have on hand matters quite a bit. Yet there are literally hundreds of distinct NVIDI…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Graph Neural Networks with Ankit Jain
1:59:03
1:59:03
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:59:03
Ankit is an experienced AI Researcher/Machine Learning Engineer who is passionate about using AI to build scalable machine learning products. In his 10 years of AI career, he has researched and deployed several state-of-the-art machine learning models which have impacted 100s of millions of users. Currently, He works as a senior research scientist …
…
continue reading
A lot of recent work going in PyTorch is all about adding new and interesting Tensor subclasses, and this all leads up to the question of, what exactly is OK to make a tensor subclass? One answer to this question comes from an old principle from Barbara Liskov called the Liskov substitution principle, which informally can be stated as S is a subtyp…
…
continue reading
In this episode I talk about reduced precision floating point formats float16 (aka half precision) and bfloat16. I'll discuss what floating point numbers are, how these two formats vary, and some of the practical considerations that arise when you are working with numeric code in PyTorch that also needs to work in reduced precision. Did you know th…
…
continue reading
Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these th…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
AI in Supply Chain and Economics of Logistics - Frank Corrigan
1:58:55
1:58:55
Αναπαραγωγή αργότερα
Αναπαραγωγή αργότερα
Λίστες
Like
Liked
1:58:55
Francis Corrigan is Director of Decision Intelligence at Target Corporation. Embedded within the Global Supply Chain, Decision Intelligence combines data science with model thinking to help decision-makers solve problems. 00:00 Intro 01:21 Data Science applications in Logistics and Supply Chain, Cost and Performance trade-off 03:21 Amazon vs Target…
…
continue reading
PyTorch operates on its input data in a batched manner, typically processing multiple batches of an input at once (rather than once at a time, as would be the case in typical programming). In this podcast, we talk a little about the implications of batching operations in this way, and then also about how PyTorch's API is structured for batching (hi…
…
continue reading