peggy1502/Amazing-Resources

List of references and online resources related to data science, machine learning and deep learning.

updated Jul 21, 2026

See the code

README

Amazing Resources

List of references and online resources related to data science, machine learning and deep learning.

👍 Courses / Tutorials

👍 Cheat Sheets

👍 AWS / SageMaker

📺 Videos

📚 Books

👍 Papers

2015 Cyclical Learning Rates for Training Neural Networks (https://arxiv.org/abs/1506.01186)
2017 Decoupled Weight Decay Regularization (https://arxiv.org/abs/1711.05101)
2018 Mixed Precision Training (https://arxiv.org/pdf/1710.03740.pdf)
2020 ReadNet: A Hierarchical Transformer Framework for Web Article Readability Analysis (https://link.springer.com/chapter/10.1007/978-3-030-45439-5_3)
2021 A Survey of Transformers (https://arxiv.org/pdf/2106.04554.pdf)
2022 Formal Algorithms for Transformers (https://arxiv.org/pdf/2207.09238.pdf)
2023 Transformer models: an introduction and catalog (https://arxiv.org/pdf/2302.07730.pdf) (http://bit.ly/3YFqRn9)

📑 Articles

🖼️ CNN

↩️ RNN

⁉️ NLP

💏 NVIDIA Recommender Systems

💏 Collaborative Filtering / Recommender Systems

👫 Similarity Search / ANNS / Vector Indexing

🆎 Code Search

🌐 Search Engine / Information Retrieval

⬆️ Search Ranking

👫 Lucene / Solr / Elasticsearch / BM25

📑 General ML/DL Articles

⏱️ Time Series

📊 EDA / Data Visualization

📘 Colab

🐍 Python

🗄️ Database / Storage

🏃 Reinforcement Learning

👍 Interesting and Fun

👍 GitHub Repositories

👍 Kaggle

📘 DeepNote

👍 Blogs

👍 Company Tech Blogs

🔟 Maths

👍 Datasets

👍 Synthetic Data

🛠️ Utilities / Tools

👍 Job / Interview / DS Portfolio

💰 Salary Negotiation

👍 System Design

🆎 Algorithms / Technical Coding

📺 Videos for Algorithms / Technical Coding / Interview Prep

🔢 Bit Hacks

👍 Google foobar

⏩ fast.ai

📚 NVIDIA eBooks

🧞 Genomic Data Science

🤖 Transformer Architecture / Anatomy / Guide

🤖 Transformer / Attention / LLM Visualization

🤖 Transformer Maths

🤖 Transformer Libraries

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [Paper] (https://github.com/mit-han-lab/llm-awq)
AutoAWQ (https://github.com/casper-hansen/AutoAWQ)
Microsoft DeepSpeed - Deep learning optimization software suite for both training and inference (https://github.com/microsoft/DeepSpeed) (https://www.deepspeed.ai/)
DeepSpeed Chat - Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales (https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) (https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat)
DeepSpeed's Bag of Tricks for Speed & Scale (https://www.kolaayonrinde.com/blog/2023/07/14/deepspeed-train.html)
🤗 PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware (https://huggingface.co/blog/peft) (https://github.com/huggingface/peft)
- 🤗 PEFT Documentation (https://huggingface.co/docs/peft/index)
- 🤗 PEFT Examples (https://github.com/huggingface/peft/tree/main/examples)
- 🤗 PEFT Patch Release (https://github.com/huggingface/peft/releases)
🤗 TRL - Transformer Reinforcement Learning (https://github.com/huggingface/trl) (https://huggingface.co/docs/trl/index)
bitsandbytes - 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization functions (https://github.com/TimDettmers/bitsandbytes)
GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers (https://github.com/IST-DASLab/gptq)
AutoGPTQ: LLMs quantization package with user-friendly apis, based on GPTQ algorithm (https://github.com/PanQiWei/AutoGPTQ)
Quantize 🤗 Transformers models (https://huggingface.co/docs/transformers/main_classes/quantization)
Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ) (https://maartengrootendorst.substack.com/p/which-quantization-method-is-right)
Optimum-Benchmark 🏋️ (https://github.com/huggingface/optimum-benchmark)
NEFTune - add random noise to the embedding vectors of the training data during the forward pass of fine-tuning (https://github.com/neelsjain/NEFTune)
LoRA+: Efficient Low Rank Adaptation of Large Models (https://github.com/nikhil-ghosh-berkeley/loraplus)
DoRA: Weight-Decomposed Low-Rank Adaptation (https://github.com/catid/dora/tree/main)
tensor_parallel - much faster than huggingface's device_map and lightweight than vLLM? (https://github.com/BlackSamorez/tensor_parallel)
Nanotron - Minimalistic large language model 3D-parallelism training (https://github.com/huggingface/nanotron/)
FastChat - platform for training, serving, and evaluating large language model based chatbots (https://github.com/lm-sys/FastChat)
Half-Quadratic Quantization (HQQ) (https://mobiusml.github.io/hqq_blog/) (https://github.com/mobiusml/hqq)
AQLM - Extreme Compression of Large Language Models via Additive Quantization (https://github.com/Vahe1994/AQLM) (https://towardsdatascience.com/the-aqlm-quantization-algorithm-explained-8cf33e4a783e)
torchtune - A Native-PyTorch Library for LLM Fine-tuning (https://github.com/pytorch/torchtune)
torchao: PyTorch Architecture Optimization (https://github.com/pytorch/ao/)
- PyTorch Native Architecture Optimization: torchao (https://pytorch.org/blog/pytorch-native-architecture-optimization/)
Prefect - Modern workflow orchestration for data and ML engineers (https://www.prefect.io/)
Modal - serverless platform to run generative AI models, large-scale batch jobs, job queues, etc (https://modal.com/)
- Beating Proprietary Models with a Quick Fine-Tune - Finetuning Quora Embeddings (https://modal.com/blog/fine-tuning-embeddings) (https://github.com/567-labs/fastllm/blob/main/applications/finetune-quora-embeddings/Readme.md)
- Fine-tune an LLM in minutes (ft. Llama 2, CodeLlama, Mistral, etc.) (https://modal.com/docs/examples/llm-finetuning) (https://github.com/modal-labs/llm-finetuning)
Model Explorer - a powerful graph visualization tool that helps one understand, debug, and optimize ML models (https://ai.google.dev/edge/model-explorer) (https://research.google/blog/model-explorer/)
Intel AutoRound - weight-only quantization algorithm designed specifically for low-bit LLM inference (https://github.com/intel/auto-round) (https://medium.com/intel-analytics-software/autoround-sota-weight-only-quantization-algorithm-for-llms-across-hardware-platforms-99fe6eac2861)

🤖 Transformer Toolkit / Techniques / Methods

🤖 RAG

🤖 Lightning AI ⚡⚡⚡

🤖 LLM Leaderboard

🤖 LLM Evaluation

FastChat - platform for training, serving, and evaluating large language model based chatbots (https://github.com/lm-sys/FastChat)
LLM Evaluation Metrics: Everything You Need for LLM Evaluation (https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation)
DeepEval - open-source LLM evaluation framework specialized for unit testing LLM outputs (https://github.com/confident-ai/deepeval)
Language Model Evaluation Harness - based on tasks (https://github.com/EleutherAI/lm-evaluation-harness) (https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md)
LLM Task-Specific Evals that Do & Don't Work (https://eugeneyan.com/writing/evals/)
RULER: What’s the Real Context Size of Your Long-Context Language Models? Evaluate long-context language models with configurable sequence length and task complexity (https://github.com/NVIDIA/RULER)

🤖 Transformer Models / Timeline

🤖 Transformer / LLM Inference / Deployment

7 Ways To Speed Up Inference of Your Hosted LLMs (https://betterprogramming.pub/speed-up-llm-inference-83653aa24c47)
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (https://github.com/vllm-project/vllm) (https://vllm.ai/)
- vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (https://blog.vllm.ai/2024/09/05/perf-update.html)
🤗 TGI: Text Generation Inference - Fast optimized inference for LLMs (https://github.com/huggingface/text-generation-inference)
LMDeploy: a toolkit for compressing, deploying, and serving LLM (https://github.com/InternLM/lmdeploy)
OpenVINO: an open-source toolkit for optimizing and deploying AI inference (https://github.com/openvinotoolkit) (https://docs.openvino.ai/2023.0/home.html)
How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (https://www.anyscale.com/blog/continuous-batching-llm-inference)
Squeeze more out of your GPU for LLM inference—a tutorial on Accelerate & DeepSpeed (https://preemo.medium.com/squeeze-more-out-of-your-gpu-for-llm-inference-a-tutorial-on-accelerate-deepspeed-610fce3025fd)
Performance bottlenecks in deploying LLMs—a primer for ML researchers (https://preemo.medium.com/performance-bottlenecks-in-deploying-llms-a-primer-for-ml-researchers-c2b51c2084a8)
Inference using the pre-trained Alpaca-LoRA (https://www.mlexpert.io/machine-learning/tutorials/alpaca-and-llama-inference) (https://colab.research.google.com/drive/15VstUxU48CT3mRudFrj3FIv6Z4cIXnon?usp=sharing)
Optimizing your LLM in production (https://huggingface.co/blog/optimize-llm)
StreamingLLM: Efficient Streaming Language Models with Attention Sinks (https://github.com/mit-han-lab/streaming-llm) (https://arxiv.org/pdf/2309.17453.pdf)
S-LoRA: Serving Thousands of Concurrent LoRA Adapters (https://github.com/s-lora/s-lora)
- Recipe for Serving Thousands of Concurrent LoRA Adapters (https://lmsys.org/blog/2023-11-15-slora/)
DeepSparse by Neural Magic - Sparsity-aware deep learning inference runtime for CPUs (https://github.com/neuralmagic/deepsparse/tree/main)
SparseML by Neural Magic - an open-source model optimization toolkit that enables you to create inference-optimized sparse models using pruning, quantization, and distillation algorithms (https://github.com/neuralmagic/sparseml)
Marlin - Mixed Auto-Regressive Linear kernel, an extremely optimized FP16xINT4 matmul kernel aimed at LLM inference (https://github.com/IST-DASLab/marlin)
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning (https://github.com/datamllab/LongLM) (https://www.reddit.com/r/LocalLLaMA/comments/18x8g6c/llm_maybe_longlm_selfextend_llm_context_window/)
Flash-Decoding for long-context inference (https://pytorch.org/blog/flash-decoding/)
10 Ways To Run LLMs Locally And Which One Works Best For You (https://matilabs.ai/2024/02/07/run-llms-locally/)
Towards 100x Speedup: Full Stack Transformer Inference Optimization (https://yaofu.notion.site/Towards-100x-Speedup-Full-Stack-Transformer-Inference-Optimization-43124c3688e14cffaf2f1d6cbdf26c6c)
Deploy Deep Learning Models at Scale using NVIDIA Triton Inference Server (https://github.com/decodingml/articles-code/tree/main/articles/computer_vision/deploy_deep_learning_at_scale_nvidia_triton_server)
LMDeploy - a toolkit for compressing, deploying, and serving LLM (https://github.com/InternLM/lmdeploy)
LLM Inference Series: 5. Dissecting model performance (https://medium.com/@plienhar/llm-inference-series-5-dissecting-model-performance-6144aa93168f)
How to compute LLM embeddings 3X faster with model quantization - with ONNX model quantization / ONNX transformer optimization (https://medium.com/nixiesearch/how-to-compute-llm-embeddings-3x-faster-with-model-quantization-25523d9b4ce5)
A Hitchhiker’s Guide to Speculative Decoding (https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/)
Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) (https://lmsys.org/blog/2024-07-25-sglang-llama3/) (https://github.com/sgl-project/sglang)
Awesome Production Machine Learning - open source libraries that will help you deploy, monitor, version, scale, and secure your production machine learning (https://github.com/EthicalML/awesome-production-machine-learning)
NanoFlow - a throughput-oriented high-performance serving framework for LLMs (https://github.com/efeslab/Nanoflow) (https://arxiv.org/pdf/2408.12757)
GuideLLM - evaluating and optimizing the deployment of large language models (LLMs) (https://github.com/neuralmagic/guidellm)
LLM Compressor - create compressed models for faster inference with vLLM (https://github.com/vllm-project/llm-compressor) (https://neuralmagic.com/blog/llm-compressor-is-here-faster-inference-with-vllm/)
bitnet.cpp - Official inference framework for 1-bit LLMs (https://github.com/microsoft/BitNet)
STRING - a training-free method to improve effective context length of popular RoPE-based LLMs (https://github.com/HKUNLP/STRING)

🤖 Transformer / LLM Platform / Software

GPT4All - run open-source LLMs on your own computer (https://github.com/nomic-ai/gpt4all)
LLaMA-Factory - Efficiently Fine-Tune 100+ LLMs in WebUI (https://github.com/hiyouga/LLaMA-Factory)
Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs (https://github.com/facebookresearch/lingua)
bolt.diy (Previously oTToDev) - Prompt, run, edit, and deploy full-stack web applications using any LLM you want! (https://github.com/stackblitz-labs/bolt.diy)
WebUI - Run AI Agent in your browser (https://github.com/browser-use/web-ui)
Free LLM API resources (https://github.com/cheahjs/free-llm-api-resources)

🤖 Transformer / LLM Data Curator

Curating Trillion-Token Datasets: Introducing NVIDIA NeMo Data Curator (https://developer.nvidia.com/blog/curating-trillion-token-datasets-introducing-nemo-data-curator/)
ftfy: fixes text for you (https://github.com/rspeer/python-ftfy)
Cosmopedia: how to create large-scale synthetic data for pre-training (https://huggingface.co/blog/cosmopedia) (https://github.com/huggingface/cosmopedia)
DataTrove - a library to process, filter and deduplicate text data at a very large scale (https://github.com/huggingface/datatrove/)
Guidance - control how LLM output is structured (https://github.com/guidance-ai/guidance)
Large-scale Near-deduplication Behind BigCode (https://huggingface.co/blog/dedup)
Dolma Toolkit - curation of large datasets for (pre)-training ML models (https://github.com/allenai/dolma)
Distilabel - framework for synthetic data and AI feedback (https://github.com/argilla-io/distilabel)
LLM Decontaminator (https://github.com/lm-sys/llm-decontaminator)
Tutorial to demonstrate how to reproduce Zyda2 dataset, curated by Zyphra in collaboration with Nvidia using NeMo Curator (https://github.com/NVIDIA/NeMo-Curator/tree/main/tutorials/zyda2-tutorial) (https://www.zyphra.com/post/building-zyda-2)

🤖 Transformer / LLM Dataset

DBPedia (https://www.dbpedia.org/resources/individual/) (http://downloads.dbpedia.org/wiki-archive/dbpedia-version-2016-04.html) (http://downloads.dbpedia.org/2016-04/core/)
Common Crawl (https://commoncrawl.org/the-data/get-started/)
c4 - A colossal, cleaned version of Common Crawl's web crawl corpus (https://tensorflow.org/datasets/catalog/c4)
c4 processed version with five variants of the data: en, en.noclean, en.noblocklist, realnewslike, and multilingual (mC4). (https://huggingface.co/datasets/allenai/c4)
RedPajama-V2: An open dataset with 30 trillion tokens for training large language models (https://www.together.ai/blog/redpajama-data-v2) (https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2)
SlimPajama-627B - Extensively deduplicated, multi-corpora, open-source dataset for training LLM (https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama) (https://huggingface.co/datasets/cerebras/SlimPajama-627B)
Falcon RefinedWeb - An English large-scale dataset (5 trillion tokens ) for the pretraining of LLM, built through stringent filtering and extensive deduplication of CommonCrawl (https://huggingface.co/datasets/tiiuae/falcon-refinedweb) (https://arxiv.org/abs/2306.01116)
Instruction tuning datasets to train (text and multi-modal) chat-based LLMs (GPT-4, ChatGPT, LLaMA, Alpaca) (https://github.com/yaodongC/awesome-instruction-dataset)
Python-Code-23k-ShareGPT (https://huggingface.co/datasets/ajibawa-2023/Python-Code-23k-ShareGPT)
UltraFeedback Binarized - A pre-processed version of the UltraFeedback dataset and was used to train Zephyr-7Β-β (https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)
[Blog post] FineWeb: decanting the web for the finest text data at scale (https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1)
- FineWeb: 15T tokens (44TB disk space) of cleaned and deduplicated english web data from CommonCrawl, for LLM pretraining (https://huggingface.co/datasets/HuggingFaceFW/fineweb)
- FineWeb-Edu: 1.3T tokens of educational web pages filtered from 🍷 FineWeb dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
- FineWeb-Edu-score-2: 5.4T tokens of educational web pages filtered from 🍷 FineWeb dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2)
Dolma - Used to train OLMo on 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials, with quality filtering, fuzzy deduplication (https://blog.allenai.org/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64) (https://huggingface.co/datasets/allenai/dolma)
TinyStories - synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary (https://huggingface.co/datasets/roneneldan/TinyStories)
GenQA - over 10M cleaned and deduplicated instruction samples generated from a handful of carefully designed prompts (https://huggingface.co/datasets/tomg-group-umd/GenQA)
Persona Hub - Scaling Synthetic Data Creation with 1,000,000,000 Personas (https://github.com/tencent-ailab/persona-hub) (https://huggingface.co/datasets/proj-persona/PersonaHub)
FinePersonas - detailed personas for creating customized, realistic synthetic data (https://huggingface.co/datasets/argilla/FinePersonas-v0.1)
APIGen Function-Calling Datasets (https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k)
The Tome - Compiled from 9 publicly available datasets, curated and designed for training LLMs with a focus on instruction following (https://huggingface.co/datasets/arcee-ai/The-Tome)
distilabel-intel-orca-dpo-pairs (for preference tuning) (https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs)
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens (https://blog.salesforceairesearch.com/mint-1t/) (https://huggingface.co/collections/mlfoundations/mint-1t-6690216ca4d0df7e518dde1c)
Zyda-2 is a 5-trillion token dataset composed of filtered and cross-deduplicated DCLM, FineWeb-Edu, Zyda-1, and Dolma v1.7's Common Crawl portion (https://huggingface.co/datasets/Zyphra/Zyda-2) (https://www.zyphra.com/post/building-zyda-2)

🤖 Transformer / LLM Sample Applications

🤖 MLOps

🤖 Q&A

🤖 Discussion on LLM Padding / Formatting Function

🤖 Merging weights with quantized model

🤖 Merge / Fusion / MoE

🤖 Prompt Engineering / Instructions

🤖 Agent

🤖 LLM - Misc

How Do Language Models put Attention Weights over Long Context? (https://yaofu.notion.site/How-Do-Language-Models-put-Attention-Weights-over-Long-Context-10250219d5ce42e8b465087c383a034e) (https://github.com/FranxYao/Long-Context-Data-Engineering)
AI Watermarking 101: Tools and Techniques (https://huggingface.co/blog/watermarking)
GGUF, the long way around (https://vickiboykis.com/2024/02/28/gguf-the-long-way-around/)
A Comprehensive Guide to Modeling Techniques in Mixed-Integer Linear Programming - Convert ideas into mathematical expressions to solve operations research problems (https://towardsdatascience.com/a-comprehensive-guide-to-modeling-techniques-in-mixed-integer-linear-programming-3e96cc1bc03d)
Mastering ML Configurations by leveraging OmegaConf and Hydra (https://decodingml.substack.com/p/mastering-ml-configurations-by-leveraging)
UltraChat - example training script with Accelerator (https://github.com/thunlp/UltraChat/blob/main/train/train_legacy/train.py)
Using LESS Data to Tune Models (https://www.cs.princeton.edu/~smalladi/blog/2024/04/04/dataselection/)
Techniques for training large neural networks - Data parallelism, Pipeline parallelism, Tensor parallelism, Mixture-of-Experts (MoE), and other memory saving designs (https://openai.com/research/techniques-for-training-large-neural-networks)
- Large Scale Transformer model training with Tensor Parallel (TP) (https://pytorch.org/tutorials/intermediate/TP_tutorial.html)
YaFSDP - a Sharded Data Parallelism framework (https://github.com/yandex/YaFSDP) (https://habr.com/ru/companies/yandex/articles/817509/)
Best Embedding Model — OpenAI / Cohere / Google / E5 / BGE - An In-depth Comparison of Multilingual Embedding Models (https://medium.com/@lars.chr.wiik/best-embedding-model-openai-cohere-google-e5-bge-931bfa1962dc) topic dataset (https://github.com/LarsChrWiik/lars_datasets/tree/main/topics_dataset_50)
Training and Finetuning Embedding Models with Sentence Transformers v3 (https://huggingface.co/blog/train-sentence-transformers)
What can LLMs never do? (https://www.strangeloopcanon.com/p/what-can-llms-never-do)
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/)
Uncensor any LLM with abliteration (https://huggingface.co/blog/mlabonne/abliteration) (https://colab.research.google.com/drive/1VYm3hOcvCpbGiqKZb141gJwjdmmCcVpR?usp=sharing)
- abliterator.py (https://github.com/FailSpy/abliterator)
- TransformerLens - A library for mechanistic interpretability of GPT-style language models (https://github.com/TransformerLensOrg/TransformerLens) (https://transformerlensorg.github.io/TransformerLens/)
Training a 70B model from scratch: open-source tools, evaluation datasets, and learnings (https://imbue.com/research/70b-intro/)
- From bare metal to a 70B model: infrastructure set-up and scripts (https://imbue.com/research/70b-infrastructure/)
- Ensuring accurate model evaluations: open-sourced, cleaned datasets for models that reason and code (https://imbue.com/research/70b-evals/)
Aleksa Gordić’s Post: Amazing list of techniques for improving the stability of training large ML models (LLMs, diffusion, etc) (https://www.linkedin.com/feed/update/urn:li:activity:7215624025639645184/)
The AdEMAMix Optimizer: Better, Faster, Older. A simple modification of the Adam optimizer with a mixture of two Exponential Moving Average (EMA) (https://github.com/nanowell/AdEMAMix-Optimizer-Pytorch/)
Generating Human-level Text with Contrastive Search in Transformers (https://huggingface.co/blog/introducing-csearch)
- A Contrastive Framework for Neural Text Generation (https://github.com/yxuansu/SimCTG)
Open Source LLM Tools (https://huyenchip.com/llama-police)
- What I learned from looking at 900 most popular open source AI tools (https://huyenchip.com/2024/03/14/ai-oss.html)
Imagen - Pytorch (https://github.com/lucidrains/imagen-pytorch)
- MinImagen - A Minimal implementation of the Imagen text-to-image model(https://github.com/AssemblyAI-Community/MinImagen)
- How Imagen Actually Works (https://www.assemblyai.com/blog/how-imagen-actually-works/)
- MinImagen - Build Your Own Imagen Text-to-Image Model (https://www.assemblyai.com/blog/minimagen-build-your-own-imagen-text-to-image-model/)
How to Beat Proprietary LLMs With Smaller Open Source Models (https://www.aidancooper.co.uk/how-to-beat-proprietary-llms/)
A Guide to Structured Outputs Using Constrained Decoding (https://www.aidancooper.co.uk/constrained-decoding/)
The 6 Best LLM Tools To Run Models Locally (https://medium.com/@amosgyamfi/the-6-best-llm-tools-to-run-models-locally-eedd0f7c2bbd)
You can now train a 70b language model at home - Training LLMs with QLoRA + FSDP (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html) (https://github.com/AnswerDotAI/fsdp_qlora/tree/main)
Bugs in LLM Training - Gradient Accumulation Fix (https://unsloth.ai/blog/gradient)
- 🤗 Fixing Gradient Accumulation (https://huggingface.co/blog/gradient_accumulation)
SynthID Text - Apply watermarks and identify AI-generated content (https://huggingface.co/blog/synthid-text) (https://deepmind.google/technologies/synthid/)
How to Build ANYTHING You Imagine With DeepSeek-R1 (Zero Coding Required) (https://iamshobhitagarwal.medium.com/how-to-build-anything-you-imagine-with-deepseek-r1-zero-coding-required-free-0d582513e7ab) $$
- BOLT.diy: Build UNLIMITED Apps in Minutes 🚀 (No Code Required) (https://www.youtube.com/watch?v=QQw47XzcKRQ)
  - Form Setup Guide - NocoDB (https://www.youtube.com/watch?v=GEsFSadshpg)
  - Email Automation - N8n (https://www.youtube.com/watch?v=gofNLKlrFCQ)

🤖 Llama

🤖 Unsloth

🤖 Transformer Alternatives

Retentive Networks (RetNet) Explained: The much-awaited Transformers-killer is here (https://medium.com/ai-fusion-labs/retentive-networks-retnet-explained-the-much-awaited-transformers-killer-is-here-6c17e3e8add8)
Mamba Explained - The State Space Model taking on Transformers (https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html)
Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model (https://www.ai21.com/blog/announcing-jamba)
ModuleFormer: a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts (https://github.com/IBM/ModuleFormer)
JetMoE: Reaching LLaMA2 Performance with 0.1M Dollars (https://research.myshell.ai/jetmoe)
SUPRA: Scalable UPtraining for Recurrent Attention - uptrain a transformer to a linear RNN (https://github.com/TRI-ML/linear_open_lm) (https://huggingface.co/TRI-ML/mistral-supra)
📺 Understanding Mamba and State Space Models (https://www.youtube.com/watch?v=iskuX3Ak9Uk)

👍 Google AI/ML Use Cases

Diffusion Models

Contributors

peggy1502

canelkaradeniz

Similar repos

ElizaLo/Data-Science

Projects and awesome list for all Data Science fields

bishwaghimire/ai-learning-roadmaps

A complete, structured hub for learning Artificial Intelligence — covering AI, Machine Learning,…

tunguz/ML_Resources

GitHub Repo with various ML/AI/DS resources that I find useful

ashara12/awesome-deeplearning

A curated list of awesome Deep Learning tutorials, projects and communities.

DataExpert-io/ai-engineer-handbook

All the links, books, and creators you need to follow to stay up to date with AI!

ChristosChristofidis/awesome-deep-learning

A curated list of awesome Deep Learning tutorials, projects and communities.

yanshengjia/ml-road

Machine Learning and Agentic AI Resources, Practice and Research

ritchieng/the-incredible-pytorch

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more…

peggy1502/Amazing-Resources

List of references and online resources related to data science, machine learning and deep learning.

updated Jul 21, 2026

See the code

README

Amazing Resources

List of references and online resources related to data science, machine learning and deep learning.

👍 Courses / Tutorials

👍 Cheat Sheets

👍 AWS / SageMaker

📺 Videos

📚 Books

👍 Papers

2015 Cyclical Learning Rates for Training Neural Networks (https://arxiv.org/abs/1506.01186)
2017 Decoupled Weight Decay Regularization (https://arxiv.org/abs/1711.05101)
2018 Mixed Precision Training (https://arxiv.org/pdf/1710.03740.pdf)
2020 ReadNet: A Hierarchical Transformer Framework for Web Article Readability Analysis (https://link.springer.com/chapter/10.1007/978-3-030-45439-5_3)
2021 A Survey of Transformers (https://arxiv.org/pdf/2106.04554.pdf)
2022 Formal Algorithms for Transformers (https://arxiv.org/pdf/2207.09238.pdf)
2023 Transformer models: an introduction and catalog (https://arxiv.org/pdf/2302.07730.pdf) (http://bit.ly/3YFqRn9)

📑 Articles

🖼️ CNN

↩️ RNN

⁉️ NLP

💏 NVIDIA Recommender Systems

💏 Collaborative Filtering / Recommender Systems

👫 Similarity Search / ANNS / Vector Indexing

🆎 Code Search

🌐 Search Engine / Information Retrieval

⬆️ Search Ranking

👫 Lucene / Solr / Elasticsearch / BM25

📑 General ML/DL Articles

⏱️ Time Series

📊 EDA / Data Visualization

📘 Colab

🐍 Python

🗄️ Database / Storage

🏃 Reinforcement Learning

👍 Interesting and Fun

👍 GitHub Repositories

👍 Kaggle

📘 DeepNote

👍 Blogs

👍 Company Tech Blogs

🔟 Maths

👍 Datasets

👍 Synthetic Data

🛠️ Utilities / Tools

👍 Job / Interview / DS Portfolio

💰 Salary Negotiation

👍 System Design

🆎 Algorithms / Technical Coding

📺 Videos for Algorithms / Technical Coding / Interview Prep

🔢 Bit Hacks

👍 Google foobar

⏩ fast.ai

📚 NVIDIA eBooks

🧞 Genomic Data Science

🤖 Transformer Architecture / Anatomy / Guide

🤖 Transformer / Attention / LLM Visualization

🤖 Transformer Maths

🤖 Transformer Libraries

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [Paper] (https://github.com/mit-han-lab/llm-awq)
AutoAWQ (https://github.com/casper-hansen/AutoAWQ)
Microsoft DeepSpeed - Deep learning optimization software suite for both training and inference (https://github.com/microsoft/DeepSpeed) (https://www.deepspeed.ai/)
DeepSpeed Chat - Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales (https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) (https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat)
DeepSpeed's Bag of Tricks for Speed & Scale (https://www.kolaayonrinde.com/blog/2023/07/14/deepspeed-train.html)
🤗 PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware (https://huggingface.co/blog/peft) (https://github.com/huggingface/peft)
- 🤗 PEFT Documentation (https://huggingface.co/docs/peft/index)
- 🤗 PEFT Examples (https://github.com/huggingface/peft/tree/main/examples)
- 🤗 PEFT Patch Release (https://github.com/huggingface/peft/releases)
🤗 TRL - Transformer Reinforcement Learning (https://github.com/huggingface/trl) (https://huggingface.co/docs/trl/index)
bitsandbytes - 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization functions (https://github.com/TimDettmers/bitsandbytes)
GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers (https://github.com/IST-DASLab/gptq)
AutoGPTQ: LLMs quantization package with user-friendly apis, based on GPTQ algorithm (https://github.com/PanQiWei/AutoGPTQ)
Quantize 🤗 Transformers models (https://huggingface.co/docs/transformers/main_classes/quantization)
Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ) (https://maartengrootendorst.substack.com/p/which-quantization-method-is-right)
Optimum-Benchmark 🏋️ (https://github.com/huggingface/optimum-benchmark)
NEFTune - add random noise to the embedding vectors of the training data during the forward pass of fine-tuning (https://github.com/neelsjain/NEFTune)
LoRA+: Efficient Low Rank Adaptation of Large Models (https://github.com/nikhil-ghosh-berkeley/loraplus)
DoRA: Weight-Decomposed Low-Rank Adaptation (https://github.com/catid/dora/tree/main)
tensor_parallel - much faster than huggingface's device_map and lightweight than vLLM? (https://github.com/BlackSamorez/tensor_parallel)
Nanotron - Minimalistic large language model 3D-parallelism training (https://github.com/huggingface/nanotron/)
FastChat - platform for training, serving, and evaluating large language model based chatbots (https://github.com/lm-sys/FastChat)
Half-Quadratic Quantization (HQQ) (https://mobiusml.github.io/hqq_blog/) (https://github.com/mobiusml/hqq)
AQLM - Extreme Compression of Large Language Models via Additive Quantization (https://github.com/Vahe1994/AQLM) (https://towardsdatascience.com/the-aqlm-quantization-algorithm-explained-8cf33e4a783e)
torchtune - A Native-PyTorch Library for LLM Fine-tuning (https://github.com/pytorch/torchtune)
torchao: PyTorch Architecture Optimization (https://github.com/pytorch/ao/)
- PyTorch Native Architecture Optimization: torchao (https://pytorch.org/blog/pytorch-native-architecture-optimization/)
Prefect - Modern workflow orchestration for data and ML engineers (https://www.prefect.io/)
Modal - serverless platform to run generative AI models, large-scale batch jobs, job queues, etc (https://modal.com/)
- Beating Proprietary Models with a Quick Fine-Tune - Finetuning Quora Embeddings (https://modal.com/blog/fine-tuning-embeddings) (https://github.com/567-labs/fastllm/blob/main/applications/finetune-quora-embeddings/Readme.md)
- Fine-tune an LLM in minutes (ft. Llama 2, CodeLlama, Mistral, etc.) (https://modal.com/docs/examples/llm-finetuning) (https://github.com/modal-labs/llm-finetuning)
Model Explorer - a powerful graph visualization tool that helps one understand, debug, and optimize ML models (https://ai.google.dev/edge/model-explorer) (https://research.google/blog/model-explorer/)
Intel AutoRound - weight-only quantization algorithm designed specifically for low-bit LLM inference (https://github.com/intel/auto-round) (https://medium.com/intel-analytics-software/autoround-sota-weight-only-quantization-algorithm-for-llms-across-hardware-platforms-99fe6eac2861)

🤖 Transformer Toolkit / Techniques / Methods

🤖 RAG

🤖 Lightning AI ⚡⚡⚡

🤖 LLM Leaderboard

🤖 LLM Evaluation

FastChat - platform for training, serving, and evaluating large language model based chatbots (https://github.com/lm-sys/FastChat)
LLM Evaluation Metrics: Everything You Need for LLM Evaluation (https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation)
DeepEval - open-source LLM evaluation framework specialized for unit testing LLM outputs (https://github.com/confident-ai/deepeval)
Language Model Evaluation Harness - based on tasks (https://github.com/EleutherAI/lm-evaluation-harness) (https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md)
LLM Task-Specific Evals that Do & Don't Work (https://eugeneyan.com/writing/evals/)
RULER: What’s the Real Context Size of Your Long-Context Language Models? Evaluate long-context language models with configurable sequence length and task complexity (https://github.com/NVIDIA/RULER)

🤖 Transformer Models / Timeline

🤖 Transformer / LLM Inference / Deployment

7 Ways To Speed Up Inference of Your Hosted LLMs (https://betterprogramming.pub/speed-up-llm-inference-83653aa24c47)
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (https://github.com/vllm-project/vllm) (https://vllm.ai/)
- vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (https://blog.vllm.ai/2024/09/05/perf-update.html)
🤗 TGI: Text Generation Inference - Fast optimized inference for LLMs (https://github.com/huggingface/text-generation-inference)
LMDeploy: a toolkit for compressing, deploying, and serving LLM (https://github.com/InternLM/lmdeploy)
OpenVINO: an open-source toolkit for optimizing and deploying AI inference (https://github.com/openvinotoolkit) (https://docs.openvino.ai/2023.0/home.html)
How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (https://www.anyscale.com/blog/continuous-batching-llm-inference)
Squeeze more out of your GPU for LLM inference—a tutorial on Accelerate & DeepSpeed (https://preemo.medium.com/squeeze-more-out-of-your-gpu-for-llm-inference-a-tutorial-on-accelerate-deepspeed-610fce3025fd)
Performance bottlenecks in deploying LLMs—a primer for ML researchers (https://preemo.medium.com/performance-bottlenecks-in-deploying-llms-a-primer-for-ml-researchers-c2b51c2084a8)
Inference using the pre-trained Alpaca-LoRA (https://www.mlexpert.io/machine-learning/tutorials/alpaca-and-llama-inference) (https://colab.research.google.com/drive/15VstUxU48CT3mRudFrj3FIv6Z4cIXnon?usp=sharing)
Optimizing your LLM in production (https://huggingface.co/blog/optimize-llm)
StreamingLLM: Efficient Streaming Language Models with Attention Sinks (https://github.com/mit-han-lab/streaming-llm) (https://arxiv.org/pdf/2309.17453.pdf)
S-LoRA: Serving Thousands of Concurrent LoRA Adapters (https://github.com/s-lora/s-lora)
- Recipe for Serving Thousands of Concurrent LoRA Adapters (https://lmsys.org/blog/2023-11-15-slora/)
DeepSparse by Neural Magic - Sparsity-aware deep learning inference runtime for CPUs (https://github.com/neuralmagic/deepsparse/tree/main)
SparseML by Neural Magic - an open-source model optimization toolkit that enables you to create inference-optimized sparse models using pruning, quantization, and distillation algorithms (https://github.com/neuralmagic/sparseml)
Marlin - Mixed Auto-Regressive Linear kernel, an extremely optimized FP16xINT4 matmul kernel aimed at LLM inference (https://github.com/IST-DASLab/marlin)
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning (https://github.com/datamllab/LongLM) (https://www.reddit.com/r/LocalLLaMA/comments/18x8g6c/llm_maybe_longlm_selfextend_llm_context_window/)
Flash-Decoding for long-context inference (https://pytorch.org/blog/flash-decoding/)
10 Ways To Run LLMs Locally And Which One Works Best For You (https://matilabs.ai/2024/02/07/run-llms-locally/)
Towards 100x Speedup: Full Stack Transformer Inference Optimization (https://yaofu.notion.site/Towards-100x-Speedup-Full-Stack-Transformer-Inference-Optimization-43124c3688e14cffaf2f1d6cbdf26c6c)
Deploy Deep Learning Models at Scale using NVIDIA Triton Inference Server (https://github.com/decodingml/articles-code/tree/main/articles/computer_vision/deploy_deep_learning_at_scale_nvidia_triton_server)
LMDeploy - a toolkit for compressing, deploying, and serving LLM (https://github.com/InternLM/lmdeploy)
LLM Inference Series: 5. Dissecting model performance (https://medium.com/@plienhar/llm-inference-series-5-dissecting-model-performance-6144aa93168f)
How to compute LLM embeddings 3X faster with model quantization - with ONNX model quantization / ONNX transformer optimization (https://medium.com/nixiesearch/how-to-compute-llm-embeddings-3x-faster-with-model-quantization-25523d9b4ce5)
A Hitchhiker’s Guide to Speculative Decoding (https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/)
Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) (https://lmsys.org/blog/2024-07-25-sglang-llama3/) (https://github.com/sgl-project/sglang)
Awesome Production Machine Learning - open source libraries that will help you deploy, monitor, version, scale, and secure your production machine learning (https://github.com/EthicalML/awesome-production-machine-learning)
NanoFlow - a throughput-oriented high-performance serving framework for LLMs (https://github.com/efeslab/Nanoflow) (https://arxiv.org/pdf/2408.12757)
GuideLLM - evaluating and optimizing the deployment of large language models (LLMs) (https://github.com/neuralmagic/guidellm)
LLM Compressor - create compressed models for faster inference with vLLM (https://github.com/vllm-project/llm-compressor) (https://neuralmagic.com/blog/llm-compressor-is-here-faster-inference-with-vllm/)
bitnet.cpp - Official inference framework for 1-bit LLMs (https://github.com/microsoft/BitNet)
STRING - a training-free method to improve effective context length of popular RoPE-based LLMs (https://github.com/HKUNLP/STRING)

🤖 Transformer / LLM Platform / Software

GPT4All - run open-source LLMs on your own computer (https://github.com/nomic-ai/gpt4all)
LLaMA-Factory - Efficiently Fine-Tune 100+ LLMs in WebUI (https://github.com/hiyouga/LLaMA-Factory)
Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs (https://github.com/facebookresearch/lingua)
bolt.diy (Previously oTToDev) - Prompt, run, edit, and deploy full-stack web applications using any LLM you want! (https://github.com/stackblitz-labs/bolt.diy)
WebUI - Run AI Agent in your browser (https://github.com/browser-use/web-ui)
Free LLM API resources (https://github.com/cheahjs/free-llm-api-resources)

🤖 Transformer / LLM Data Curator

Curating Trillion-Token Datasets: Introducing NVIDIA NeMo Data Curator (https://developer.nvidia.com/blog/curating-trillion-token-datasets-introducing-nemo-data-curator/)
ftfy: fixes text for you (https://github.com/rspeer/python-ftfy)
Cosmopedia: how to create large-scale synthetic data for pre-training (https://huggingface.co/blog/cosmopedia) (https://github.com/huggingface/cosmopedia)
DataTrove - a library to process, filter and deduplicate text data at a very large scale (https://github.com/huggingface/datatrove/)
Guidance - control how LLM output is structured (https://github.com/guidance-ai/guidance)
Large-scale Near-deduplication Behind BigCode (https://huggingface.co/blog/dedup)
Dolma Toolkit - curation of large datasets for (pre)-training ML models (https://github.com/allenai/dolma)
Distilabel - framework for synthetic data and AI feedback (https://github.com/argilla-io/distilabel)
LLM Decontaminator (https://github.com/lm-sys/llm-decontaminator)
Tutorial to demonstrate how to reproduce Zyda2 dataset, curated by Zyphra in collaboration with Nvidia using NeMo Curator (https://github.com/NVIDIA/NeMo-Curator/tree/main/tutorials/zyda2-tutorial) (https://www.zyphra.com/post/building-zyda-2)

🤖 Transformer / LLM Dataset

DBPedia (https://www.dbpedia.org/resources/individual/) (http://downloads.dbpedia.org/wiki-archive/dbpedia-version-2016-04.html) (http://downloads.dbpedia.org/2016-04/core/)
Common Crawl (https://commoncrawl.org/the-data/get-started/)
c4 - A colossal, cleaned version of Common Crawl's web crawl corpus (https://tensorflow.org/datasets/catalog/c4)
c4 processed version with five variants of the data: en, en.noclean, en.noblocklist, realnewslike, and multilingual (mC4). (https://huggingface.co/datasets/allenai/c4)
RedPajama-V2: An open dataset with 30 trillion tokens for training large language models (https://www.together.ai/blog/redpajama-data-v2) (https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2)
SlimPajama-627B - Extensively deduplicated, multi-corpora, open-source dataset for training LLM (https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama) (https://huggingface.co/datasets/cerebras/SlimPajama-627B)
Falcon RefinedWeb - An English large-scale dataset (5 trillion tokens ) for the pretraining of LLM, built through stringent filtering and extensive deduplication of CommonCrawl (https://huggingface.co/datasets/tiiuae/falcon-refinedweb) (https://arxiv.org/abs/2306.01116)
Instruction tuning datasets to train (text and multi-modal) chat-based LLMs (GPT-4, ChatGPT, LLaMA, Alpaca) (https://github.com/yaodongC/awesome-instruction-dataset)
Python-Code-23k-ShareGPT (https://huggingface.co/datasets/ajibawa-2023/Python-Code-23k-ShareGPT)
UltraFeedback Binarized - A pre-processed version of the UltraFeedback dataset and was used to train Zephyr-7Β-β (https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)
[Blog post] FineWeb: decanting the web for the finest text data at scale (https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1)
- FineWeb: 15T tokens (44TB disk space) of cleaned and deduplicated english web data from CommonCrawl, for LLM pretraining (https://huggingface.co/datasets/HuggingFaceFW/fineweb)
- FineWeb-Edu: 1.3T tokens of educational web pages filtered from 🍷 FineWeb dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
- FineWeb-Edu-score-2: 5.4T tokens of educational web pages filtered from 🍷 FineWeb dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2)
Dolma - Used to train OLMo on 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials, with quality filtering, fuzzy deduplication (https://blog.allenai.org/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64) (https://huggingface.co/datasets/allenai/dolma)
TinyStories - synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary (https://huggingface.co/datasets/roneneldan/TinyStories)
GenQA - over 10M cleaned and deduplicated instruction samples generated from a handful of carefully designed prompts (https://huggingface.co/datasets/tomg-group-umd/GenQA)
Persona Hub - Scaling Synthetic Data Creation with 1,000,000,000 Personas (https://github.com/tencent-ailab/persona-hub) (https://huggingface.co/datasets/proj-persona/PersonaHub)
FinePersonas - detailed personas for creating customized, realistic synthetic data (https://huggingface.co/datasets/argilla/FinePersonas-v0.1)
APIGen Function-Calling Datasets (https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k)
The Tome - Compiled from 9 publicly available datasets, curated and designed for training LLMs with a focus on instruction following (https://huggingface.co/datasets/arcee-ai/The-Tome)
distilabel-intel-orca-dpo-pairs (for preference tuning) (https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs)
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens (https://blog.salesforceairesearch.com/mint-1t/) (https://huggingface.co/collections/mlfoundations/mint-1t-6690216ca4d0df7e518dde1c)
Zyda-2 is a 5-trillion token dataset composed of filtered and cross-deduplicated DCLM, FineWeb-Edu, Zyda-1, and Dolma v1.7's Common Crawl portion (https://huggingface.co/datasets/Zyphra/Zyda-2) (https://www.zyphra.com/post/building-zyda-2)

🤖 Transformer / LLM Sample Applications

🤖 MLOps

🤖 Q&A

🤖 Discussion on LLM Padding / Formatting Function

🤖 Merging weights with quantized model

🤖 Merge / Fusion / MoE

🤖 Prompt Engineering / Instructions

🤖 Agent

🤖 LLM - Misc

How Do Language Models put Attention Weights over Long Context? (https://yaofu.notion.site/How-Do-Language-Models-put-Attention-Weights-over-Long-Context-10250219d5ce42e8b465087c383a034e) (https://github.com/FranxYao/Long-Context-Data-Engineering)
AI Watermarking 101: Tools and Techniques (https://huggingface.co/blog/watermarking)
GGUF, the long way around (https://vickiboykis.com/2024/02/28/gguf-the-long-way-around/)
A Comprehensive Guide to Modeling Techniques in Mixed-Integer Linear Programming - Convert ideas into mathematical expressions to solve operations research problems (https://towardsdatascience.com/a-comprehensive-guide-to-modeling-techniques-in-mixed-integer-linear-programming-3e96cc1bc03d)
Mastering ML Configurations by leveraging OmegaConf and Hydra (https://decodingml.substack.com/p/mastering-ml-configurations-by-leveraging)
UltraChat - example training script with Accelerator (https://github.com/thunlp/UltraChat/blob/main/train/train_legacy/train.py)
Using LESS Data to Tune Models (https://www.cs.princeton.edu/~smalladi/blog/2024/04/04/dataselection/)
Techniques for training large neural networks - Data parallelism, Pipeline parallelism, Tensor parallelism, Mixture-of-Experts (MoE), and other memory saving designs (https://openai.com/research/techniques-for-training-large-neural-networks)
- Large Scale Transformer model training with Tensor Parallel (TP) (https://pytorch.org/tutorials/intermediate/TP_tutorial.html)
YaFSDP - a Sharded Data Parallelism framework (https://github.com/yandex/YaFSDP) (https://habr.com/ru/companies/yandex/articles/817509/)
Best Embedding Model — OpenAI / Cohere / Google / E5 / BGE - An In-depth Comparison of Multilingual Embedding Models (https://medium.com/@lars.chr.wiik/best-embedding-model-openai-cohere-google-e5-bge-931bfa1962dc) topic dataset (https://github.com/LarsChrWiik/lars_datasets/tree/main/topics_dataset_50)
Training and Finetuning Embedding Models with Sentence Transformers v3 (https://huggingface.co/blog/train-sentence-transformers)
What can LLMs never do? (https://www.strangeloopcanon.com/p/what-can-llms-never-do)
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/)
Uncensor any LLM with abliteration (https://huggingface.co/blog/mlabonne/abliteration) (https://colab.research.google.com/drive/1VYm3hOcvCpbGiqKZb141gJwjdmmCcVpR?usp=sharing)
- abliterator.py (https://github.com/FailSpy/abliterator)
- TransformerLens - A library for mechanistic interpretability of GPT-style language models (https://github.com/TransformerLensOrg/TransformerLens) (https://transformerlensorg.github.io/TransformerLens/)
Training a 70B model from scratch: open-source tools, evaluation datasets, and learnings (https://imbue.com/research/70b-intro/)
- From bare metal to a 70B model: infrastructure set-up and scripts (https://imbue.com/research/70b-infrastructure/)
- Ensuring accurate model evaluations: open-sourced, cleaned datasets for models that reason and code (https://imbue.com/research/70b-evals/)
Aleksa Gordić’s Post: Amazing list of techniques for improving the stability of training large ML models (LLMs, diffusion, etc) (https://www.linkedin.com/feed/update/urn:li:activity:7215624025639645184/)
The AdEMAMix Optimizer: Better, Faster, Older. A simple modification of the Adam optimizer with a mixture of two Exponential Moving Average (EMA) (https://github.com/nanowell/AdEMAMix-Optimizer-Pytorch/)
Generating Human-level Text with Contrastive Search in Transformers (https://huggingface.co/blog/introducing-csearch)
- A Contrastive Framework for Neural Text Generation (https://github.com/yxuansu/SimCTG)
Open Source LLM Tools (https://huyenchip.com/llama-police)
- What I learned from looking at 900 most popular open source AI tools (https://huyenchip.com/2024/03/14/ai-oss.html)
Imagen - Pytorch (https://github.com/lucidrains/imagen-pytorch)
- MinImagen - A Minimal implementation of the Imagen text-to-image model(https://github.com/AssemblyAI-Community/MinImagen)
- How Imagen Actually Works (https://www.assemblyai.com/blog/how-imagen-actually-works/)
- MinImagen - Build Your Own Imagen Text-to-Image Model (https://www.assemblyai.com/blog/minimagen-build-your-own-imagen-text-to-image-model/)
How to Beat Proprietary LLMs With Smaller Open Source Models (https://www.aidancooper.co.uk/how-to-beat-proprietary-llms/)
A Guide to Structured Outputs Using Constrained Decoding (https://www.aidancooper.co.uk/constrained-decoding/)
The 6 Best LLM Tools To Run Models Locally (https://medium.com/@amosgyamfi/the-6-best-llm-tools-to-run-models-locally-eedd0f7c2bbd)
You can now train a 70b language model at home - Training LLMs with QLoRA + FSDP (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html) (https://github.com/AnswerDotAI/fsdp_qlora/tree/main)
Bugs in LLM Training - Gradient Accumulation Fix (https://unsloth.ai/blog/gradient)
- 🤗 Fixing Gradient Accumulation (https://huggingface.co/blog/gradient_accumulation)
SynthID Text - Apply watermarks and identify AI-generated content (https://huggingface.co/blog/synthid-text) (https://deepmind.google/technologies/synthid/)
How to Build ANYTHING You Imagine With DeepSeek-R1 (Zero Coding Required) (https://iamshobhitagarwal.medium.com/how-to-build-anything-you-imagine-with-deepseek-r1-zero-coding-required-free-0d582513e7ab) $$
- BOLT.diy: Build UNLIMITED Apps in Minutes 🚀 (No Code Required) (https://www.youtube.com/watch?v=QQw47XzcKRQ)
  - Form Setup Guide - NocoDB (https://www.youtube.com/watch?v=GEsFSadshpg)
  - Email Automation - N8n (https://www.youtube.com/watch?v=gofNLKlrFCQ)

🤖 Llama

🤖 Unsloth

🤖 Transformer Alternatives

Retentive Networks (RetNet) Explained: The much-awaited Transformers-killer is here (https://medium.com/ai-fusion-labs/retentive-networks-retnet-explained-the-much-awaited-transformers-killer-is-here-6c17e3e8add8)
Mamba Explained - The State Space Model taking on Transformers (https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html)
Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model (https://www.ai21.com/blog/announcing-jamba)
ModuleFormer: a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts (https://github.com/IBM/ModuleFormer)
JetMoE: Reaching LLaMA2 Performance with 0.1M Dollars (https://research.myshell.ai/jetmoe)
SUPRA: Scalable UPtraining for Recurrent Attention - uptrain a transformer to a linear RNN (https://github.com/TRI-ML/linear_open_lm) (https://huggingface.co/TRI-ML/mistral-supra)
📺 Understanding Mamba and State Space Models (https://www.youtube.com/watch?v=iskuX3Ak9Uk)

👍 Google AI/ML Use Cases

Diffusion Models

Similar repos

ElizaLo/Data-Science

Projects and awesome list for all Data Science fields

bishwaghimire/ai-learning-roadmaps

A complete, structured hub for learning Artificial Intelligence — covering AI, Machine Learning,…

tunguz/ML_Resources

GitHub Repo with various ML/AI/DS resources that I find useful

ashara12/awesome-deeplearning

A curated list of awesome Deep Learning tutorials, projects and communities.

DataExpert-io/ai-engineer-handbook

All the links, books, and creators you need to follow to stay up to date with AI!

ChristosChristofidis/awesome-deep-learning

A curated list of awesome Deep Learning tutorials, projects and communities.

yanshengjia/ml-road

Machine Learning and Agentic AI Resources, Practice and Research

ritchieng/the-incredible-pytorch

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more…

Contributors

peggy1502

canelkaradeniz

peggy1502/Amazing-Resources

README

Amazing Resources

👍 Courses / Tutorials

👍 Cheat Sheets

👍 AWS / SageMaker

📺 Videos

📚 Books

👍 Papers

📑 Articles

🖼️ CNN

↩️ RNN

⁉️ NLP

💏 NVIDIA Recommender Systems

💏 Collaborative Filtering / Recommender Systems

👫 Similarity Search / ANNS / Vector Indexing

🆎 Code Search

🌐 Search Engine / Information Retrieval

⬆️ Search Ranking

👫 Lucene / Solr / Elasticsearch / BM25

📑 General ML/DL Articles

Gradient / Momentum

Optimization / Outliers / Overfitting / Regularization / Imbalance Dataset

Regression

⏱️ Time Series

📊 EDA / Data Visualization

📘 Colab

🐍 Python

🗄️ Database / Storage

🏃 Reinforcement Learning

👍 Interesting and Fun

👍 GitHub Repositories

👍 Kaggle

📘 DeepNote

👍 Blogs

👍 Company Tech Blogs

🔟 Maths

👍 Datasets

👍 Synthetic Data

🛠️ Utilities / Tools

👍 Job / Interview / DS Portfolio

💰 Salary Negotiation

👍 System Design

🆎 Algorithms / Technical Coding

📺 Videos for Algorithms / Technical Coding / Interview Prep

🔢 Bit Hacks

👍 Google foobar

🔦 PyTorch-Related

🔦 PyTorch-Related Discussions

⏩ fast.ai

📚 NVIDIA eBooks

🧞 Genomic Data Science

🤖 Transformer Architecture / Anatomy / Guide

🤖 Transformer / Attention / LLM Visualization

🤖 Transformer Maths

🤖 Transformer Libraries

🤖 Transformer Toolkit / Techniques / Methods

🤖 RAG

🤖 Lightning AI ⚡⚡⚡

🤖 LLM Leaderboard

🤖 LLM Evaluation

🤖 Transformer Models / Timeline

🤖 Transformer / LLM Inference / Deployment

🤖 Transformer / LLM Platform / Software

🤖 Transformer / LLM Data Curator

🤖 Transformer / LLM Dataset

🤖 Transformer / LLM Sample Applications

🤖 MLOps

🤖 Q&A

🤖 Discussion on LLM Padding / Formatting Function

🤖 Merging weights with quantized model

🤖 Merge / Fusion / MoE

🤖 Prompt Engineering / Instructions

🤖 Agent

🤖 LLM - Misc

🤖 Llama

🤖 Unsloth

🤖 Transformer Alternatives

👍 Google AI/ML Use Cases

Diffusion Models