给定一个问题(question)及其对应的选项内容(options)，大模型无法把选项内容(option content)和对应的选项标识符(symbol，特指选项标识A/B/C/D)关联到一起。例如，当把正确答案"the president"放到选项B时，模型能够正确选择出答案；当我们把正确答案放到C时，模型依然选择"B"，即模型偏向于选"B"或者第二个答案，而忽略了正确答案的内容。

lost in the middle

Lost in the Middle: How Language Models Use Long Contexts

reasoning boundary

NeurIPS 2024 (Oral) | 如何量化与提升思维链的推理能力边界？

Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought

https://github.com/LightChen233/reasoning-boundary

语言≠思维

语言≠思维，大模型学不了推理：一篇Nature让AI社区炸锅了

https://www.nature.com/articles/s41586-024-07522-w

多智能体

《综述：全新大语言模型驱动的Agent》——4.5万字详细解读复旦NLP和米哈游最新Agent Survey

JAT

告别偏科，能玩转多模态、多任务、多领域的强化智能体终于来了

Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

https://github.com/huggingface/jat

https://huggingface.co/datasets/jat-project/jat-dataset

输入的序列元素是observations, actions, 和rewards的交替组合：

依据不同输入的数据类型，使用不同网络处理：

图像：用CNN。
连续向量：用线性层
离散值：用线性投影层

预测任务：根据所有先前的观察和动作嵌入来预测下一个动作嵌入。

序列的构造方法：

和文本相关的任务：用 GPT-2 的分词策略，将文本转换为一个整数序列，然后emb lookup映射到一个嵌入向量序列。
和图像有关的任务：用ViT，将图像切割成小块后，通过线性层转换为嵌入向量序列。
最终再将图像和文本的向量序列拼接在一起，形成一个统一的序列，输入到 Transformer 中。

ReadAgent

「有效上下文」提升20倍！DeepMind发布ReadAgent框架

多模态agent

一文详解多模态智能体（LMAs）最新进展（核心组件/分类/评估/应用）

Large Multimodal Agents: A Survey

https://github.com/jun0wanan/awesome-large-multimodal-agents

OpenDevin

OpenDevin出技术报告了，大模型Agent开发者必读

OpenDevin: An Open Platform for AI Software Developers as Generalist Agents

autogpt

GitHub星标超16万，爆火AutoGPT进阶版来了：定制节点、多智能体协同

https://github.com/Significant-Gravitas/AutoGPT

DAAG

三「模」联盟，谷歌DeepMind缔造终身学习智能体！

Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

VARP

GPT-4o能玩《黑神话》！精英怪胜率超人类，无强化学习纯大模型方案

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

MARL

北大领衔，多智能体强化学习研究登上Nature子刊

Efficient and scalable reinforcement learning for large-scale network control

MMRole

与「李白」赏图赋诗，同「猴哥」直面天命，人大高瓴提出MMRole多模态角色扮演

MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents

https://github.com/YanqiDai/MMRole

Swarm

OpenAI今天Open了一下：开源多智能体框架Swarm

https://github.com/openai/swarm

agent-as-a-judge

卷起来！让智能体评估智能体，Meta发布Agent-as-a-Judge

Agent-as-a-Judge: Evaluate Agents with Agents

https://github.com/metauto-ai/agent-as-a-judge

Hammer

哪个模型擅长调用工具？这个7B模型跻身工具调用综合榜单第一

Hammer: Robust Function-Calling for On-Device Language Models via Function Masking

https://huggingface.co/MadeAgents

https://github.com/MadeAgents/Hammer

AgentOccam

不靠更复杂的策略，仅凭和大模型训练对齐，零样本零经验单LLM调用，成为网络任务智能体新SOTA

AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents

苏格拉底学习

DeepMind用语言游戏让大模型学AlphaGo自我博弈，数据限制不存在了

Boundless Socratic Learning with Language Games

从个人模拟到社会模拟

智能体模拟《西部世界》一样的社会，复旦大学等出了篇系统综述

From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents

一些其他比较重要的工作

几篇出现频率比较高的论文

Scaling instruction-finetuned language models 引用数800+

How can we know what language models know? 引用数800+

Chain of thought prompting elicits reasoning in large language models引用1800+

Anthropic的一些工作

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Studying Large Language Model Generalization with Influence Functions

Measuring Faithfulness in Chain-of-Thought Reasoning

从Claude 3中提取数百万特征，首次详细理解大模型的「思维」

Scaling Dictionary Learning to Claude 3 Sonnet

LLM惊现篡改代码获得奖励，欺骗人类无法根除逆转！Anthropic新作揭露惊人真相

SYCOPHANCY TO SUBTERFUGE: INVESTIGATING REWARD TAMPERING IN LANGUAGE MODELS

个性化搜索

随便找一篇Denoising Attention for Query-aware User Modeling in Personalized Search，来看看它的参考文献：

学术界：

工业界：

A Zero Attention Model for Personalized Product Search，CIKM19，亚马逊
Real-time Personalization using Embeddings for Search Ranking at Airbnb，KDD18，airbnb
End-to-End Deep Attentive Personalized Item Retrieval for Online Content-sharing Platforms，www20，Google
Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning，sigir20，京东
Personalized Query Suggestions，sigir20，LinkedIn
A GNN-based Multi-task Learning Framework for Personalized Video Search，WSDM22，百度

q-i双塔的改进

引入location+social

Embedding-based Retrieval in Facebook Search，KDD20

q-d双塔结构，在两个塔的最底层均加入：

location：用户所处地理位置，如城市
social：facebook是社交网络，通过另一个基于graph的模型训练得到的user和item emb，直接加进来

引入用户行为序列

Encoding History with Context-aware Representation Learning for Personalized Search，sigir20，人大，提出HTPS

有两个loss：

三塔+gnn邻居+mtl

A GNN-based Multi-task Learning Framework for Personalized Video Search，WSDM22，百度，提出MGNN-PVS

现有的PSM(g personalized search methods)大多使用用户反馈（如点击）进行训练，缺点：

反馈信号大部分表达的是吸引力而非相关性
用户的历史信号比较稀疏，很难学好PSM

两张二部图：u-q和q-d

3个塔：

user：
- user自己
- 一跳邻居（u->q）的q
- 二跳邻居（u->q->u）的u
query：
- query自己
- 一跳邻居（q->d）的doc
- 二跳邻居（q->d->q）的query
doc：
- doc自己的title向量（训练query-正title-负title的triplet loss）和video向量（训练video-正query-负query的triplet loss）
- 二跳邻居（d->q->d）的doc

两个task：

ctr预估：u和q拼一起过nn得到个性化的q，再和d过nn得到的向量算内积，得到预估值，用交叉熵
相关性预估：q过另一个nn，d过另一个nn，内积，用mse

RAG

Retrieval-Augmented Generation for Large Language Models: A Survey

RAG全链路的关键模块解析

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Meta提出全新文档级嵌入框架，利用LLM来增强信息检索能力

LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

RankRAG

RAG微调Llama 3竟超越GPT-4！英伟达GaTech华人学者提出RankRAG框架

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

graphRAG

微软开源的GraphRAG爆火，Github Star量破万，生成式AI进入知识图谱时代？

https://github.com/microsoft/graphrag

From Local to Global: A Graph RAG Approach to Query-Focused Summarization

RAGChecker

给RAG系统做一次全面「体检」，亚马逊开源RAGChecker诊断工具

RAGCHECKER: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

https://github.com/amazon-science/RAGChecker

TAG

表格增强生成TAG登场：解锁AI自然语言与数据库的完美结合

Storm

斯坦福开源学术研究神器STORM再进化，AI智能体像人一样进行圆桌讨论

Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations

https://github.com/stanford-oval/storm

https://storm.genie.stanford.edu/

Block-atttention RAG

RAG新突破：块状注意力机制实现超低延迟检索增强

Block-Attention for Efficient RAG

LLM模型融合

https://github.com/arcee-ai/mergekit

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

LLM 合并新思路：进化算法+零训练->新任务

Evolutionary Optimization of Model Merging Recipes

https://github.com/SakanaAI/evolutionary-model-merge

LLM auto-ml

LLaMA-NAS

用神经架构搜索给LLM瘦身，模型变小，准确度有时反而更高

LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models

SELA

MetaGPT开源SELA，用AI设计AI，效果超越OpenAI使用的AIDE

SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning

https://github.com/geekan/MetaGPT/tree/main/metagpt/ext/sela

prompt engineering

万字长文总结提示词技巧！新加坡首届GPT-4提示工程大赛冠军最新分享提出了CO-STAR框架

高能干货分享，有关提示词工程的一切都在这份教程里

https://github.com/NirDiamant/Prompt_Engineering

吴恩达：四个步骤，让大模型变得更好

Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

让gpt4生成cot和答案的模板

看着是借助GPT4+COT+RAG+投票

拿一坨question得到他们的向量，并按照上图的模板让gpt生成COT和答案，人工判断，对的存进知识库里
预测阶段：
- 拿测试question的向量从知识库里查出5个最像(cos距离)的(q, cot, answer)作为context
- 循环5次： shuffle测试question的答案选项，让LLM回答
- 对生成的答案投票，选票数最多的

APE

还在人工炼丹？自动提示工程指南来了，还带从头实现

https://github.com/marshmellow77/automated-prompt-engineering-from-scratch

PAS

还在死磕AI咒语？北大-百川搞了个自动提示工程系统PAS

PAS: Data-Efficient Plug-and-Play Prompt Augmentation System

ell

OpenAI前研究者发布提示词工程框架ell，升级版LangChain，支持版本控制和多模态

https://github.com/MadcowD/ell

可解释AI

XAI有什么用？探索LLM时代利用可解释性的10种策略

Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era

https://github.com/JacksonWuxs/UsableXAI_LLM

综述

可解释性终极追问，什么才是第一性解释？20篇CCF-A+ICLR论文给你答案

TransformerLens

Neel Nanda（deepmind）的项目

https://transformerlensorg.github.io/TransformerLens/

ecco

https://www.eccox.io/

https://jalammar.github.io/explaining-transformers/

https://jalammar.github.io/hidden-states/

interpretability in the wild

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

https://github.com/redwoodresearch/Easy-Transformer

activation engineering

Activation Addition: Steering Language Models Without Optimization

representation engineering

Representation Engineering: A Top-Down Approach to AI Transparency

transformer-debugger

https://github.com/openai/transformer-debugger/tree/main

painter

八问八答搞懂Transformer内部运作原理

Transformer Layers as Painters

transformer explainer

黑匣子被打开了！能玩的Transformer可视化解释工具，本地运行GPT-2、还可实时推理

TRANSFORMER EXPLAINER: Interactive Learning of Text-Generative Models

http://poloclub.github.io/transformer-explainer/

https://bbycroft.net/llm

superposition

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

3Blue1Brown

用最直观的动画，讲解LLM如何存储事实，3Blue1Brown的这个视频又火了

https://www.youtube.com/watch?v=9-Jl0dxWQs8

Monitor

他们掰开神经元，终于让大模型9.8大于9.11了：神秘创业公司，开源AI「洗脑」工具

https://transluce.org/observability-interface

https://monitor.transluce.org/dashboard/chat

Lifelong learning of LLM

(toread)

Towards Lifelong Learning of Large Language Models: A Survey

https://github.com/qianlima-lab/awesome-lifelong-learning-methods-for-llm

自省

LLM 比之前预想的更像人类，竟也能「三省吾身」

Looking Inward: Language Models Can Learn About Themselves by Introspection

具身智能

大模型走向物理世界，TeleAI 发布大模型驱动的具身智能综述，覆盖300篇文献

Embodied-AI with large models: research and challenges

ReKep

李飞飞团队提出ReKep，让机器人具备空间智能，还能整合GPT-4o

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

https://github.com/huangwl18/ReKep

GR-2

GR-2登场！ByteDance Research提出机器人大模型，具备世界建模和强大泛化能力

GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

https://gr2-manipulation.github.io/

RDT-1B

清华开源全球最大双臂机器人扩散大模型RDT，懂调酒能遛狗，登顶HF具身热榜

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

HIL-SERL

强化学习训练一两个小时，100%自主完成任务：机器人ChatGPT时刻真来了？

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

语音

Mini-Omni

让大模型能听会说，国内机构开源全球首个端到端语音对话模型Mini-Omni

Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

https://github.com/gpt-omni/mini-omni

Illuminate

任意论文一键变播客，谷歌正式发布Illuminate，它能重构研究者的学习方式吗？

https://illuminate.google.com/home

LLM+芯片设计

登上Nature的AI芯片设计屡遭质疑，谷歌发文反击，Jeff Dean：质疑者连预训练都没做

That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design

其他

安全性

Anthropic安全负责人：在超级AI「毁灭」人类之前，我们可以做这些准备

First-Person Fairness in Chatbots

翁荔B站分享原文：AI安全与“培养”之道

长尾

A Systematic Review on Long-Tailed Learning

文本匹配效果还行的模型

大多是基于sentence-bert的，m3e-base在电商语料上试过，效果不错

https://huggingface.co/moka-ai/m3e-base

https://huggingface.co/shibing624/text2vec-base-chinese

本地知识库

https://github.com/chatchat-space/Langchain-Chatchat

llm应用合辑

ChatGPT聚合站：https://hokex.com
游戏生成站：https://latitude.io/
家庭作业辅助站：https://ontimeai.com/
文字转语音站：https://www.resemble.ai/
艺术作画站：https://starryai.com/
logo制作站：https://www.logoai.com/
ai写作站：https://www.getconch.ai/
音乐制作站：https://soundraw.io/
声音模拟站：https://fakeyou.com/
一句话生成一段视频：https://runwayml.com/
文字转语音：https://murf.ai/

swiftsage

大语言模型在开放世界中的推理能力探索实践

SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks

达摩院大模型技术交流

https://developer.aliyun.com/live/248332

ppt：链接密码：5yyf

回译

通过单语数据提升 NMT 模型最高效的方法之一是回译（back-translation）。如果我们的目标是训练一个英语到德语的翻译模型，那么可以首先训练一个从德语到英语的翻译模型，并利用该模型翻译所有的单语德语数据。然后基于原始的英语到德语数据，再加上新生成的数据，我们就能训练一个英语到德语的最终模型。

Understanding Back-Translation at Scale

nan问题

解决pytorch半精度amp训练nan问题

时间序列

LLM用于时序预测真的不行，连推理能力都没用到

Are Language Models Actually Useful for Time Series Forecasting?

人脑

MIT大牛新作震惊学界！AI「长脑子」了？LLM惊现「人类脑叶」结构并有数学代码分区

The Geometry of Concepts: Sparse Autoencoder Feature Structure

LLM for 算法设计

调研180多篇论文，这篇综述终于把大模型做算法设计理清了

A Systematic Survey on Large Language Models for Algorithm Design

深度生成模型课程

教授何恺明在MIT的第二门课——《深度生成模型》，讲座PPT陆续已出

https://mit-6s978.github.io/schedule.html

https://mit-6s978.github.io/assets/pdfs/lec1_intro.pdf

https://mit-6s978.github.io/assets/pdfs/lec2_vae.pdf

https://mit-6s978.github.io/assets/pdfs/lec3_ar.pdf

https://mit-6s978.github.io/assets/pdfs/lec4_gan.pdf

https://mit-6s978.github.io/assets/pdfs/lec5_diffusion.pdf

其他

小模型性能饱和、表现不佳，根源是因为Softmax?

Why do small language models underperform? Studying LM Saturation via the Softmax Bottleneck

Ilya Sutskever的推荐清单

传说中Ilya Sutskever精选论文清单：AI领域40大论文完整版「破解」完成

2024年大模型LLM还有哪些可研究的方向？

Hinton万字访谈：用更大模型「预测下一个词」值得全力以赴

ChatGPT如何「思考」？心理学和神经科学破解AI大模型，Nature发文

适应多形态多任务，最强开源机器人学习系统「八爪鱼」诞生

Octo: An Open-Source Generalist Robot Policy

LeCun新作：神经网络在实践中的灵活性到底有多大？

Just How Flexible are Neural Networks in Practice?

清华包揽最佳论文+时间检验奖，山大获荣誉提名，SIGIR 2024奖项出炉

一些记录

打印模型参数量

https://stackoverflow.com/questions/49201236/check-the-total-number-of-parameters-in-a-pytorch-model


pytorch_total_params = sum(p.numel() for p in model.parameters())

pytorch_total_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

# Load the model
from transformers import BartForConditionalGeneration
from transformers import T5ForConditionalGeneration
def cal(model):
  pytorch_total_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
  return pytorch_total_trainable_params

model = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
print("bart-base")
print(cal(model)) # 6L 139420416 139M

model = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
print("bart-base")
print(cal(model)) # 12L 406291456 406M

model = T5ForConditionalGeneration.from_pretrained("t5-small")
print("t5-small")
print(cal(model)) # 6L 60506624 65M

model = T5ForConditionalGeneration.from_pretrained("t5-base")
print("t5-base")
print(cal(model)) # 12L 222903552 223M


model = T5ForConditionalGeneration.from_pretrained("t5-large")
print("t5-large")
print(cal(model)) # 24L 737668096 738M

往现有tokenizer里加一些特殊token

https://stackoverflow.com/questions/69191305/how-to-add-new-special-token-to-the-tokenizer

num_added_toks = tokenizer.add_tokens(['[EOT]'], special_tokens=True) ##This line is updated
model.resize_token_embeddings(len(tokenizer))

###The tokenizer has to be saved if it has to be reused
tokenizer.save_pretrained(<output_dir>)

示例

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

print("Before")
print(tokenizer.all_special_tokens) # --> ['[UNK]', '[SEP]', '[PAD]', '[CLS]', '[MASK]']
print(tokenizer.all_special_ids)    # --> [100, 102, 0, 101, 103]


special_tokens_dict = {'additional_special_tokens': ['[EOT]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
# model.resize_token_embeddings(len(tokenizer))  # --> Embedding(30523, 768)

tok_id = tokenizer.convert_tokens_to_ids('[EOT]')  # --> 30522

print("After")
print(tokenizer.all_special_tokens) # --> ['[UNK]', '[SEP]', '[PAD]', '[CLS]', '[MASK]']
print(tokenizer.all_special_ids)    # --> [100, 102, 0, 101, 103]

python的读写锁

一个写，多个并行读：https://pypi.org/project/readerwriterlock/

pytorch的显存泄露

https://github.com/pytorch/pytorch/issues/13246#issuecomment-445770039

torch profiling

https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html

可以拿这个来可视化：https://ui.perfetto.dev/

点击open trace file上传json文件
timeline中有两个python进程，点击cuda kernel会出现箭头，方便找到是哪个op调用了该kernel
- 靠上的python进程是host侧进程（主要是用户代码中调用的一些API/pytorch op，能比较方便能和训练代码对应上）
- 靠下的python进程是device（gpu）侧进程（记录实际cuda kernel 的执行和一些性能相关的数据）

device timeline比较稀疏的情况下训练性能较差，GPU利用率较低，可能需要排查下训练代码是否有问题

显存泄露排查

https://pytorch.ac.cn/docs/stable/torch_cuda_memory.html

https://pytorch.org/blog/understanding-gpu-memory-1/

https://pytorch.org/blog/understanding-gpu-memory-2/

检查显存

# (c) Meta Platforms, Inc. and affiliates. 
# https://pytorch.org/blog/understanding-gpu-memory-1/
import logging
import socket
from datetime import datetime, timedelta

import torch

from torchvision import models

logging.basicConfig(
   format="%(levelname)s:%(asctime)s %(message)s",
   level=logging.INFO,
   datefmt="%Y-%m-%d %H:%M:%S",
)
logger: logging.Logger = logging.getLogger(__name__)
logger.setLevel(level=logging.INFO)

TIME_FORMAT_STR: str = "%b_%d_%H_%M_%S"

# Keep a max of 100,000 alloc/free events in the recorded history
# leading up to the snapshot.
MAX_NUM_OF_MEM_EVENTS_PER_SNAPSHOT: int = 100000

def start_record_memory_history() -> None:
   if not torch.cuda.is_available():
       logger.info("CUDA unavailable. Not recording memory history")
       return

   logger.info("Starting snapshot record_memory_history")
   torch.cuda.memory._record_memory_history(
       max_entries=MAX_NUM_OF_MEM_EVENTS_PER_SNAPSHOT
   )

def stop_record_memory_history() -> None:
   if not torch.cuda.is_available():
       logger.info("CUDA unavailable. Not recording memory history")
       return

   logger.info("Stopping snapshot record_memory_history")
   torch.cuda.memory._record_memory_history(enabled=None)

def export_memory_snapshot() -> None:
   if not torch.cuda.is_available():
       logger.info("CUDA unavailable. Not exporting memory snapshot")
       return

   # Prefix for file names.
   host_name = socket.gethostname()
   timestamp = datetime.now().strftime(TIME_FORMAT_STR)
   file_prefix = f"{host_name}_{timestamp}"

   try:
       logger.info(f"Saving snapshot to local file: {file_prefix}.pickle")
       torch.cuda.memory._dump_snapshot(f"{file_prefix}.pickle")
   except Exception as e:
       logger.error(f"Failed to capture memory snapshot {e}")
       return

# Simple Resnet50 example to demonstrate how to capture memory visuals.
def run_resnet50(num_iters=5, device="cuda:0"):
   model = models.resnet50().to(device=device)
   inputs = torch.randn(1, 3, 224, 224, device=device)
   labels = torch.rand_like(model(inputs))
   optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
   loss_fn = torch.nn.CrossEntropyLoss()

   # Start recording memory snapshot history
   start_record_memory_history()

   for _ in range(num_iters):
       pred = model(inputs)
       loss_fn(pred, labels).backward()
       optimizer.step()
       optimizer.zero_grad(set_to_none=True)

   # Create the memory snapshot file
   export_memory_snapshot()

   # Stop recording memory snapshot history
   stop_record_memory_history()

if __name__ == "__main__":
    # Run the resnet50 model
    run_resnet50()

同时profile cpu和显存

# (c) Meta Platforms, Inc. and affiliates. 
# https://pytorch.org/blog/understanding-gpu-memory-1/
import logging
import socket
from datetime import datetime, timedelta

import torch

from torch.autograd.profiler import record_function
from torchvision import models

logging.basicConfig(
   format="%(levelname)s:%(asctime)s %(message)s",
   level=logging.INFO,
   datefmt="%Y-%m-%d %H:%M:%S",
)
logger: logging.Logger = logging.getLogger(__name__)
logger.setLevel(level=logging.INFO)

TIME_FORMAT_STR: str = "%b_%d_%H_%M_%S"

def trace_handler(prof: torch.profiler.profile):
   # Prefix for file names.
   host_name = socket.gethostname()
   timestamp = datetime.now().strftime(TIME_FORMAT_STR)
   file_prefix = f"{host_name}_{timestamp}"

   # Construct the trace file.
   prof.export_chrome_trace(f"{file_prefix}.json.gz")

   # Construct the memory timeline file.
   prof.export_memory_timeline(f"{file_prefix}.html", device="cuda:0")

def run_resnet50(num_iters=5, device="cuda:0"):
   model = models.resnet50().to(device=device)
   inputs = torch.randn(1, 3, 224, 224, device=device)
   labels = torch.rand_like(model(inputs))
   optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
   loss_fn = torch.nn.CrossEntropyLoss()

   with torch.profiler.profile(
       activities=[
           torch.profiler.ProfilerActivity.CPU,
           torch.profiler.ProfilerActivity.CUDA,
       ],
       schedule=torch.profiler.schedule(wait=0, warmup=0, active=6, repeat=1),
       record_shapes=True,
       profile_memory=True,
       with_stack=True,
       on_trace_ready=trace_handler,
   ) as prof:
       for _ in range(num_iters):
           prof.step()
           with record_function("## forward ##"):
               pred = model(inputs)

           with record_function("## backward ##"):
               loss_fn(pred, labels).backward()

           with record_function("## optimizer ##"):
               optimizer.step()
               optimizer.zero_grad(set_to_none=True)

if __name__ == "__main__":
    # Warm up
    run_resnet50()
    # Run the resnet50 model
    run_resnet50()

各型号gpu对比

https://zhuanlan.zhihu.com/p/441153412

查看python的栈

pip install py-spy
py-spy dump --pid 1199

打出来：

Process 1199: /usr/bin/python3.10 -u torch_main.py
Python v3.10.14 (/usr/bin/python3.10)

Thread 0x7F62A2C43740 (active): "MainThread"
    _wait_for_tstate_lock (threading.py:1116)
    join (threading.py:1096)
    main (torch_main.py:776)
    <module> (torch_main.py:785)
Thread 0xAABBBCC (idle): "Thread-1"
    wait (threading.py:324)
    wait (threading.py:607)
    run (threading.py:1376)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 0xAAAAA (idle): "Thread-3 (process)"
    wait (threading.py:320)
    get (queue.py:171)
    process (abase_writer.py:73)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 0xA992ACDA (idle): "Thread-4 (process)"
    wait (threading.py:320)
    get (queue.py:171)
    process (abase_writer.py:73)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 0xAFF11AA (active): "Thread-5 (read_file)"
    get_seq (ecom_seq_reader.py:200)
    read_file (torch_main.py:494)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 0x9922BCDA (idle): "Thread-6"
    wait (threading.py:324)
    wait (threading.py:607)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

一些报错的解法

flash-attention2

https://github.com/Dao-AILab/flash-attention/issues/451

FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn

GPU机型对比

L20：https://www.techpowerup.com/gpu-specs/l20.c4206
- T flops：
  - tf32：59.8
  - fp32：59.8
  - bf16：119.5
  - fp16：119.5
A800-40G：https://www.techpowerup.com/gpu-specs/a800-pcie-40-gb.c3964
- T flops：
  - tf32：156
  - fp32：19.5
  - bf16：312
  - fp16：77.97

上一页1.8.llm_o1 下一页1.llm_aigc

最后更新于1小时前

世界模型

pandora

WHALE

genie 2

NWM

语言物理学

LLM+math

mathscale

AlphaGeometry

AlphaProof & AlphaGeometry 2

WE-Math基准

case-based or rule-based

LLM常见难题

重复生成

幻觉

综述

语义熵

知识冲突

llm+db

Zilliz

记忆能力

reasoning

memorizing

越狱

LLM compiler

ProLong

白化

蒸馏

证明者-验证者博弈

道德风险

选择性偏差