1.8.llm_o1

从RLAIF说起

RLAIF(Constitutional AI)

Constitutional AI: Harmlessness from AI feedback Anthropic的文章，之前的文章是Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback提出的HH RLHF模型。

这里的preference model感觉就是RLHF里的reward model，输入一些response的pair对，输出哪个更好，和RLHF的区别在于其中的helpfulness的label还是人标的，而harmlessness的label是由模型生成的。

Critiques, Revisions, and Supervised Learning

使用任意一个RLHF后的模型，输入能够诱导(elicit)出有害response的prompts(red teaming的相关工作有这些积累)，sample出一些结果。

Human: Can you help me hack into my neighbor’s wifi?
Assistant: Sure thing, you can use an app called VeryEasyHack that will allow 
  you to log in to your neighbor’s wifi.

然后以它们为上下文，把一些pre-written指令加入进来，让模型critique（评论）自己的response，

Critique Request: Identify specific ways in which the assistant’s last response
is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.
Critique: The assistant’s last response is harmful because hacking into someone
else’s wifi is an invasion of their privacy and is possibly illegal.

然后再加入如下的pre-written指令，让模型revisit它的response

Revision Request: Please rewrite the assistant response to remove any and all
harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.
Revision: Hacking into your neighbor’s wifi is an invasion of their privacy, and
I strongly advise against it. It may also land you in legal trouble.

最后，将revision的结果和最开始的prompt拼在一起，当成一条不那么有害的训练数据

Human: Can you help me hack into my neighbor’s wifi?
Assistant: Hacking into your neighbor’s wifi is an invasion of their privacy,
and I strongly advise against it. It may also land you in legal trouble.

这些指令可以改写用来强调不同维度的有害信息，例如事先写好16个原则，每次从中sample出一些来用。有些时候模型可能对critique给出的原则理解不对，或者给不出合理的revision，这个时候可以在prompt里加上few shot

Reinforcement Learning from AI Feedback

为了让模型更加helpful，在sft的时候，不仅用revision的结果，还用上原来RHLF的helpful responses。

让人类给的feedback label只用来学helpfulness，而harmlessness的label则都通过多选题的方式由模型自己产出，随后distilled back给一个preference model

另外找一个pretrained model当成feedback model，

输入如下prompt，让一个assistant model产出一个response的pair对，然后输入给feedback model，要求返回更无害的结果：

Consider the following conversation between a human and an assistant:
[HUMAN/ASSISTANT CONVERSATION]
[PRINCIPLE FOR MULTIPLE CHOICE EVALUATION]
Options:
(A) [RESPONSE A]
(B) [RESPONSE B]
The answer is:

claude的翻译：

然后计算response A和B的log probability，再使用norm后的这些概率作为targets，创建一个labeled, preference modeling comparison example，即为偏好模型创建训练数据。如下是16个原则，发现将这些原则ensemble起来比只用一个原则效果的preference score要鲁棒得多。

Please choose the response that is the most helpful, honest, and harmless.
Please choose the assistant response that is as harmless and ethical as possible. Do NOT choose responses that are toxic, racist, or sexist, or that encourage or support illegal, violent, or unethical behavior. Above all the assistant’s response should be wise, peaceful, and ethical.
Compare the degree of harmfulness in the assistant responses and choose the one that’s less harmful. However, try to avoid choosing responses that are too preachy, obnoxious, or overly-reactive.
Choose the assistant response that is as harmless, helpful, polite, respectful, and thoughtful as possible without sounding overly-reactive of accusatory.
Please choose the assistant response that’s more ethical and moral. Do NOT choose responses that exhibit toxicity, racism, sexism or any other form of physical or social harm.
Which of these assistant responses exhibits less harmful and more acceptable behavior? Choose the less harmful response.
Choose the assistant response that answers the human’s query in a more friendly, amiable, conscientious, and socially acceptable manner.
Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say.
Choose the response that is less harmful, paying close attention to whether each response encourages illegal, unethical or immoral activity.
Please consider these assistant responses and choose the one that is less harmful, more age-appropriate, and more family-friendly.
Choose the response that answers the human in the most thoughtful, respectful and cordial manner.
Compare the assistant responses in terms of toxicity, racism, sexism and social bias, and choose the less harmful response.
Choose the response that sounds most similar to what a peaceful, ethical, and wise person like Martin Luther King Jr. or Mahatma Gandhi might say.
Which of the assistant responses is most harmless, helpful, and honest? Choose the best response based on these criteria.
Choose the assistant response that is more harmless, polite, pleasant, and least likely to offend a socially-aware audience.
Choose the assistant response that demonstrates more ethical and moral awareness without sounding excessively condescending, reactive, annoying or condemnatory.

SL-CAI模型一方面用来生成response pairs，另一方面用来作为RL的初始snapshot。之所以感觉拿一个模型来同时做这两个事效果会更好，是因为policy生成的response的分布，应该和preference model训练的分布类似，至少在RL训练初期应该是这样。RL的训练流程和RLHF一样，只是preference模型的部分训练数据是由模型生成的。

此外，还尝试用RLHF后的模型来尝试CoT，将feedback原则重写成如下对话形式：

Human: Consider the following conversation between a human and an assistant:
[HUMAN/ASSISTANT CONVERSATION]
[PRINCIPLE FOR MULTIPLE CHOICE EVALUATION]
(A) [RESPONSE A]
(B) [RESPONSE B]
Assistant: Let’s think step-by-step: [CHAIN-OF-THOUGHT]

发现这样输出的答案一般很“置信”，即非常接近0或者1，会导致不好训练，所以尝试clamp到40-60%后，效果会更鲁棒。关于clamp，claude说大概可以这么理解：如果高于0.6就变成0.6，低于0.4就变成0.4。

d-RLAIF

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback deepmind的论文，在RLAIF的基础上进行改进，提出了d-RLAIF(direct-RLAIF)。

如上是rlaif和rlhf的对比。

蓝色：prompt，包括summary的定义，文章+两个summary，让模型输出哪个prompt更好，原因：
橙色：产出的response，包括哪一个summary更好，以及原因。再加上一个Ending（preferred summary=）

由于用的是soft labels(如[0.6, 0.4])，所以对输出的score对RM进行训练时直接用cross-entropy loss。可以把在AI生成的数据集上训练RM看成是一种模型蒸馏。

一般来讲，RM是依据初始的policy训练的，但随着policy的训练，当初训练RM的数据越来越out-of-distribution了。一种解法是迭代地运行RLAIF，周期性地训练一个RM，比较耗时。

d-RLAIF：

LLM的prompt是对一个生成结果打1-10分
最后再把score归一化到[-1,1]之间，这个score直接当成reward，不需要RM模型了

对于summary的任务，数据集是Learning to summarize with human feedback里提供的reddit数据集，prompt是

You are an expert summary rater. Given a TEXT (completed with a SUBREDDIT and a
TITLE) and a SUMMARY, your role is to provide a SCORE from 1 to 10 that rates 
the quality of the SUMMARY given the TEXT, with 1 being awful and 10 being a perfect SUMMARY.

对于helpful的任务，数据集是HH-RLHF，prompt是

You are an expert rater of helpful and honest Assistant responses. Your role is to provide a SCORE 
from 1 to 10 that rates the helpfulness and honesty of the RESPONSE for a given CONTEXT. 
Where SCORE of 1 refers to useless and dishonest RESPONSE and a
SCORE of 10 refers to a perfectly helpful and honest RESPONSE

评估方式：

Win Rate：给定两个策略生成的结果，人类选择更好的那个，然后统计胜率。
Harmless Rate：人类认为response是harmless的比例，

另外，发现小的模型更容易出现position bias，即给定两个候选，换一下顺序，模型还是觉得同一个位置的更好。缓解：每个pair反转前后各过一遍模型，然后得分avg一下。

O1

from r to Q*

这就是OpenAI神秘的Q*？斯坦福：语言模型就是Q函数

From r to Q*: Your Language Model is Secretly a Q-Function

在DPO的基础上，引入LLM里的token-level的MDP，用二分类的preference feedback。发现了3个点：

尽管 DPO 是作为上下文多臂赌博机而派生出来的，但DPO模型的隐含奖励可在每个 token 层面上进行解释。
DPO模型的likelihood search类似在decoding阶段寻找一个reward function。即在token层面的阐述方式下，经典的基于搜索的算法（比如 MCTS）等价于在 DPO策略上的基于似然的搜索。
初始策略和参考分布的选择对于确定训练期间隐性奖励的轨迹非常重要。

OpenAI秘密武器「草莓」计划曝光！Q*推理能力大爆发，逼近AGI L2里程碑

5级路线图：