跳到内容

测试回调

AbstractStopAndGoCallback

基类：ABC, BaseInterruptedVsContinuousCallback

用于停止和继续回调的抽象基类，用于比较暂停训练之前和恢复训练之后的元数据。

此基类提供了实用方法，以帮助简化停止和继续比较。

提供的方法
  • init：使用给定的模式初始化回调。
  • get_metadata：应重写以从训练器和 pl_module 获取元数据的抽象方法。
默认行为
  • 在停止模式下，在 on_validation_epoch_end 时获取和比较元数据。
  • 在继续模式下，在 on_train_epoch_start 时获取和保存元数据。

如有必要，覆盖这些行为。

源代码位于 bionemo/testing/testing_callbacks.py 中 
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
class AbstractStopAndGoCallback(ABC, BaseInterruptedVsContinuousCallback):
    """Abstract base class for stop-and-go callback to compare metadata before pausing and after resuming training.

    This base class provides utility methods to help streamline stop and go comparison.

    Provided methods:
        - __init__: initializes the callback with the given mode.
        - get_metadata: abstract method that should be overridden to get metadata from the trainer and pl_module.

    Default behaviors:
        - in stop mode, metadata is gotten and compared on_validation_epoch_end.
        - in go mode, metadata is gotten and saved on_train_epoch_start.

    Override these behaviors if necessary.
    """

    def __init__(self, mode: Mode = Mode.STOP):
        """Initialize StopAndGoCallback.

        Args:
            mode (str, optional): Mode to run in. Must be either Mode.STOP or Mode.RESUME. Defaults to Mode.STOP.

        Notes:
            User must override get_metadata to get metadata from the trainer and pl_module.
        """
        if mode not in [Mode.STOP, Mode.RESUME]:
            raise ValueError(f"mode must be 'stop' or 'go', got {mode}")
        self.mode = mode
        super().__init__()

    @abstractmethod
    def get_metadata(self, trainer: Trainer, pl_module: LightningModule) -> Any:
        """Get metadata from trainer and pl_module."""
        raise NotImplementedError

    def on_train_epoch_start(self, trainer: Trainer, pl_module: LightningModule):  # noqa: D102
        if self.mode == Mode.RESUME:
            self.data = self.get_metadata(trainer, pl_module)

    def on_validation_epoch_end(self, trainer: Trainer, pl_module: LightningModule):  # noqa: D102
        if not trainer.sanity_checking and self.mode == Mode.STOP:
            self.data = self.get_metadata(trainer, pl_module)

__init__(mode=Mode.STOP)

初始化 StopAndGoCallback。

参数

名称 类型 描述 默认值
mode str

运行模式。必须是 Mode.STOP 或 Mode.RESUME。默认为 Mode.STOP。

 STOP
注释

用户必须覆盖 get_metadata 以从训练器和 pl_module 获取元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
237
238
239
240
241
242
243
244
245
246
247
248
249
def __init__(self, mode: Mode = Mode.STOP):
    """Initialize StopAndGoCallback.

    Args:
        mode (str, optional): Mode to run in. Must be either Mode.STOP or Mode.RESUME. Defaults to Mode.STOP.

    Notes:
        User must override get_metadata to get metadata from the trainer and pl_module.
    """
    if mode not in [Mode.STOP, Mode.RESUME]:
        raise ValueError(f"mode must be 'stop' or 'go', got {mode}")
    self.mode = mode
    super().__init__()

get_metadata(trainer, pl_module) abstractmethod

从训练器和 pl_module 获取元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
251
252
253
254
@abstractmethod
def get_metadata(self, trainer: Trainer, pl_module: LightningModule) -> Any:
    """Get metadata from trainer and pl_module."""
    raise NotImplementedError

BaseInterruptedVsContinuousCallback

基类：Callback, CallbackMethods, IOMixin

用于序列化停止和继续回调的基类，用于比较连续训练与中断训练。

此类通过扩展回调并将数据收集到 self.data 属性中来使用。然后，在连续训练和中断训练之间比较此数据。

有关可用的回调方法，请参阅 nemo.lightning.megatron_parallel.CallbackMethods。

源代码位于 bionemo/testing/testing_callbacks.py 中 
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
class BaseInterruptedVsContinuousCallback(Callback, CallbackMethods, io.IOMixin):
    """Base class for serializable stop-and-go callback to compare continuous to interrupted training.

    This class is used by extending a callback and collecting data into the `self.data` attribute. This data is then
    compared between continuous and interrupted training.

    See nemo.lightning.megatron_parallel.CallbackMethods for the available callback methods.
    """

    def __init__(self):
        """Initializes the callback."""
        self.data = []

    def __deepcopy__(self, memo):
        """Don't actually attempt to copy this data when this callback is being serialized."""
        ...

__deepcopy__(memo)

当序列化此回调时，实际上不要尝试复制此数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
77
78
79
def __deepcopy__(self, memo):
    """Don't actually attempt to copy this data when this callback is being serialized."""
    ...

__init__()

初始化回调。

源代码位于 bionemo/testing/testing_callbacks.py 中 
73
74
75
def __init__(self):
    """Initializes the callback."""
    self.data = []

ConsumedSamplesCallback

基类：BaseInterruptedVsContinuousCallback

停止和继续回调，用于检查暂停训练之前和恢复训练之后消耗的样本。

源代码位于 bionemo/testing/testing_callbacks.py 中 
102
103
104
105
106
107
108
109
110
111
112
113
class ConsumedSamplesCallback(BaseInterruptedVsContinuousCallback):
    """Stop-and-go callback to check consumed samples before pausing and after resuming training."""

    def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
        """Get consumed samples as metadata."""
        if step.trainer.training:
            data_sampler = step.trainer.datamodule.data_sampler
            consumed_samples = data_sampler.compute_consumed_samples(
                step.trainer.global_step - step.trainer.datamodule.init_global_step
            )
            self.data.append(np.array(consumed_samples))
        return step

on_megatron_step_start(step)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
105
106
107
108
109
110
111
112
113
def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
    """Get consumed samples as metadata."""
    if step.trainer.training:
        data_sampler = step.trainer.datamodule.data_sampler
        consumed_samples = data_sampler.compute_consumed_samples(
            step.trainer.global_step - step.trainer.datamodule.init_global_step
        )
        self.data.append(np.array(consumed_samples))
    return step

GlobalStepStateCallback

基类：BaseInterruptedVsContinuousCallback

用于 global_step 的停止和继续回调，用于暂停训练之前和恢复训练之后。

源代码位于 bionemo/testing/testing_callbacks.py 中 
92
93
94
95
96
97
98
99
class GlobalStepStateCallback(BaseInterruptedVsContinuousCallback):
    """Stop-and-go callback for global_step before pausing and after resuming training."""

    def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
        """Get learning rate as metadata."""
        if step.trainer.training:
            self.data.append(np.array(step.trainer.global_step))
        return step

on_megatron_step_start(step)

获取学习率作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
95
96
97
98
99
def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
    """Get learning rate as metadata."""
    if step.trainer.training:
        self.data.append(np.array(step.trainer.global_step))
    return step

LearningRateCallback

基类：BaseInterruptedVsContinuousCallback

用于学习率的停止和继续回调，用于暂停训练之前和恢复训练之后。

源代码位于 bionemo/testing/testing_callbacks.py 中 
82
83
84
85
86
87
88
89
class LearningRateCallback(BaseInterruptedVsContinuousCallback):
    """Stop-and-go callback for learning rate before pausing and after resuming training."""

    def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
        """Get learning rate as metadata."""
        if step.trainer.training:
            self.data.append(np.array(step.trainer.optimizers[0].param_groups[0]["lr"]))
        return step

on_megatron_step_start(step)

获取学习率作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
85
86
87
88
89
def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
    """Get learning rate as metadata."""
    if step.trainer.training:
        self.data.append(np.array(step.trainer.optimizers[0].param_groups[0]["lr"]))
    return step

OptimizerStateCallback

基类：BaseInterruptedVsContinuousCallback

停止和继续回调，用于检查暂停训练之前和恢复训练之后的优化器状态。

源代码位于 bionemo/testing/testing_callbacks.py 中 
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
class OptimizerStateCallback(BaseInterruptedVsContinuousCallback):
    """Stop-and-go callback to check optimizer states before pausing and after resuming training."""

    def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
        """Get optimizer states as metadata."""
        if step.trainer.training:
            self.data.append(
                recursive_detach(
                    [
                        optimizer.mcore_optimizer.optimizer.state_dict()["state"]
                        for optimizer in step.trainer.optimizers
                    ]
                )
            )
        return step

on_megatron_step_start(step)

获取优化器状态作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
207
208
209
210
211
212
213
214
215
216
217
218
def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
    """Get optimizer states as metadata."""
    if step.trainer.training:
        self.data.append(
            recursive_detach(
                [
                    optimizer.mcore_optimizer.optimizer.state_dict()["state"]
                    for optimizer in step.trainer.optimizers
                ]
            )
        )
    return step

SignalAfterGivenStepCallback

基类：Callback, CallbackMethods

一个回调，在定义的步骤向当前进程发出给定的信号。

将此回调用于基于 pytest 的停止和继续测试。

源代码位于 bionemo/testing/testing_callbacks.py 中 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
class SignalAfterGivenStepCallback(Callback, CallbackMethods):
    """A callback that emits a given signal to the current process at the defined step.

    Use this callback for pytest based Stop and go tests.
    """

    def __init__(self, stop_step: int, signal_: signal.Signals = signal.SIGUSR2):
        """Initializes the callback with the given stop_step."""
        self.stop_step = stop_step
        self.signal = signal_

    def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
        """Stop training if the global step is greater than or equal to the stop_step."""
        if step.trainer.global_step >= self.stop_step:
            os.kill(os.getpid(), self.signal)
        return step

__init__(stop_step, signal_=signal.SIGUSR2)

使用给定的 stop_step 初始化回调。

源代码位于 bionemo/testing/testing_callbacks.py 中 
52
53
54
55
def __init__(self, stop_step: int, signal_: signal.Signals = signal.SIGUSR2):
    """Initializes the callback with the given stop_step."""
    self.stop_step = stop_step
    self.signal = signal_

on_megatron_step_start(step)

如果全局步骤大于或等于 stop_step，则停止训练。

源代码位于 bionemo/testing/testing_callbacks.py 中 
57
58
59
60
61
def on_megatron_step_start(self, step: MegatronStep) -> MegatronStep:
    """Stop training if the global step is greater than or equal to the stop_step."""
    if step.trainer.global_step >= self.stop_step:
        os.kill(os.getpid(), self.signal)
    return step

StopAfterValidEpochEndCallback

基类：Callback, CallbackMethods

一个在验证 epoch 后停止训练的回调。

将此回调用于基于 pytest 的停止和继续测试。

源代码位于 bionemo/testing/testing_callbacks.py 中 
34
35
36
37
38
39
40
41
42
43
class StopAfterValidEpochEndCallback(Callback, CallbackMethods):
    """A callback that stops training after the validation epoch.

    Use this callback for pytest based Stop and go tests.
    """

    def on_validation_epoch_end(self, trainer: Trainer, pl_module: LightningModule):  # noqa: D102
        if trainer.sanity_checking:
            return
        trainer.should_stop = True

TrainInputCallback

基类：BaseInterruptedVsContinuousCallback

收集训练输入样本以进行比较。

源代码位于 bionemo/testing/testing_callbacks.py 中 
116
117
118
119
120
121
122
123
124
125
126
127
128
class TrainInputCallback(BaseInterruptedVsContinuousCallback):
    """Collect training input samples for comparison."""

    def on_megatron_microbatch_end(
        self,
        step: MegatronStep,
        batch: DataT,
        forward_callback: "MegatronLossReduction",
        output: Any,
    ) -> None:
        """Get consumed samples as metadata."""
        if step.trainer.training:
            self.data.append(recursive_detach(batch))

on_megatron_microbatch_end(step, batch, forward_callback, output)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
119
120
121
122
123
124
125
126
127
128
def on_megatron_microbatch_end(
    self,
    step: MegatronStep,
    batch: DataT,
    forward_callback: "MegatronLossReduction",
    output: Any,
) -> None:
    """Get consumed samples as metadata."""
    if step.trainer.training:
        self.data.append(recursive_detach(batch))

TrainLossCallback

基类：BaseInterruptedVsContinuousCallback

收集训练损失样本以进行比较。

源代码位于 bionemo/testing/testing_callbacks.py 中 
176
177
178
179
180
181
182
183
184
185
186
187
class TrainLossCallback(BaseInterruptedVsContinuousCallback):
    """Collect training loss samples for comparison."""

    def on_megatron_step_end(
        self,
        step: MegatronStep,
        microbatch_outputs: List[Any],
        reduced: Optional[Union[torch.Tensor, Dict[str, torch.Tensor]]] = None,
    ) -> None:
        """Get consumed samples as metadata."""
        if step.trainer.training:
            self.data.append(recursive_detach(reduced))

on_megatron_step_end(step, microbatch_outputs, reduced=None)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
179
180
181
182
183
184
185
186
187
def on_megatron_step_end(
    self,
    step: MegatronStep,
    microbatch_outputs: List[Any],
    reduced: Optional[Union[torch.Tensor, Dict[str, torch.Tensor]]] = None,
) -> None:
    """Get consumed samples as metadata."""
    if step.trainer.training:
        self.data.append(recursive_detach(reduced))

TrainOutputCallback

基类：BaseInterruptedVsContinuousCallback

收集训练输出样本以进行比较。

源代码位于 bionemo/testing/testing_callbacks.py 中 
146
147
148
149
150
151
152
153
154
155
156
157
158
class TrainOutputCallback(BaseInterruptedVsContinuousCallback):
    """Collect training output samples for comparison."""

    def on_megatron_microbatch_end(
        self,
        step: MegatronStep,
        batch: DataT,
        forward_callback: "MegatronLossReduction",
        output: Any,
    ) -> None:
        """Get consumed samples as metadata."""
        if step.trainer.training:
            self.data.append(recursive_detach(output))

on_megatron_microbatch_end(step, batch, forward_callback, output)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
149
150
151
152
153
154
155
156
157
158
def on_megatron_microbatch_end(
    self,
    step: MegatronStep,
    batch: DataT,
    forward_callback: "MegatronLossReduction",
    output: Any,
) -> None:
    """Get consumed samples as metadata."""
    if step.trainer.training:
        self.data.append(recursive_detach(output))

TrainValInitConsumedSamplesStopAndGoCallback

基类：AbstractStopAndGoCallback

停止和继续回调，用于检查暂停训练之前和恢复训练之后消耗的样本。

这是当前唯一不符合直接比较连续训练和中断训练的新模式的回调，因为数据加载器不跟踪检查点恢复之前和之后的 consumed_samples。

源代码位于 bionemo/testing/testing_callbacks.py 中 
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
class TrainValInitConsumedSamplesStopAndGoCallback(AbstractStopAndGoCallback):
    """Stop-and-go callback to check consumed samples before pausing and after resuming training.

    This is currently the only callback that doesn't fit with the new pattern of directly comparing continuous and
    interrupted training, since the dataloaders don't track their consumed_samples before and after checkpoint
    resumption.
    """

    @override
    def get_metadata(self, trainer: Trainer, pl_module: LightningModule) -> Any:
        """Get consumed samples as metadata."""
        # return trainer.datamodule.state_dict()["consumed_samples"]  # TODO why state_dict can be empty despite working lines below
        train_data_sampler: MegatronPretrainingSampler = trainer.train_dataloader.batch_sampler
        val_data_sampler: MegatronPretrainingSampler = trainer.val_dataloaders.batch_sampler
        return train_data_sampler.consumed_samples, val_data_sampler.consumed_samples

get_metadata(trainer, pl_module)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
273
274
275
276
277
278
279
@override
def get_metadata(self, trainer: Trainer, pl_module: LightningModule) -> Any:
    """Get consumed samples as metadata."""
    # return trainer.datamodule.state_dict()["consumed_samples"]  # TODO why state_dict can be empty despite working lines below
    train_data_sampler: MegatronPretrainingSampler = trainer.train_dataloader.batch_sampler
    val_data_sampler: MegatronPretrainingSampler = trainer.val_dataloaders.batch_sampler
    return train_data_sampler.consumed_samples, val_data_sampler.consumed_samples

ValidInputCallback

基类：BaseInterruptedVsContinuousCallback

收集验证输入样本以进行比较。

源代码位于 bionemo/testing/testing_callbacks.py 中 
131
132
133
134
135
136
137
138
139
140
141
142
143
class ValidInputCallback(BaseInterruptedVsContinuousCallback):
    """Collect validation input samples for comparison."""

    def on_megatron_microbatch_end(
        self,
        step: MegatronStep,
        batch: DataT,
        forward_callback: "MegatronLossReduction",
        output: Any,
    ) -> None:
        """Get consumed samples as metadata."""
        if step.trainer.validating:
            self.data.append(recursive_detach(batch))

on_megatron_microbatch_end(step, batch, forward_callback, output)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
134
135
136
137
138
139
140
141
142
143
def on_megatron_microbatch_end(
    self,
    step: MegatronStep,
    batch: DataT,
    forward_callback: "MegatronLossReduction",
    output: Any,
) -> None:
    """Get consumed samples as metadata."""
    if step.trainer.validating:
        self.data.append(recursive_detach(batch))

ValidLossCallback

基类：BaseInterruptedVsContinuousCallback

收集训练损失样本以进行比较。

源代码位于 bionemo/testing/testing_callbacks.py 中 
190
191
192
193
194
195
196
197
198
199
200
201
class ValidLossCallback(BaseInterruptedVsContinuousCallback):
    """Collect training loss samples for comparison."""

    def on_megatron_step_end(
        self,
        step: MegatronStep,
        microbatch_outputs: List[Any],
        reduced: Optional[Union[torch.Tensor, Dict[str, torch.Tensor]]] = None,
    ) -> None:
        """Get consumed samples as metadata."""
        if step.trainer.validating:
            self.data.append(recursive_detach(reduced))

on_megatron_step_end(step, microbatch_outputs, reduced=None)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
193
194
195
196
197
198
199
200
201
def on_megatron_step_end(
    self,
    step: MegatronStep,
    microbatch_outputs: List[Any],
    reduced: Optional[Union[torch.Tensor, Dict[str, torch.Tensor]]] = None,
) -> None:
    """Get consumed samples as metadata."""
    if step.trainer.validating:
        self.data.append(recursive_detach(reduced))

ValidOutputCallback

基类：BaseInterruptedVsContinuousCallback

收集验证输出样本以进行比较。

源代码位于 bionemo/testing/testing_callbacks.py 中 
161
162
163
164
165
166
167
168
169
170
171
172
173
class ValidOutputCallback(BaseInterruptedVsContinuousCallback):
    """Collect validation output samples for comparison."""

    def on_megatron_microbatch_end(
        self,
        step: MegatronStep,
        batch: DataT,
        forward_callback: "MegatronLossReduction",
        output: Any,
    ) -> None:
        """Get consumed samples as metadata."""
        if step.trainer.validating:
            self.data.append(recursive_detach(output))

on_megatron_microbatch_end(step, batch, forward_callback, output)

获取消耗的样本作为元数据。

源代码位于 bionemo/testing/testing_callbacks.py 中 
164
165
166
167
168
169
170
171
172
173
def on_megatron_microbatch_end(
    self,
    step: MegatronStep,
    batch: DataT,
    forward_callback: "MegatronLossReduction",
    output: Any,
) -> None:
    """Get consumed samples as metadata."""
    if step.trainer.validating:
        self.data.append(recursive_detach(output))