Mixed precision for bfloat16-pretrained models

[stas]

We started compiling a wiki of how different models were pre-trained, please add your knowledge there - thanks!

Model pre-training precision database: fp16, fp32, bf16

There is an emerging need to know how a given model was pre-trained: fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic since fp16 and bf16 numerical ranges don’t overlap too well. We are discussing adding a new field to models that will tell users how it was trained. Some papers don’t disclose how a model was trained. So …