Hao Wu

Ongoing research training transformer models at scale

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

distributed-embeddings is a library for building large embedding based models in Tensorflow 2.

NeMo: a toolkit for conversational AI