Open-Source GenAI Models Built by AMD

SDS prunes PLMs in three steps—one-shot pruning, sparse regularization, and a second prune—achieving better weight distribution and outperforming SparseGPT and Wanda

Dual Learnable Ternarization (DLT) and Outlier-Friendly Feature Knowledge Distillation (OFF) handle outliers in weights and activations, enabling TernaryLLM to outperform prior low-bit methods in text generation and zero-shot tasks.

DL-QAT is a novel approach for quantization-aware training in large language models that combines weight decomposition and low-rank matrices to optimize quantized weights with minimal parameter changes.

Týr-the-Pruner is an end-to-end search-based global structural pruning framework for LLMs. It constructs a supernet via local pruning across sparsity ratios and uses an iterative prune-and-search strategy. It retains 97% of the dense model's performance while pruning 50% of Llama-3.1-70B's parameters.

An improved FFN (IFFN) module for vision transformers that uses AGeLU function and multiple instances to enhance non-linearity, reducing hidden dimensions and computational cost.

QT-ViT replaces softmax-based attention with second-order Taylor expansion, accelerating it via a fast approximation algorithm. It achieves superior performance without knowledge distillation or high-order attention residuals, outperforming previous models.

FDViT employs a flexible downsampling layer to reduce feature map sizes smoothly. Combined with a masked auto-encoder for training, it decreases redundant calculations and information loss.

A domain shift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images.

A reward-guided approach that directly learns negative embeddings through gradient descent. The negative embeddings exhibit strong generalization capabilities and can be seamlessly adaptable to T2I and T2V models.

MSWA improves SWA by using diverse window sizes across Transformer heads and layers, enhancing performance and efficiency. It assigns smaller windows to shallow layers and larger ones to deep layers. 

Using an enhanced post-training approach based on intermediate layer distillation and optimized layer selection, our models dramatically reduce KV-cache requirements — without compromising quality.     

A continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token,

increasing the representation capacity of the latent space.

An efficient post-training approach to convert MHA models to multi-head latent attention (MLA) using knowledge distillation.