NVIDIA Launches GenAI-Perf for Optimizing Generative AI Model Performance

NVIDIA has unveiled a new tool, GenAI-Perf, aimed at enhancing the performance measurement and optimization of generative AI models. According to the NVIDIA Technical Blog, this tool is incorporated into the latest release of NVIDIA Triton and is designed to aid machine learning engineers in finding the optimal balance between latency and throughput, especially crucial for large language models (LLMs).

Key Metrics for LLM Performance

When dealing with LLMs, performance metrics extend beyond traditional latency and throughput. Key metrics include:

• Time to first token: The time between when a request is sent and the receipt of the first response.
• Output token throughput: The number of output tokens generated per second.
• Inter-token latency: The time between intermediate responses divided by the number of generated tokens.

These metrics are essential for applications where quick and consistent performance is paramount, with time to first token often being the highest priority.

Introducing GenAI-Perf

GenAI-Perf is designed to accurately measure these specific metrics, helping users determine optimal configurations for peak performance and cost-effectiveness. The tool supports industry-standard datasets like OpenOrca and CNN_dailymail and facilitates standardized performance evaluations across various inference engines through an OpenAI-compatible API.

GenAI-Perf is intended to be the default benchmarking tool for all NVIDIA generative AI offerings, including NVIDIA NIM, NVIDIA Triton Inference Server, and NVIDIA TensorRT-LLM. This facilitates easy comparisons among different serving solutions that support the OpenAI-compatible API.

Supported Endpoints and Usage

Currently, GenAI-Perf supports three OpenAI endpoint APIs: Chat, Chat Completions, and Embeddings. As new model types emerge, additional endpoints will be introduced. GenAI-Perf is also open source, accepting community contributions.

To get started with GenAI-Perf, users can install the latest Triton Inference Server SDK container from NVIDIA GPU Cloud. Running the container and server involves specific commands tailored to the type of model being used, such as GPT2 for chat and chat-completion endpoints, and intfloat/e5-mistral-7b-instruct for embeddings.

Profiling and Results

For profiling OpenAI chat-compatible models, users can run specific commands to measure performance metrics such as request latency, output sequence length, and input sequence length. Sample results for GPT2 show metrics like:

• Request latency (ms): Average of 1679.30, with a minimum of 567.31 and a maximum of 2929.26.
• Output sequence length: Average of 453.43, ranging from 162 to 784.
• Output token throughput (per sec): 269.99.

Similarly, for profiling OpenAI embeddings-compatible models, users can generate a JSONL file with sample texts and run GenAI-Perf to obtain metrics such as request latency and request throughput.

Conclusion

GenAI-Perf provides a comprehensive solution for benchmarking generative AI models, offering insights into critical performance metrics and facilitating optimization. As an open-source tool, it allows for continuous improvement and adaptation to new model types and requirements.