As Artificial Intelligence (AI) continues to shift from cloud-based operations to on-device applications, consumers are often left wondering how to determine the performance of AI-powered apps on different devices. This knowledge could significantly impact the efficiency of tasks, potentially saving valuable time. MLCommons, an industry group known for its AI-related hardware benchmarking standards, aims to simplify this process with the introduction of performance benchmarks for consumer PCs, also known as “client systems”.
MLPerf Client: A New Working Group for AI Benchmarks
MLCommons recently announced the establishment of a new working group, MLPerf Client, dedicated to creating AI benchmarks for desktops, laptops, and workstations running on various operating systems including Windows and Linux. The group promises that these benchmarks will be “scenario-driven”, focusing on real end-user use cases and incorporating community feedback.
The first benchmark from MLPerf Client will concentrate on text-generating models, specifically Meta’s Llama 2. This model has already been integrated into MLCommons’ other benchmarking suites for datacenter hardware. Meta has also collaborated with Qualcomm and Microsoft to optimize Llama 2 for Windows, which will greatly benefit devices running on this operating system.
MLPerf Client Working Group Members
The MLPerf Client working group comprises several industry giants including AMD, Arm, Asus, Dell, Intel, Lenovo, Microsoft, Nvidia, and Qualcomm. However, Apple is notably absent from the group and is also not a member of MLCommons. This absence is not entirely surprising given that a Microsoft engineering director co-chairs the MLPerf Client group. Consequently, any AI benchmarks developed by MLPerf Client will not be tested on Apple devices in the near future.
Despite this, it will be interesting to see the benchmarks and tools that emerge from MLPerf Client. Given the increasing prevalence of AI, these metrics could play a significant role in future device-buying decisions. Ideally, the MLPerf Client benchmarks will resemble the many PC build comparison tools available online, providing an indication of the AI performance one can expect from a specific machine. With the participation of Qualcomm and Arm, both heavily invested in the mobile device ecosystem, these benchmarks may even expand to cover phones and tablets in the future.
