Photo by Lars Kienle on Unsplash
THE REASONS TO BENCHMARK SEMICONDUCTOR PRODUCTS
Benchmarking a product is one of the most common evaluation processes, and from software to hardware, benchmarking is extensively used.
In the semiconductor industry, benchmarking is mainly used to evaluate products against their predecessors and also competitors. CPU and GPU get benchmarked more often than any other type of silicon product, and the reason is the heavy dependence on day-to-day computing on these two types of processing units.
Benchmarking: Capturing technical characteristics and comparing them against other reference products to showcase where the new product stands.
Comparing one semiconductor product with another or old one is one of the reasons to benchmark. Benchmarking enables several key data points and makes the decision-making process easier for end customers. In many cases, it also pushes the competitors to launch new products.
Evaluation: Benchmarking provides a path to unravel all the internal features of a new semiconductor product. Evaluating products using different workloads presents a clear technical picture of device capabilities.
Performance: The majority of the semiconductor products get designed to balance power and performance, while several are also focused purely on peak performance without considering the power consumption. Either way, executing the benchmarking workload on a silicon product allows capturing of detailed performance characteristics.
Characterization: Power, performance, voltage, and time are few technical data points that enable characterization. Benchmarking tools are capable of capturing these details by stressing the product with different operating conditions. Such data point provides a way to capture the capabilities of a product over different settings.
Bugs: Stressing a product using different benchmarking workloads can reveal if there are bugs in the product. Bugs are captured based on whether the benchmarking criteria are leading to expected data as per the specification. If not, then designers and manufacturers can revisit the development stage to fix the issue.
Adaptability: Benchmarking also provides a path to capture how adaptive the semiconductor product is. It can be done by simple experiments wherein the product is stressed using benchmarking workloads under different temperature to voltage settings. Any failure or deviating results during such benchmarking can provide a way to capture and correct issues before mass production.
Benchmarking also reveals several data points to the buyers and empowers them with information about why a specific new product is better than the other. Relying on benchmarking process has become a norm in the computing industry. It is also why any new semiconductor product launch (CPU or GPU) comes loaded with benchmarking data.
With several new semiconductor products coming out in the marking and catering to different domains (wireless, sensor, computing, and many more), benchmarking presents a way to capture the true potential of the new product.
However, correctly executing a benchmarking process is critical, and any mistake can present a false impression about the product getting evaluated. Hence it is vital to benchmark a product correctly.
THE CORRECT WAY TO BENCHMARK SEMICONDUCTOR PRODUCTS
Benchmarking semiconductor products like XPU (and several others) is not an easy task. It requires detailed knowledge of internal features to ensure the workload used for benchmarking is correctly utilizing all the new embedded features.
A false benchmarking process can make or break a product, and it can also invite several questions on any previous product that used a similar benchmarking process. To correctly benchmark a product requires covering several unique points so that all the features get evaluated.
Mapping: The benchmarking world has several workloads. However, not all are designed and then tested by correctly mapping the software on top of the hardware. For correct benchmarking, it is critical to capture all the features that enable the correct overlay of the workload on top of the silicon product. Doing so ensures that the benchmarking workload can take benefits of all the internal architectural features before.
Architecture: Understanding different features and architectural optimization is a vital part of correctly benchmarking the products. There are generic benchmarking tools and workloads, but not all can take advantage of all the register level techniques to optimize the data flow. A good understanding (which also requires detailed documentation from the semiconductor company) of architecture is necessary before any benchmarking is executed. This also enables a fair comparison without overlooking any features.
Reference: The major goal of benchmarking is to showcase how good the new product is. To showcase such results require a reference, which can be a predecessor product from the same company or a competitor. Without a reference data point, there is no value in positive benchmarking results. Hence, having as many references benchmarking data points as possible is a good way to compare results.
Open: To drive fair benchmarking, open-sourcing the software (workloads) code can instill a high level of confidence in the results. The open process also allows code contribution, which can improve the workloads, and thus the benchmarking results will be more reliable than ever.
Data: Sharing as much benchmarking data as possible is also a good strategy. Peer review of the data points also improves the benchmarking process of future products. Historical benchmarking data points also drives contribution from data enthusiast and thus can help improve the benchmarking process and standardization.
Several tools and workloads are available to evaluate and benchmark a semiconductor product. However, the majority of these workloads/tools are written without 100% information about the internal features of any given product, which might lead to false-positive/negative benchmarking data points.
All this pushes the case for standardizing the benchmarking process so that any semiconductor product when compared against others (in the same domain), gets evaluated for set/standard data points. On top, as more complex XPUs and similar products (neuromorphic chips) come out in the market, standard benchmarking protocols will provide a way to correctly evaluate all the new technologies (and design solutions) that several old/emerging companies are launching.
Benchmarking is not a new process and has been around in the semiconductor industry for several decades, and it will be part of the semiconductor industry for decades to come. The only question is how fair the future benchmarking process will be.
