Are laptop battery life benchmarks outdated?

By: Raghav Kapoor

Should we use MobileMark 2007, 3DMark2006, or a completely different approach?

  As we move towards mobile computing the first thing that comes to our mind is battery life. Consumers are now demanding portable devices such as netbooks, smartbooks, MIDs, etc. where the focus is on all day on the move computing. Users use these devices to connect to the internet, e-mailing, chating, video conferencing, social networking, listening music, editing documents, etc. So, the sole purpose is to have a device that can perform these tasks for the maximum time possible, on a single charge.

Whenever a user goes out to buy a netbook/notebook, he takes a look at the company claims regarding the battery life of that particular model. This claim is made by testing the netbook/notebook using a software benchmark. The standard in use today, is the MobileMark benchmark, created by the Business Applications Performance Corp. (BAPCo), an industry consortium whose members include Intel Corp., Dell Inc., Hewlett-Packard Co., Advanced Micro Devices Inc. and others. Critics have often charged companies of using these benchmarks deceptively.

Rob Enderle, an analyst says, "Everyone in the industry knows this benchmark is wildly optimistic and that the actual battery life you'll get is often less than half what MobileMark suggests, this is because MobileMark measures battery life much like you might measure gas mileage if you started the car, put it in neutral, and coasted down a long hill.”

The latest MobileMark 2007 report measures laptop battery life under three scenarios: reading a document, watching a DVD movie, and performing a mix of productivity tasks, such as reading and composing documents, editing photos, and encoding Flash videos. Rather than using an average time based on all three measures, BAPCo designates its third scenario as the way most people use their laptop.

This method of analyzing battery performance has been criticized by many including AMD. It says that – First, the productivity test assumes that the notebook is idle 90% or more of the time. Secondly, MobileMark's "productivity" scenario assumes that users, when active, are using only software such as Adobe Photoshop or Microsoft Office. They don't test usage of music or video applications such as iTunes or Windows Media Player, games or Web browsing. The test also assumes that Wi-Fi is turned off (which is not the real case now-a-days). Third, MobileMark 2007 allows PC vendors to set their laptop screen brightness at the lowest possible setting i.e. 60-nits which is actually really dim (according to an informal poll at Neowin.net, less than 15% of users use their screens at such low brightness levels).

AMD also complained that MobileMark essentially discriminates against its chipsets because they are more graphically powerful than Intel's, is all for reforming MobileMark. In a blog entitled "There has to be a better way," AMD suggests turning the widely used performance benchmark, 3DMark06, into "an active battery life test" that it argues would be a more accurate measure of average battery life.

However, BAPCo defended its MobileMark benchmarks by saying that "The content of BAPCo benchmarks are vigorously debated and cooperatively developed by BAPCo members according to a long and rigorous process. As is the case with all BAPCo benchmarks, MobileMark 2007 was approved by BAPCo according to a democratic voting process similar to ones used by most industry work groups."

Talking about the vendors, despite all the criticism, they are ready to tout the battery life from the Productivity test as their overall MobileMark score.

Jeff Chu, Segment Marketing Manager, Mobile Computing, ARM has in a blog stated, the key requirements for benchmarks moving forward, these are summed below:

·         The battery performance should be tied to specific usage models, as done in the mobile phone market with talk time and standby time. Consumers should be able to mix and match the results according to their usage and derive the battery time.

·         Be transportable across different operating systems esp. with the availability of large number of operating systems like Windows XP, Windows Vista, Windows 7, Linux, Android, etc.

·         Be transportable across chipsets and architectures.

·         Based on realistic assumptions i.e. don’t calculate battery performance with 20% screen brightness and wireless switched off because it is unlikely that a user will use the device in that manner.

I personally think that be it MobileMark 2007 or 3DMark2006, the tests should be conducted in such a way so as to simulate the way in which the customers actually use their mobile devices. It should measure performance when the Wi-Fi is switched on, screen brightness is at a comfortable level, and performance should be measured while browsing, editing document, playing music, etc. If a company claims that the device will work for X hours then the company should clearly state the underlying conditions for such a battery life (e.g. 3 hours with 40% screen brightness, WiFi switched off).