Notebookcheck

Our test criteria

The following article aims to give the reader an understanding of our standard tests and their respective procedures. We will give a detailed explanation of how we take measurements (equipment, tests setups, settings,...), and how we interpret the results we receive from the various tools and software.

Introduction

Our editors have years of experience and have tested numerous notebooks from various manufacturers. We used this extensive experience to create a catalog of test criteria which cover all purchase relevant aspects of a notebook.
Consumer's demands change over the years and laptops evolve. As such, we must also constantly update and add to our test criteria so as to give the reader the information s/he is looking for.

To do our job well, we need your input. We are always happy to receive an email or a new post in our German forum. Our editors and a long line of helpful moderators are available and will gladly take your tips.

The following are the main aspects of every review performed by Notebookcheck.net (explanations are included): 

Case

The case of our test model is examined very closely. We judge the test model with the following criteria:

  • Design (colors, shape, material, feel, measurements, weight...)
  • Workmanship (gaps, finish, edges, precision; how secure every component sits,...)
  • Sturdiness (How the notebook reacts to pressure - at a single point, on the entire surface, torsion resistance of the display, how well the notebook sits in the hand,...) 
  • Hinges (power, how well they hold the display, longevity,...)
  • Maintenance (possible upgrades which can be performed by the user, cleaning of the device/cooler fan,...) 

The rating of the workmanship is decided from the score given by the respective editor and by the editorial team which compares the device to previously-tested models.

Connectivity

Next we take a look at the ports and interfaces available on the test model. We judge their positioning and how useful they are in everyday use. Laptops of different classes are judged by different standards.

  • Port types
  • Quality (transfer rates,...)
  • Quantity
  • Positioning

We also take a look at the communication features, such as, LAN, WiFi, Bluetooth, UMTS, etc, in this section. 

The delivery package contents and the various security features and accessories available for this model are also noted here.

Input Devices

Input devices play a decisive role in notebooks. The quality of the keyboard, touchpad, trackpoint (if available) and/or touch display (Tablet PCs) is very important for potential buyers. The following criteria are taken into consideration:

Keyboard

  • Layout (positioning, size, grouping, function keys, inscription...)
  • Typing feel (key travel, pressure point, impact, noise,...)
  • if available - additional keys

Touchpad

  • Response (surface, multi-touch,...)
  • Mouse keys (use, noise,...)

If available - Touch Display

  • Response (precision, reaction time,...)
  • Virtual keyboard (layout, feedback, response, key size,..)
  • Sensors
  • if available - Digitizer (capacitive display)

Display

Display brightness: Measurement points of the Gossen Mavo Monitor tool
Display brightness: Measurement points of the Gossen Mavo Monitor tool

We use the Mavo Monitor tool (made by Gossen) to measure the brightness of the display.
The measurement of the brightness is taken after the screen has stayed 100% white for a period of 10 minutes. Device settings, such as automatic adjustment of brightness, are deactivated and the color profile of the device has factory settings (not user defined).
The black value is also measured after a time period of 10 minutes, during which time the screen is 100% black (@max. brightness). The measurements are then taken from the central area of the screen while in a completely dark room to avoid potential ambient light influences. These measurements are used by us to calculate the maximum contrast of the display (max. brightness/black value).
We calculate the illumination by comparing the maximum brightness to the brightness of the display at its borders.

The following criteria are considered for the display rating:

  • Resolution and format (pixel density, overview of the display, ease of use with multiple windows...)
  • Display brightness [cd/m²] (power-/battery mode,...)
  • Illumination (dark areas, bleeding,...)
  • Contrast (black value,...)

We measure the color range of the screen using the display calibrator tool: X-Rite i1 / Spyder 3 Elite from Datacolor. The measured color space is compared to the sRGB standard space and/or compared to other displays.

Outdoor use is part of the display test. We check to see how well the display can display contents (legibility, reflections, etc.) in bright surroundings (3000-10000 cd/m² - cloudy to sunny). The list of criteria for this test is:

  • Type of display (matt panels prevent reflections)
  • Brightness of the picture 
  • also the picture contrast

The viewing angle stability describes how the picture content changes as the display is turned to certain angles. We test the viewing angle stability of the notebook subjectively (using the test model) and by turning the display to fixed angles (45° turns). For this test, the display is photographed at a fixed shutter speed and aperture inside a dark room.
The photo gallery we present at the end of the test is supposed to delineate how the picture content changes (color falsification, inversion, etc.) as the user's viewing angle deviates from the optimal viewing angle.

Performance

Our performance tests vary depending on the device class and the expected performance from this class. This section includes a variety of benchmarks (which either test a single component or the entire system and present the result in points), and the practical everday test, in which we run various programs and computer games.
The benchmarks used for this section are usually the 3D Mark and PC Mark benchmarks from Futuremark, Cinebench R10, and various games.

We always use the pre-installed graphics driver, unless specifically mentioned beforehand. Updating the driver can give boosts in performance while playing games. However, we believe that it is the duty of the manufacturer to provide the customer with a laptop with updated drivers, and as such do not usually change the driver ourselves. 
In the following sections, all aspects of the computer performance are covered:

  • CPU (Turbo analysis, Cinebench R10, R11.5, wprime, SuperPi,...)
  • System (PCMark Vantage, PCMark 7)
  • RAM (HDTune, Crystaldiskmark, ASSSD,...)
  • GPU (3D Mark 2006 1280x1024, Vantage 1280x1024, 3D Mark 11 1280x720, Unigine Heaven,...)
  • Gaming (many of the latest games)

Additional information:

Emissions

Noise emission: position of gauge
Noise emission: position of gauge

Noise Emission

We use a noise level gauge (Voltcraft SL-300 or similar) and a standard test setup to measure the emissions of a test model. The gauge is fixed 15 cm from the notebook and is secured against vibrations emanating from the test model. The measurements are taken in dB(A) (Decibel). The following are our test categories:

  • Idle
    Minimum: minimum noise emission while laptop is idle (Windows power plan: "Energy Saving"
    Medium: Average noise emission recorded while laptop is idle (Power Plan: "Energy Saving")
    Maximum: Highest noise emission measured while the laptop is idle (Power Plan: "High Performance")
  • High System Use
    Medium: Average noise emission while the computer is running at high level of system use. (3D Mark 2006, Power plan: Power Plan: "High Performance")
    Maximum: Highest possible noise emission while the system is under heavy load (Power Plan: "High Performance", 100% CPU and GPU usage - thanks to Prime95 and Furmark)

The following may help the reader better understand the results: 

In a quiet room, the human ear can hear background noise, which should amount to around 28 dB. A conversation at a normal volume ranges at 60 dB. All these values are dependent on the distance from the source of the noise. This is why we fix our gauge into place at a constant distance from our test models. This allows us to get clear results which can be compared with each other. The measurements are presented graphically and can be judged subjectively:

  • Under 30 dB: barely audible
  • Up to 35 dB: audible but not distracting. Ideal level of noise emission for a laptop running office programs.
  • Up to 40 dB: clearly audible, and might be distracting after a while
  • Up to 45 dB: might disturb the user if s/he is in a quiet room. Still acceptable while playing games. 
  • Over 50 dB: notebook emissions over this level are uncomfortably loud
Measurement quadrants for the surface temperature
Measurement quadrants for the surface temperature

Temperature

The distribution of surface temperature (which can be felt by the user directly) is measured with a infrared thermometer (Raytek Raynger ST or similar) which never touches the test model. The top and bottom of the notebook is split into nine quadrants, and the maximum measurable temperature in each quadrant is recorded.

The measurements are taken after an idle period of 60 minutes and a stress period of 60 minutes (100% CPU and GPU usage - Prime95 and Furmark). 
In addition we also closely observe the GPU and CPU during the stress test with software (tools: HWInfo64, HWMonitor, GPUz,...) and note any significant variations in performance (drops due to throttling).
The following scale describes the categories we put our measurements in:

  • Less than 30°C: Barely noticeable increase in temperature.
  • 30 - 40°C: Temperature rises noticeably but is bearable.
  • 40 - 50 °C: Contact with the notebook over a long period of time at these temperatures will be uncomfortable.
  • Over 50°C: Very hot. Problematic if using the notebook on the lap.

Speakers

This rating depends on the positioning, maximum volume and sound quality of the speakers. The rating is taken subjectively after taking into consideration previously tested laptops of this class.

  • Sound (highs, lows, different music genres, voice playback,...)
  • Max volume (drops in sound quality, good enough for film playback?,...)
  • Position of the speakers (playback with the notebook on the lap,...)

Power Consumption

The power consumption of the notebook is measured in various scenarios. The test settings for each scenario are as follows:

  • Idle: power consumption while the notebook is idle.

Minimum: all additional modules are off (WiFi, Bluetooth,...), minimum brightness, and Windows power plan is set to "Energy Saving". 

Medium: maximum brightness, additional modules off, Windows power plan: "Balanced". 

Maximum: maximum power consumption while notebook is idle. All modules are on (WiFi, Bluetooth,...), maximum brightness and power plan set to "High Performance".

  • High Load: notebook runs with maximum brightness, all modules on and power plan set to "High Performance". 

Medium: For this test we used 3D Mark 2006 and record the average power consumption in the first part of the test.

Maximum: stress test with 100% CPU and GPU load using Prime95 and Furmark benchmarks. Maximum power consumption possible on the test model.

A notebook is judged by its battery life. As battery life varies depending on the usage, we run our test models through 4 different tests, so as to get the most accurate measurements of their run times:

 

  • Minimum run time: we use the "Classic" test of the Battery Eater Classic to measure the minimum run time of the test model. For this test, the screen brightness is set to maximum and all communication modules, such as, WLAN, Bluetooth, etc, are turned on. Additionally, the Windows power plan is set to "High Performance". For our Android-based test models we use the App "Stability Test" to judge the minimum run time. If the App does not run on the device (due to compatibility issues) then we run a 3D game which simulates high load, thus, allowing us to measure the minimum run time of the test model.
  • Maximum run time: the "Readers" test of the Battery Eater tool is used to measure the maximum run time of the test model. The brightness is set to minimum and all power-saving options are turned on. The Windows power plan is set to "Power Saver" and WLAN and Bluetooth are switched off. Android-based devices are tested with a script which loads text pages from the site: http://www.notebookcheck.com/fileadmin/Notebooks/book1.htm
  • WiFi mode: the possible battery life while surfing the Internet via WiFi with medium brightness (~150 cd/m²) and power-saving options ("Power Saver" mode) switched on. We measure the run time by letting the device run an automatic script, which picks a mix of websites and switches between them every 40 seconds.
  • DVD playback: runtime while the laptop is playing a DVD with maximum brightness, WLAN and Bluetooth off, and power-saving options turned on (such as, the Windows "Power Saver" or higher - whichever is necessary for fluid playback of the DVD).

The reader should take into account the fact that our test models are usually new laptops. This means that the battery of the laptop will have to be emptied and recharged a few times before it can deliver its peak battery life. Furthermore, our tests provide results which are taken over a relatively brief period of time. More information on how to optimize the battery life of your laptop can be found here: FAQ Article.

Rating

Each review we write is capped off with a final rating. The test model receives ratings for each section mentioned in this article, and is also given a rating after being compared to other models of the same class. The final rating is influenced by 12 points and the influence each point has on the final rating varies from class to class: netbook, gaming notebook, etc. We present the rating on a scale of 0-100 percent (higher is better).
The various rating criteria (the case and input device ratings are excluded) are processed with a special algorithm, which uses the various measurements and benchmark data in our database to deliver the result. 

More information about the rating system can be found here.

> Notebook / Laptop Reviews and News > FAQ / Tips / Technics > Our Test Criteria
Author: J. Simon Leitner, 2009-04-10 (Update: 2012-05-26)