U.S. Cloud Benchmark:

The JDN/CloudScreener/Cedexis U.S. Cloud Benchmark is published by the JDN in partnership with CloudScreener and Cedexis.

Download the U.S. Cloud Benchmark



JDN (Journal du Net) is a French news website which specializes in IT and economics. It belongs to Le Figaro Group, —the first web media group in Europe— which receives 23.5 million readers per month.

CloudScreener as a Cloud Decision Engine, allows companies from all around the world to make data-driven decisions to improve their performances and reduce their cloud costs.

Cedexis optimizes the cloud in terms of network performance for companies that want to ensure 100% availability and extend their reach to new global markets.




Contact Us

Methodology

The JDN/CloudScreener/Cedexis U.S. Cloud Benchmark is based on three sub-rankings: the first is cloud price, the second is cloud performance, and the third is cloud service level. It was built based on data from the cloud monitoring services of CloudScreener (for VM performance, VM prices, and VM level of service) and of Cedexis (for network performance).

Ranking Performance/Price/Service Level

The performance/price/service level index is the average of three indexes: the performance index, the price index, and the service level index.

1. Performance Ranking

The cloud providers' performances (network and virtual machine) are assessed relative to a reference benchmark value that is considered state of the art by JDN, CloudScreener, and Cedexis. To determine virtual machine performance, a performance index is first calculated for each criterion (Disks, CPU, and RAM). It is based on the difference between the average value of measurements (corrected by the standard deviation) and the reference benchmark value. The standard deviation coefficient reveals the degree of infrastructure stability. The virtual machine index is limited to 100 points for any value over the reference value.

Formula for the Performance Ranking Index:

Cloud Benchmark

The network performance indexes (for the rate of availability and response time) correspond to the state-of-the-art standard deviation. The performance indexes for the rate of availability and response time are zero if the availability rate is below 98%.

The final performance index is the average of Disks, CPU, RAM, and network indexes. Each of the Network, Disks, CPU, and RAM indexes has the same weighting (25%) in the average that makes up the performance index. The performance ranking is, then, organized in descending order, beginning with the highest index.

Each performance measure of virtual machines is performed by CloudScreener more than 150 times per month for each provider, and randomly across several U.S. systems.

Precision of Velocity Measurements of I/O and CPU Performance:

  • Median response time (in milliseconds): measured by Cedexis for all of the U.S.'s cloud providers (see Cedexis' methodology)
  • Availability rates: measured by Cedexis for all of the U.S.'s cloud providers (see Cedexis' methodology)
  • Disk IOPS: average number of Input/Output Operations per Second, measured in 4K random
  • Disk Bandwidth (KB/s): measured by Cedexis (1 MB sequential writing)
  • CPU: number of workloads measured over 32 processes in 60 seconds (sysbench)
  • RAM (MB/s): CPU/RAM transfer speed
Kinds of systems and services analyzed
Source : JDN/CloudScreener/Cedexis
Amazon Web Services m4.large
Rackspace compute 1.8
Microsoft Azure D2 v2
Google Compute Engine n1.standard-2
IBM SoftLayer 2 cores, 8 GB of RAM






2. Price Ranking

The price index is based on three standard configurations: 1 Windows virtual machine (large), 1 Linux virtual machine (large), and a small cluster of virtual machines. The price index is based on the median price of each configuration corrected by an index of 50/100. The price index for each configuration is the deviation from the average prices of the provider and the median price. The most expensive cloud provider gets 25 points in the index, and the cheapest gets 100 points in the index. The price ranking is organized in descending order, beginning with the highest index. Details of project types, systems, and services analyzed by each provider are shown in the table above.

Standard configurations
Source : JDN/CloudScreener/Cedexis
Windows  Windows VM, large (2 vCPU, 8 GB of RAM and 100GB of storage).
Linux Linux VM, large (2 vCPU, 8 GB of RAM and 100GB of storage).
Use case (cluster Project) 6 VM, large (font end) + 3 XL VM (back end) +  2000 GB of outbound traffic / 200GB of inbound traffic + 500 GB of Block storage.

Disclosure: SoftLayer offers VM that have up to 16 cores (not vCPUs) and 64 GB of RAM, and "bare metal" servers that have up to 40 cores and 512 GB RAM. SoftLayer guarantees 100% uptime on administration portal and network, with credits to customers for any interruption beyond 30 consecutive minutes (99.93%). SoftLayer doesn't have any uptime guarantee on VM, but a guarantee of recovery in 2 hours on the underlying hardware (i.e. 99.73% monthly).


3. Service Level Ranking

The service level index is based on a group of qualitative criterion: the geographical coverage (its presence in the U.S. and outside of the U.S.), the number of certifications, the SLA, and the range of VM. The service level index is the average of these criterion, with each service level criterion weighted equally.

The certification index depends on the number of certifications of each cloud provider. The geographical coverage index depends on the number of data center locations. For the SLA criteria, a 100% uptime guarantee gets 100 points in the index, whereas a 99.5% uptime guarantee gets 25 points. A provider not disclosing any SLA gets 0 points. The Range of VM index is based on the VM sizes proposed by each provider:

  • 0 to 16 vCPU : Index of 50
  • 17 to 32 vCPU : Index of 75
  • 33 vCPU and higher : Index of 100
  • 0 to 64 GB of RAM : Index of 50
  • 65 to 262 GB of RAM : Index of 75
  • 263 GB of RAM and higher : Index of 100
Play

JDN

Journal du Net (JDN) is a French news website which specializes in IT and economics. It belongs to Le Figaro Group, —the first web media group in Europe— which receives 23.5 million readers per month.

Thank you for your interest in the JDN/CloudScreener/Cedexis U.S. Cloud Benchmark. Please contact us using the information below.


JDN

  • 94 rue de Provence, 75009 Paris FRANCE
  • +33 (0) 147 795 000

CloudScreener

  • 15 rue du Caire - 75002 Paris FRANCE
  • +33 (0) 967 367 367

Cedexis

  • 317 SW Alder St, #650 Portland, OR 97204
  • +1 (650) 587-3518

© Copyright 2015 — All Rights Reserved