Fundamentals

What is CFD?

A plain-language overview of computational fluid dynamics for data centers: what a simulation answers, why its mesh is built from box-shaped cells, and what a result can and can't tell you.

CT
CoolSim Team
· Updated · 11 min read
Four rows of server racks on a raised floor, cut open by a vertical plane. At the left the plane shows the bare computational mesh; across the middle it is colored by the solved air temperature: a blue column of cold air rising from the floor tiles between two rows, and warm orange air above the racks. Red and blue lines on the right trace hot exhaust air and cold supply air.

Computational fluid dynamics (CFD) computes how air, heat and pressure move through a space by solving the equations of fluid flow on a computer. For a data center, that turns questions like Will this row run hot if a cooling unit fails? or How much of our cooling is reaching the racks? into questions you can ask of a model instead of the room.

What it answers

Data center cooling is a spatial problem. A room can have more total cooling capacity than its heat load and still overheat a rack, because the cold air never reaches that rack’s intake. Averages don’t show that; a spatially resolved model does. CFD lets a team:

  • find hot spots and trace them to their cause, such as recirculated exhaust, a starved tile or bypass air;
  • check whether the room could run with less cooling airflow or a warmer supply, which is where PUE improvements usually come from;
  • run failure cases, such as each cooling unit off in turn, on the model instead of the room; and
  • compare options such as containment, plenum depth or tile layout before committing capital.
The hottest point on each rack’s intake, seen from above, in the room at the top of this page: 48 racks of 4 kW in four rows with contained hot aisles, cooled by four units through 48 floor tiles (dotted). Every rack above ASHRAE’s recommended 80.6 °F is at the end of a row, and the four hottest, at 83.8 °F, are at the ends of the middle cold aisle. An average over the room would not show any of them. Synthetic room.

How a simulation works

Every CFD study follows the same four steps.

  1. Define the domain. Build the geometry and set the boundary conditions. For a data hall that means the room, the raised floor or ceiling plenum, racks with their heat loads and airflows, cooling units with their supply temperatures and airflow, perforated tiles, and any containment.
  2. Mesh it. Divide the space into many small cells. The solver keeps one temperature, one air velocity and one pressure for each cell, so the mesh decides what the model can see. CoolSim builds the mesh from the model when the job is submitted.
  3. Solve. The solver iterates until the mass, momentum and energy balances hold in every cell and the quantities being watched stop changing.
  4. Post-process. The solution becomes something you can act on: temperature and velocity maps, pathlines showing where the supply air goes, rack intake temperatures, and metrics built from them.
Four views of the same vertical cut through a room with four rows of racks on a raised floor. 1, the model: racks, the floor tiles and the hot-aisle roofs. 2, the mesh: a grid of small square cells, finer near the racks. 3, the solved temperature: blue supply air in the cold aisles and under the floor, warm orange air in the hot aisles and above the racks. 4, lines tracing the air: cold air rising from the tiles into the rack fronts, warm exhaust rising out of the hot aisles and turning over above the rows.

The four steps on one vertical cut through the room at the top of this page. 1: the model, with the raised floor, its tiles (dotted) and the hot-aisle roofs (dashed). 2: the mesh, 5,910 cells on this cut. 3: the solved temperature, one value per cell. 4: lines tracing the air, colored by its temperature: supply air rises from the tiles into the rack fronts, and exhaust rises from the hot aisles and turns over above the rows.

CoolSim solves for the steady state: the room as it settles with its heat loads and cooling held fixed. That answers where the air and heat end up. How quickly a room heats up after a failure is a different question. It needs a transient (time-dependent) calculation, which we run as a separate cooling failure study.

The mesh

Of the four steps, the mesh gets the least attention and decides a great deal. Each cell holds a single value of each quantity, so anything smaller than a cell is averaged away. And the way neighboring cells meet decides how accurately the solver can pass heat and air between them.

A close view from above of a mesh cut at rack mid-height between two rows of racks, every cell outlined and filled with its own temperature: a deep blue cold aisle with small cells, orange hot aisles at the sides, and larger cells in the lighter blue room air beyond the row ends.

A cut through the room at the top of this page, 1 m (3.3 ft) above the floor, between two rows of racks (gray). Every cell is outlined and filled with its own solved temperature. The cells are 3.75 cm (1.5 in) against the racks, 7.5 cm (3 in) in the aisles and 15 cm (6 in) in the open room beyond the row ends.

Small enough to see the airflow

A 2 ft (0.6 m) floor tile is a good yardstick. We want at least three cells across every tile, and four or five is better. With two, the jet leaving the tile is described by two values across its width, so it can’t have a fast core and slower edges.

Three panels, each with the same blue curve: the speed of the air rising from a 2 ft floor tile, fastest in the middle and slower at the edges. Gray bars show what 2, 4 and 8 cells across the tile record: two equal bars, then four, then eight bars that follow the curve closely.

The same jet above a floor tile (blue), and what a mesh with 2, 4 or 8 cells across the tile can record (gray): each cell holds the jet’s average over its width. The jet’s shape is drawn for illustration; the cell values are its exact averages.

Smaller cells cost time. Halving the cell size multiplies the number of cells by eight, and the solve usually takes more than eight times as long. How much model is enough? solves one room on three meshes and shows which answers moved as the cells got smaller.

Why box-shaped cells

Cells can be any shape: boxes, tetrahedra with four triangular faces, or many-sided polyhedra. A data hall is made of boxes. Racks, cooling units, floor tiles, containment panels and plenums are rectangular and line up with the walls, so a mesh of box-shaped cells puts every one of their surfaces exactly on cell faces. Nothing has to be approximated to make the geometry fit.

The bigger reason is accuracy, and it comes down to the angle at which neighboring cells meet. Heat, momentum and turbulent mixing spread from cell to cell through the faces the cells share. To work out how much crosses a face, the solver takes the difference between the two cells’ values and divides it by the distance between their centers. That is exact only when the line between the two centers crosses the face square-on.

Two small grids of cells. A: square cells; the line from the center of cell P to the center of its neighbor N crosses their shared face at a right angle. B: slanted cells; the line from P to N crosses the shared face at an angle theta away from the face's normal.

A: on box cells, the line from one cell’s center (P) to its neighbor’s (N) crosses the face they share at a right angle, so the whole difference between P and N is change across the face. B: on slanted cells the line is θ off square, and part of the difference runs along the face instead.

On box cells the line always crosses square-on (A). On slanted cells it doesn’t (B). It meets the face at an angle, called the non-orthogonality, and part of the difference between the two cells is change along the face rather than across it. The solver has to estimate that part from the cells around them and take it out, a step called non-orthogonal correction. The estimate carries its own error, it adds work to every iteration, and as the angle grows the solution gets less stable; mesh checks commonly flag faces past about 70°. On box cells the correction is zero, because there is nothing to correct.

This is not a fine point. Turbulence is modeled as extra mixing (see Turbulence below), and the solver computes that mixing face by face in exactly this way.

What we measured

We checked the meshes of 21 customer jobs, from 20 customers, ranging from 8,400 to 17 million cells.

  • In 14, every face between two cells was square to the line between their centers, to within a hundredth of a degree. That includes all 51 million faces in the largest.
  • In five more, 98 to 99.8 percent of the faces were.
  • The other two mixed in tetrahedral and polyhedral cells, and 84 percent of their faces were square.

In the seven meshes with any slanted cells, the worst face in each was between 54° and 86° off square.

One dot per customer job: the share of faces between cells that are square to the line between the cell centers, to within a hundredth of a degree, against the size of the mesh. Filled dots are meshes of box-shaped cells only.

What box cells give up

A round pipe, a column or an angled wall doesn’t line up with box cells. The mesh either steps around it in a staircase or slants cells to fit it, and slanted cells bring the correction back. And because box cells line up in rows, a fine spacing needed in one place, such as a narrow gap beside a rack, carries on until it meets a wall or a solid object, adding cells where they aren’t needed. For a room of rectangular equipment, we think that is the right trade.

Two diagrams. A: a round column drawn as a dashed circle over a grid of square cells; the cells inside it are shaded and form a staircase outline. B: two racks with a narrow gap between them in a coarse grid; the extra grid lines needed to fit the gap run the full height of the room, stopping only at the racks.

A: a round column as box cells see it, a staircase. B: the extra lines needed to fit a narrow gap between two racks carry on across the room, stopping only where they meet a solid.

Turbulence

Air in a data hall is turbulent. The smallest eddies are far smaller than any practical cell, so turbulence is modeled rather than resolved. Its main effect is faster mixing of momentum and heat, and models represent that as an added, turbulent viscosity.

General-purpose CFD codes often compute that viscosity from two extra transport equations, the k-ε and k-ω families. CoolSim uses a simpler zero-equation, or mixing-length, model. It computes the turbulent viscosity directly from the local flow and the distance to the nearest wall, with no extra equations to solve. Zero-equation models are a common choice for enclosed rooms such as data centers, where the questions are about where air goes at the scale of racks and aisles.

Convergence

A converged solution means the equations are balanced on this mesh. It does not mean the inputs are right. A model is only as good as its rack heat loads and airflows, its cooling-unit settings, and the leakage paths it includes. Balanced is also not the same as finished. The residuals, which measure how far each equation is from balanced, can level off while the answer is still moving, so the temperatures you care about have to be watched until they stop changing. For that reason, the difference between two runs of the same model (with and without containment, with one cooling unit off) is usually more trustworthy than any one absolute temperature.

The solve behind the image at the top of this page. The pressure residual stops falling by iteration 500, but the hottest intake temperatures (one gray line per rack) keep moving by several degrees until about iteration 1,100. The run stopped at 1,701, once a reading at each rack’s intake had changed by less than 0.1 °F over 500 iterations.

The equations behind it

Fluid flow is governed by three conservation laws: mass cannot appear or disappear, force equals the rate of change of momentum, and energy is conserved. Together they are usually called the Navier–Stokes equations. Below, ρ \rho is the air density, u \mathbf{u} the air velocity, p p the pressure and T T the temperature.

Three panels, each a single square cell. Mass: equal arrows of air into and out of the cell. Momentum: pressure arrows pushing on the left and right faces, friction arrows along the top and bottom, and a downward arrow for the air's weight. Energy: cool air in on the left, warm air out on the right, and heat S added inside.

Each equation is a balance on every cell. Mass: what flows in flows out. Momentum: pressure, friction and the air’s weight change how the air moves. Energy: heat carried in, plus heat added (S), is carried out.

Conservation of mass says that air flowing into any small volume either flows out again or changes the density inside it:

∂ρ∂t+∇⋅(ρ u)=0 \frac{\partial \rho}{\partial t} + \nabla \cdot (\rho\,\mathbf{u}) = 0

Conservation of momentum relates the air’s acceleration to pressure differences, the stress τ \boldsymbol{\tau} from viscosity and turbulent mixing, and the weight of the air, ρ g \rho\,\mathbf{g} :

∂(ρ u)∂t+∇⋅(ρ u u)=−∇p+∇⋅τ+ρ g \frac{\partial (\rho\,\mathbf{u})}{\partial t} + \nabla \cdot (\rho\,\mathbf{u}\,\mathbf{u}) = -\nabla p + \nabla \cdot \boldsymbol{\tau} + \rho\,\mathbf{g}

The weight term is where buoyancy comes from. Air in a data hall moves too slowly for pressure to change its density noticeably, but temperature does, and that is why rack exhaust rises toward the ceiling. CoolSim computes density from temperature with the ideal gas law, at the air pressure for the site’s altitude, so warm air rises in the model as it does in the room.

Cooling is about moving heat, so an energy equation tracks temperature as the flow carries it and the air conducts it, with k k the conductivity (turbulent mixing included) and S S the heat added, for example by the equipment in a rack:

∂(ρ cpT)∂t+∇⋅(ρ cp u T)=∇⋅(k ∇T)+S \frac{\partial (\rho\,c_p T)}{\partial t} + \nabla \cdot (\rho\,c_p\,\mathbf{u}\,T) = \nabla \cdot (k\,\nabla T) + S

For the steady state, CoolSim drops the time derivatives, the first term of each equation. The ∇⋅τ \nabla \cdot \boldsymbol{\tau} and ∇⋅(k ∇T) \nabla \cdot (k\,\nabla T) terms are the ones the solver computes face by face from the difference between neighboring cells, the step that box-shaped cells keep free of correction. There is no closed-form solution to these equations for a real room full of racks, ducts and obstructions, which is why they are solved numerically on a mesh.

Where CoolSim fits

Traditionally, CFD meant general-purpose software and a dedicated analyst. CoolSim packages the workflow for data centers. Models are built in a Windows application from libraries of racks, cooling units, tiles and other data center objects. The meshing and the Ansys Fluent solve run in the cloud, and the results come back as a report with rack intake temperatures, maps, pathlines and the RCI and RTI cooling metrics.

The top of a CoolSim summary report for an example data hall: the result, Overcooled, with counts of racks too cold, in range and too hot against the ASHRAE range, and findings on racks, cooling, airflow and energy.

The first page of the report for CoolSim’s example data hall, 234 racks on a 655,782-cell mesh. Every one of that mesh’s 1.9 million faces between cells is square.

For a longer introduction, with two example rooms, see WP107: Using CFD for Data Center Design and Analysis.

See it on your layout

Model your own data center in CoolSim.

Put these ideas to work on your facility — spin up a 30-day trial or talk through a scoped consulting engagement with our CFD team.

Keep reading