Skip to content

On Precision

I recently talked about precision in a lab meeting, where I connected it to Helical semantics. This post summarizes some of the background and observations that informed that chalk talk.

There are three uses of precision that I focused (and will focus on in this post):

  1. finite precision/floating point representations,
  2. measurement precision of an instrument, and
  3. statistical precision (inverse covariance).

First though, I'll start with an OED review of the noun:

  1. Chiefly Philosophy. The action or an act of separating or cutting off, esp. the mental separation of one fact or idea from another; abstraction, definition. (1529-)
    1. An instance of exactness or preciseness; a particular, nicety, minute detail, esp. of language. (1695-)
    2. The fact, condition, or quality of being precise; exactness, accuracy. (1698-)
    3. The degree of refinement in a measurement, calculation, or specification, esp. as represented by the number of digits given. Contrasted with accuracy (the closeness of the measurement, etc., to the correct value). (1842-)
    4. Statistics. The reproducibility or reliability of a measurement or numerical result; a quantity expressing this. (1876-)
    5. The accuracy of an information retrieval system, expressed as the proportion of items retrieved by a particular search that are relevant. Cf. recall, noun 1.3b.

2.c. and 2.d. are the closest to what I'll be discussing.

Finite precision

Representing real numbers with finite bits

This is the classic and familiar-to-computer-scientists problem of how to represent real numbers with finite resources.1

For the uninitiated, this the problem where both 1.0000... and 0.9999.... represent the real number we call "one." This representation with ellipses informally captures the left and right limits of one-as-a-real-number. We can capture this a bit more formally in limit notation: \(\lim_{\epsilon\rightarrow 0}\frac{x\pm\epsilon}{x} = \text{one-as-a-real-number}\)

The infintesimal calculus2 prescribes an algeba of limits over which we can reason symbolically. Unfortunately, we could easily find ourselves in a situation where the representation of numbers becomes inordinately large and incomprehensible to a person. In such situations, it would make more sense to use an approximation for these numbers.

In mathematics, this approximation task is known as a "numerical" (rather than analytical) solution. In computer science, we have a subdiscipline known as "numerical computing." Computing systems that operate over symbolic representations of limits and other expressions can be forced into a representation that "fits" within the available resources (see footnote 1). My understanding is that systems like Maple support this style of computation.

Conversely, if you've ever written a program in a general-purpose computing systems — specifically, a general-purpose programming language — you'll know that none force you to specify limits as such. Rather, if you say that \(x\) isequal to 1.00, in many languages this will be interpreted as an approximation of a real number (i.e., in "floating point" representation). The fundamental challenge of using these approximations is that their errors add up. A naive representation of numbers could result in the computation of observationally equivalent expressions to be different. People have put a significant amount of effort into making such situations unlikely. This property is known as "numerical stability."

All of this is to say that "precision" to computer scientists typically refers to this problem of approximating real numbers with finite bits.

Measurement, precision, and significant figures

In primary school, when we are first learning about measurment, those of educated in the US are typically taught that precision refers to the consistency of the number you get in the act of measuring. This could be taking the temperature ten times, or you and your lab mate taking turns mesuring the distance that a toy car traveled using a yardstick. It is usually in this context that the notion of significant figures is introduced.

<1-- While we phrased this problem as one of representation on a computer, it significantly predates the computing discipline, as the notion of significant figures ... or so I thought (more on that after the definition/review). -->

What are significant figures?

Significant figures or significant digits refer to the number of digits you report in empirical contexts, usually in scientific notation. Recall that scientific notation is where you convert a number to the form \(d_1.d_2\ldots d_n \times 10^{\pm n}\), where \(\pm\) is either positive or negative, depending on whether the number you are encoding is greater than or less than one. Once in scientific notation, you then must determine how many digits are significant.

Example. The number 12345 would be \(1.2345\times 10^{4}\) and 0.0012345 would be \(1.2345\times 10^{-3}\). Both correspond to reporting five significant figures (the number of digits in the multiplier or mantissa).

We are generally taught how to determine this value via recipe, but in summary the number of digits corresponds to the number of digits of initial input (where input is typically the result of measurement). Determining the number of significant digits gets more complicated when we report the results of computation (similar to our numerical stability problem above).

Example. A given thermometer reports body temperature down to one tenth of a degree. On a day you feel good, r you read your temperature as 98.4F, or \(9.84\times 10^{1}\) in scientific notation. On a day you feel bad, you read your temperature as 101.0, or \(1.010\times 10^{2}\) in scientific notation. The first reading has three significant figures, but the second has four.

Here is where precision in significant figures is confusing, relative to the notion of finite precision: I've found some evidence that around the mid-20th century, the number of significant figures in the mantissa corresponded with accuracy, while the exponent/scale corresponded with precision...

"The precision of an approximate number is determined by the size of the unit of measurement. The accuracy depends upon the number of significant digits involved."

Cecil B. Read, Comments on Computation with Approximate Numbers, Presented at the Thirtheenth Christmas Meeting of the National Council of Teachers of Mathematics, December 30, 1952.8

...whereas today standards bodies would consider both the mantissa and exponent the precision:

"Precision ... deals with the "closeness of agreement between test results."

Brad Kelechava, ISO 5725-2:2025 – Accuracy Method For Repeatability, December 26, 2019

Where things get confusing is that we can imagine that the finite width of our approximate representation is related to both, but not equivalent to either, the mantissa and the exponent. Under the ANSI definitions of precision and accuracy, the entirety of scientific notation is equivalent to the bit representation problem, since we now define precision as a form a numeric stability.

What is the history of significant figures?

I had assumed that the notion of significant figures pre-dated the finite precision problem, but after some searching, I am no longer quite so sure.3 As should be clear in the section immediately above, the definitions of the terms "precision" and "accuracy" have evolved over the past 100 years.

I had little luck finding the origins of the term:

  • There is no entry in the OED.
  • There is an entry in the Encyclopedia Britannica, which gives a nice summary of the rules of and principles behind reasoning with significant figures, but it had no information about its origins.
  • I checked the appendix of Stephen M. Stigler's The History of Statistics: The Measurement of Uncertainty before 1900, but found no entry for significant figures.
  • I tried search Google scholar with queries like "'significant figures' history measurement" and various iterations, but found little.
  • I tried searching r/AskHistorians and found nothing useful.

While I hit a dead end in tracking down the origins of the term, the principles behind them have obviously been in development for a long time, in metrology, or the study of measurment.4 To learn more, I'd ideally browse stacks of the metrology section of the library. However, there are several problems with this approach:

  • Most of Northeastern's stacks are in storage; the actual stacks occupy 1/4-1/3 of one floor.
  • Metrology doesn't have its own section according to the Library of Congress classification system. I decided to try to take a look at some example classifications, to get a sense of where they could end up.
  • Many of the books now in Northeastern's system are electronic-only; the Northeastern system does not display the Library of Congress number in these cases. Since a publisher needs to request a LoC number, it is possible that some of these eBooks do not have one (I searched for several of the Snell eBooks by ISBN and could not find anything on the LoC website).
  • If I restrict my search to physical books, there are only 9 results. The breakdown in categories are:
    • TA5 (Engineering, General): 5
    • C (Auxiliary Sciences of History, General): 1
    • Y9 (???): 2
    • QA (Mathematics): 1

This doesn't mean all is lost: unsuprisingly, metrology spans many disciplines and got its start in ancient philosophy. We can reason a bit about significant figures through what we know about imprecision in measurement.

Stigler identifies the year 1100 as the first recorded instance he could find of procedures for dealing with imprecision in measurement.5 Our modern history of metrology is deeply intertwined with physics, statistics, and politics, with its modern catalyst being the French Revolution. To summarize from the first chapter of Stigler's book: the history of combining multiple observations is fairly recent, and doing so in a principled way that we would recognize as part of our current curriculum is only about 150 years old. The big intellectual leap was to use methods developed for aleatory uncertainty (e.g., games of chance) to model epistemic uncertainty (fundamentally what imprecision in measurement actually is). Prior to the rigorous application of statistical methods to measurement, natural scientists used a variety of ad hoc methods for determining the best/most accurate measurement from an imprecise instrument.

Given that in 1952, educators were still discussing the best ways to teach significant figures and their meaning, it might even be the case that finite precision approximation developed concurrently with significant figures.

Some fun resources I'd found along the way:

Do significant figures even matter in computing?

I can say as someone who has reviewed many academic papers that a surprising number of computer scientists seem to forget about significant figures when performing empirical analyses!6

As of Feb 11, 2026, there are 630 papers in the ACM Digital Library that contain the phrase "significant figures" (according to the DL's search function). Just skimming the titles and abstract, the earliest work (from the 1950s) seems to be focused on numerical stability as the problem under study, rather than using the algebra of significant figures for empirical research questions. More recent work — especially work on cyber-physical systems — seems to evince sensitivity to limitations of measurement.

Unfortunately, the ACM DL does not provide an API that would allow efficient meta-analysis; in order to do a proper analysis, I would need to painstakingly download all of the relevant papers and metadata by hand (or through a browser-based bot) and extract data for queries myself. This is just a blog post, so I'm not going to put the effort in at this time. However, my search tells me to expect some forthcoming work from Daniel Olszewski that characterizes the prevalence of significant figures, so I am looking forward to that. :)

In any case, I wanted to use this section to reflect on the fact that computers have long been treated as both instruments of measurement and as objects of study (i.e., those whose behavior is being measured). Much of the Helical work was motivated by this observation and how it can confound our ability to measure accurately.

Statistical/Probabilistic precision

Many practicing computer scientists work at the boundary between computing and statistics/probabilistic modeling.7 For them, without context, precision might refer to the statistical definition, which is closer to the measurement stability definition than the finite bits approximation definition (although all three are related).

As usual, we have some faux amis when PL and statistics meet. In statistics, variance is a functional whose input must have the signature of a random variable (i.e., a map from outcomes to reals).

\(\mathbb{V}(X) = \mathbb{E}\left[(X - \mu)^2\right] = \int_{x}(x - \mu)^2 f(x)dx\)

As a reminder: \(\mu\) is a constant in the context of fixed \(X\) and \(f : \mathbb{R} \rightarrow (0, 1)\), whose value can be calculated from these parameters:

\(\mu = \mathbb{E}[X] = \int_{x} x f(x)dx\)

Variance is also constant for fixed \(X\) and \(f(X)\); when the context is clear, we often write the variance constant as \(\sigma^2\).

Precision for a single variable is then defined as the inverse of this quantity: \(1/\sigma^2\).

Intuitive notion of variance

Setting aside formal statistical definitions, we can see how we might come up with a quantity like variance in order to describe dispersion from the mean: we might, for example, compute the average distance from the mean, weighted by the likelihood of that event, e.g.:

\(\int_{x\in\mathcal{X}}|x - \mu| f(x) dx\)

We prefer the squared version for a variety of reasons, most out of mathematical convenience: the quantity is now differentiable, this definition (also called the "second central moment") is related to the "method of moments" approach to proving the central limit theorem, and there are probably deep connections to L-p spaces and topology that I barely understand.

The important part is understanding that (1) variance is a positive real number and (2) greater average distance from the mean means larger variance. Thus, as variance approaches zero, precision approaches infinity. In layman's terms, we can see how this maps quite nicely to our notion of stability in measurement.

This is all makes intuitive sense, but things get weird if we try to use intuition when thinking about random variables, generally. For example, if we say that \(X\) is a random variable corresponding to height in the US, it seems strange to talk, in layman's term, about its precision. This variable captures characteristics of a population; it's unclear what measurement we intend to perform, and therefore what precision in layman's terms refers to.

While there will be times when certain statistical terminology overlaps with our common understanding of the word, because the mathematical definitions supercede the motivating origins, we generally want to reach for the mathematical definitions first. That is, in the context of random variables, we should assume that "precision" refers to inverse variance (or in the case of multiple random variables, inverse covariance), rather than implying measurement or estimation.

Final Thoughts

This post is a big longer (and took a surprising amount of time to write!) than I'd like, so in sum I just want to say that "precision" is one of these words that has a generally agreed-upon meeaning, but when used in different technical contexts, carries slightly different nuances. It's distinct from the vagueness or quantum state of suitcase words — the definitions are actually quite precise and specific, it's just the context that determines their facets.

Related post: What even is a "parameter"'?

  1. By finite resources we generally mean space or time; there are other, more niche resources like randomness or quantum...something (not my area!). For space, we typically mean representing a real number using a finite number of bits. For example, one approach to computing over real numbers is to use a ``floating point'' representation, where interpret the contents of a fixed number of bits according to a shared definition over a truncated range of numbers. Under this representation, we can only, for example, calculate pi to a finite and fixed number of digits. We might try to get around this by treating the digits of pi as the result of a computation. In that case, we would need less space, supplying the digits on demand. Unfortunately, the computation would likely run forever, which is generally an unacceptable property of programs designed to terminate. ↩

  2. The infintesimal calculus is one of topics covered in courses called "calculus" or "pre-calculus." I had a friend who, echoing greater thinkers than either of us, referred to "calculus" as a "bastard amalgamation of real analysis, set theory, numerical methods, and several other calculi cobbled together in order to rapidly train Americans to beat the Russians to the moon." I'm not sure how historically faithful this take is, but it does suggest major flaws in American mathematics education that easily lead to confusion among students and likely stand in the way of developing mathematical maturity. ↩

  3. I am not a historian. This means I am mostly cobbling my impressions together from Wikipedia, books I'm reading — namely Stephen M. Stigler's The History of Statistics: The Measurement of Uncertainty before 1900 —, historical references in various textbooks, Google Scholar, and standards organizations that have an interest in metrology (e.g. NIST, ANSI, W3C, etc.). ↩

  4. Analogously to precision, let's take a look at the non-obsolete and relevant OED definitions of mesurement: The action or an act of measuring or calculating a length, quantity, value, etc. (1590-) (3a) A dimension ascertained by measuring; a magnitude, quantity, or extent calculated by the application of an instrument or device marked in standard units. (1590–) (4) A system of measuring or of measures. (1838-). I didn't find this definition particularly insightful and since we have more formal definitions under metrology, it didn't seem worth discussing in the body of this post. ↩

  5. This refers to "The Trial of the Pyx," the daily weighing of coinage during the Norman conquest of the British Isles. Stigler's text focuses on the Western European development of statistics; since I'm interested in how significant figures came to be under Western schooling, I leave investigation into other, independent origins to actual historians. ↩↩

  6. I'm guilty of this, too! There's a paper I wrote early in my academic career that commits this sin. ↩

  7. Henceforth, I'm going to use "statistical," rather than "statistical/probabilistic," but most of this section would make more sense in terms of the wordier "probabilistic modeling," since it's about computation on random variables, rather than estimation of parameters via functions of samples. ↩

  8. Now, according to Google scholar, this paper was only cited once. I've found the accuracy (heh) of Google Scholar results before 2000 to be...questionable, so the actual impact or influence of this talk remains uncertain. ↩

  9. Y is not a LoC category; these resources are listed as being on microform and in the law library. ↩