Precision is how precise a numerical data type can be.

For example, the integer data type of most programming languages is only precise to the integer, meaning that only whole numbers can be represented. So, if you need a more precise number, than 1 or 2, say 1.5, you're frankly just screwed if you are just using the integer data type.

Most programming languages also offer a real or floating point data type that can contain a decimal number and therefore be more precise than the integer data type. Some programming languages separate these into single and double precision data types. The names "single" and "double" does not mean that two decimal points are used. It simply means that the double can have twice as many numbers after the decimal point than the single and therefore can be twice as precise as the single.

Rounding Errors

In real life, numbers can be infinitely precise, such as PI, which continues on forever after the decimal point. Computers simply do not have this capacity and most programming languages round off the more precise part of a number when it is assigned to a less precise data type.

For example, if you were to assign 1.5 to an integer data type, it would become 1. In more precise data types, this means that a tiny fraction of very precise numbers will be rounded off, simply because the computer cannot handle them. Over several thousand calculations, this tiny fraction could add up to a large inaccuracy. These inaccuracies are known as rounding errors.

Related Pages

Backlinks

These pages link back to this page. You may find them helpful.

Ask a Question

Ask a question about precision.