Numeric precision

The precision of a numeric value is the number of digits that are used to show that value.

All numbers in the input and numeric literals in map rules are converted to multiple precision and converted to the format required at output time.

  • If the multiple precision number does not overflow but does not fit into the maximum size of a numeric item, that item is filled with the number sign character (#) up to its maximum size.
  • An arithmetic overflow is recognized when an arithmetic expression includes:
    • Divide by zero
    • A multiple precision number greater than 1.0E254

Arithmetic overflow is designated in the output object with up to a maximum of ten number sign (#) characters.