precision question

2012-10-25  Source: Original Site  Category:C++  Views:0 


Consider the equation (fly dynamics stuff):

Yaw (Degrees) = Azimuth Angle(Radians) * 180 (Degrees) /
3.1415926535897932384626433832795 (Radians)

I'm perusing numeric_limits and isdigit10 on my PC for single and double precision is 6 and 15 respectively.

The plan it to use single precision (there's a valid reason) floating point types and the argument was raised that its quite possible that my my pi representation is useless beyond 6 digits.

Am I led to believe that I'll get truncation beyond 6 decimal places?
What's the approach to creating a test case for checking the round off and truncation errors of both single and double precision types? I'm trying to get a feel for the realtive difference and I tend to struggle with numeric_limits sometimes.

Related articles
  • c++ precision question 2012-03-11

    I have a question about doing arithmetic with C++. Say I have something like: (71/10) - 7.0 and it gives me something like: 0.0999999. How can I get it to give me 0.1.

  • precision question 2012-10-25

    Consider the equation (fly dynamics stuff): Yaw (Degrees) = Azimuth Angle(Radians) * 180 (Degrees) / 3.1415926535897932384626433832795 (Radians) I'm perusing numeric_limits and isdigit10 on my PC for single and double precision is 6 and 15 respective

  • Ubuntu Ok. a precise question that needs a precise answer please. 2013-02-16

    I am about to do an encrypted install with an alternate cd of ubuntu 64 bit. I will be doing the LVM thingy. I have heard it is reccommended to create a seperate partition for your /Home. If I choose to do a guided install and then when i boot up go

  • Oracle sdo_aggr_mbr accuracy/precision question 2013-10-20

    This: select sdo_aggr_mbr( MDSYS.SDO_GEOMETRY(2006,2236,null,MDSYS.SDO_ELEM_INFO_ARRAY(1,2,2,19,4,3,19,2,1,23,2,2,27,2,1,31,2,1),MDSYS.SDO_ORDINATE_ARRAY(893182.9105114,531842.5799909,893182.9560784,531842.6011526,893182.9693087,531842.6496204,893183

  • precise question 2014-06-05

    how does the propositional logic and set theory support computing? answer for me this question.

  • Single-Precision Question 2013-10-26

    I'm using Mastering Visual Basic 2010 <book and it tells me... In a button's Click event handler, declare two variables as follows: Dim a As Single, b As Double Then enter the following statements: a = 1 / 3 Debug.WriteLine(a) Run the application,

  • some questions about RFT 8.0 2012-07-12

    I have some precise questions for RFT 8.0 Java, and thank you in advance for your answers: Is it possible to recover the value of a variable, from a page, and to use it separately? (pass on as value for example). How to pass on values from a script t

  • Ubuntu How to create alsa .deb-packages? 2012-10-06

    I would like to get the newest alsa sources as a .deb package which would allow me to use e.g. Synaptic for installation/deinstallation. So I have started with * downloading and unpacking of e.g. alsa-driver-1.0.14rc2.tar.bz2 in /usr/src/alsa by sudo

  • Ubuntu Dual Boot quesition 2012-10-19

    I got 'problem', to be precise question. Something similiar to this post here DualBoot (, so because there is no answer there maybe someone could help me here. So, I got two hard drives, one with Win oth

  • hosting a vBulletin site 2013-05-07

    I know there have been many threads about hosting issues... I obviously read as much of them as I could. But I have a very precise question here: I plan to host a vBulletin forum. I keep the subject secret for now ;). Just don't worry it has definitl

  • prepared statements with mysql php api and query cache 2014-11-05

    I have a very precise question, I have read in "Beginning with 5.1.17, prepared statements use the query cache under certain conditions, which differ depending on the preparation metho

  • Good and precise guide to asking questions 2012-06-13

    First off, thanks to dagon who posted the link in this post ( While I've allready read a bunch of "before posting, read this", "Smart Questions (

  • beginner questions about math functions and double precision variables 2012-10-07

    I've been programming in Visual Basic 6.0 for years, but I just got Visual yesterday. Unfortunately, the documentation doesn't answer my questions. In the 600-page book Microsoft Visual Step by Step, the index doesn't contain sin or c

  • high precision timer question 2012-01-03

    I have an application where I need a very high precision timer. Normal WM_TIMER messages do not work, as they have a resolution of about 55 ms. I need a timer which would offer a resolution of 1 ms. I came across the QueryPerformanceCounter() functio

  • float to have 2 decimal precision? 2013-03-24

    How do I have float precision of 2 digit. ex: I want this: 10.23 and not 10.2

  • Oracle Quick question on Data size/Precision increase 2013-03-25

    DB version:10gR2 We have NUMBER column TRACK_DTL.ITEM_FS_NBR whose datasize/Precision is 6. TRACK_DTL.ITEM_FS_NBR gets populated by NUMBER column SHIP_MASTER.ITEM_SEQ_NUMBER whose datasize/precision is 8. We were getting 'ORA-01438: value larger than

  • Oracle Precision and scale question 2013-09-10

    So, I was looking through the documentation to see what the difference between NUMBER and INTEGER was, and I came across this def'n of NUMBER which confuses me: NUMBER(p,s) Number having precision p and scale s. The precision p can range from 1 to 38

  • Question on double precision numbers 2014-02-10

    Lets say you made a definition for on a separate file for main lets say it looks something like this (example) double Invoice::getInvoiceAmount(double price, int quantity) { double invoiceAmt = quantity * price; return invoiceAmt; } I tryed to use it

  • Oracle Principal question about Memory Pool behaviour (more precisely) 2014-06-24

    I have observed some interisting behaviour with the Shared Memory Subsystem which I cannot explain by the Docs. Could anyone give me advice on that please? Context - Linux x86_64 Suse 10.0 - BerkeleyDB 4.5.20 Environment - Initialized using the follo

  • Ubuntu amd64 on Dell Precision 490 memory question 2014-07-29

    I have installed Dapper amd64 on my new Dell Precision 490 (x64 3.2 Ghz Intel/4Gb ram) in a breeze. The only extra adaption it needed was the nvidia-glx package in order to run my 2001fp (20") monitor properly. Default hyperthreading was installed, a