mirror/littlefs
3
0
Fork 0
mirror of https://github.com/littlefs-project/littlefs.git synced 2025-12-01 12:20:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

scripts: Fixed rounding-towards-zero issue in si/si2 prefixes

Browse Source 
This should be floor (rounds towards -inf), not int (rounds towards
zero), otherwise sub-integer results get funky:

- floor si(0.00001) => 10u
- int   si(0.00001) => 0.01m

- floor si(0.000001) => 1u
- int   si(0.000001) => m (???)
This commit is contained in:
Christopher Haster
2025-05-15 16:08:04 -05:00
parent e606e82ecb
commit 48daeed509
2 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
scripts/plotmpl.py
View File

@ -105,7 +105,7 @@ def si(x):
if x == 0:
return '0'
# figure out prefix and scale
p = 3*int(mt.log(abs(x), 10**3))
p = 3*mt.floor(mt.log(abs(x), 10**3))
p = min(18, max(-18, p))
# format with 3 digits of precision
s = '%.3f' % (abs(x) / (10.0**p))
@ -121,7 +121,7 @@ def si2(x):
if x == 0:
return '0'
# figure out prefix and scale
p = 10*int(mt.log(abs(x), 2**10))
p = 10*mt.floor(mt.log(abs(x), 2**10))
p = min(30, max(-30, p))
# format with 3 digits of precision
s = '%.3f' % (abs(x) / (2.0**p))