Yes. When defining font sizes in a design system you should use REM. Otherwise the design system becomes a single point of failure on accessibility for product which uses it.
To make it easy, you can divide pixel font sizes by the base font-size of your product. The default for the browser is 16px, so if you don't override it, you just divide your pixel sizes by 16 to get the REM value.
So 16px is 1rem because 16 / 16 = 1.
19px is 1.1875rem, because 19 / 16 = 1.1875.
So you can work it all out how you want it in pixels, and then just convert it all to REM if it's easier. If you're overriding your font sizes, like: html { font-size: 19; } then you'd adjust your math based on that.
For Swift apps, instead of hard coding font sizes, you can use predefined styles like 'body', 'headline' and 'subheadline', these will adjust based on the accessibility options on IOS.
I'm not well versed in Flutter, but just looking at the docs, you should be able to use the MediaQuery widget. Like:
Text(
'Some Text',
style: TextStyle(fontSize: 16 * MediaQuery.of(context).textScaleFactor),
)