Locale Support in Qt 5: Difference between revisions

Latest revision as of 09:41, 5 September 2017

WARNING: This page is an out of date "WORK-IN-PROGRESS" and so may not be complete or accurate.

Early Qt localization support was weak with a design based on Windows, but with its own code and data to allow to consistent cross-platform support in an era when localization was consistently poor on all platforms. Over time new features have been introduced, but the current QLocale implementation now lags a long way behind modern localization libraries. It is incomplete, inconsistent, not fully integrated with the host system locale, and unnecessarily bloats the core library. It is clear that Qt needs a new approach.

This page documents the current state of Qt localization and researches possible solutions to the issues.

Current Support

Qt provides two core options for localization:

System Locale: Use the default system locale settings, the implementation being platform specific.
Custom Locale: An application can create any custom locale supported by the CLDR data embedded in QtCore which is then used by Qt's internal code routines. These may or may not match the host's available locales.

In theory, the System Locale should use the facilities of the host system, falling back to Qt's CLDR-based facilities where the host system doesn't provide the required data or API. In practice, Windows and Mac use an inconsistent mixture of the host facilities and Qt's CLDR-based facilities. On all other platforms, such as Linux and Android, Qt's built-in CLDR-based facilities are always used.

A number of key requirements for future improvements have been identified:

Minimise locale data shipped with Qt
Minimise localization code needing to be maintained by Qt
Integrate fully with the user/system locale on all platforms
Integrate fully with any user level overrides
Add support for time zones
Add support for calendar systems
Add support for collation
Add support for advanced formatting/parsing options such as spell-out, ordinal, duration, etc.

These can be summarised as reducing the Qt code base, integrating fully with the host platform, and supporting advanced features.

In the past, Qt has emphasised consistent behaviour across all platforms as being the key driver, at the expense of Qt apps looking and behaving the same as all other apps on the users platform, i.e. making life easier for the developer rather than the end user. A better balance needs to be found between the two positions, but with the emphasis on fitting in with the users expectations for the fundamentals, i.e. date and number formats must be consistent with the environment the app is running in unless the developer has good reason. This means better using the host facilities for the core localization features, and providing optional advanced features for those apps that need them.

Reference: CLDR, the Unicode Common Locale Data Repository.

Solution Options

A number of options have been investigated for implementing these improvements.

Plan A was to extend the existing QLocale code and data with new code, code donated from KDE, and more data from CLDR. As a test case calendar system support was added using KDE code and CLDR data, but this proved to be unacceptable in terms of the library size required by the new data. It would also result in a lot of new code that Qt would struggle to code and support, especially in difficult areas like collation. It also didn't improve host integration.

Plan B was to utilise ICU as the localization back-end on all platforms to minimise code and data requirements within Qt. This had the advantage of a single code base and consistent behaviour and feature set across platforms. The main disadvantage was that ICU does not respect a users personal preference overrides or even the host settings, so could be inconsistent. This option proved unworkable however due to resistance from Windows devs to the extra dependency and download, Apple App Store policies preventing linking to ICU, and Android only shipping the Java version.

Plan C was to implement individual system back-ends for each host platform to fully utilise the host system locale resources. While more code than option 2, and limited to a lowest common denominator feature set, it would at least provide a fully integrated appearance with the host system, and is the only design pattern that will work on all platforms without additional large amounts of code or data or external library dependencies. Unfortunately the lowest common feature set was too low, limited to what the Win32 API in Windows Vista or Windows Embedded 2013 supports, a very limited feature set little better than the current QLocale support. Given that Windows 7 will need to be supported for at least 5 more years (official end-of-support is 2020) then we are unlikely to be be able to add any advanced formatting support into QtCore during the 5.x series, and possibly even a future 6.x series. This will fail to meet the needs of many developers who require advanced formatting.

Plan D is now proposed which is essentially an evolution of Plan C with parts of the Plans A and B merged in. This will implement new backends on all host platforms for a full feature set in a new API. QLocale will wrap this new API as a simplified API for legacy platforms. On platforms where the host API is not sufficient for the new API the option will be provided to use ICU or CLDR instead but will be off by default. It is yet to be determined if the advanced API will be included in QtCore or in a new external library QtGlobalization. This is the option now documented below. Detailed planning for the ICU option 2 can be found at Qt-5-ICU. Many of these details will still apply to the new separate library, host-based plan.

Localization Features

This section describes the different localization features, their current implementation in Qt, and possible changes.

User and System Locales

The System Locale is the default locale configured for the entire system, i.e. the default any new user or a root user would have. The User Locale is the locale chosen by the current user which may differ from the System Locale, and/or may have individual settings customised by the user, e.g. usually date format.

Windows and Mac have System and User locales, but Qt only uses/exposes the User Locale including user customizations.
Unix has System and User locale IDs, but Qt only uses/exposes the User IDs. The user can customize setting groups but not individual settings.

Ideally, QLocale would provide separate static methods for systemLocale() and userLocale(), and a way on UNIX to customize individual overrides.

Custom Locales

A Custom Locale is one where the app developer creates a locale of their own choice, for example if they want to show output in different currencies, languages or locations. Qt currently supports this by shipping a sub-set of the CLDR data for the features it supports. This is only useful for a very small set of developers on Windows at the cost of a lot of data in qtcore. The data is currently required for all other platforms though.

Windows provides a limited set of locales depending on the regional version installed or language packs downloaded, and allows passing of the required locale into it's localization api
Mac and most Unix systems ship the full set of locales from CLDR/ICU so do not need Qt's CLDR data
WinRT and Android may also ship the full set of CLDR data (TODO: confirm)

In short, on all main platforms host data can be utilised to create custom locales, reducing the need to ship Qt's copy of CLDR on those platforms. It can also be argued this is an advanced feature that should be enabled if required by the app developer on platforms that don't provide enough host data, either by enabling Qt's inbuilt data or choosing to build against ICU.

Locale Changes

In theory, all Qt apps should fix the locale used at start-up and not change that locale until the entire UI can be refreshed at the same time. The app should respond to a change locale system event by refreshing the QLocale and then refreshing the GUI. Unfortunately this facility appears to be incomplete on most if not all platforms, with no obvious api available. (More research required).

On Windows and Unix, the locale ID is fixed on creation and an internal refresh call is available.
On Mac, CFLocaleCopyCurrent is used for every call then released, so the cached object may be reused but this is not guaranteed. No internal refresh call is available

This support needs to be properly completed for all platforms.

Number Localization

All number localization is done using the internal Qt code:

System Locales use cached host data for number symbols
Custom Locales use Qt's CLDR data for number symbols
All locales use Qt internal code routines, and libdouble where available
The Qt internal code only supports single-character number symbols whereas CLDR and host data can be multi-character (TODO: check how many actually are)
The Qt internal code does not support alternative number-grouping such as Chinese or Indian.
The QLocale API requires code routines for int64, uint64, and double, all other types can be derived from these
The Qt internal code is reused by other Qt classes QByteArray, QString, QTextStream, QIpAddress, QVersionNumber, QtGui::QValidator, and in external modules QtXmlPatterns and QtDeclarative (TODO: check if really need to use Qt internal routines, i.e. if only doing non-localized C conversions could use host, or if localized could use public QLocale api)

Ideally Qt would not have to implement it's own number localization code as it is complex and hard to get exactly right for all cases and so imposes a maintenance burden and possibly a performance burden. Ideally we would also support advanced number localization options such as Spell-out, Ordinal, etc as done by ICU-based platforms.

There exists 3 options here:

Continue to use our own code, but cleaned-up and enhanced with multi-char and grouping support
Use the host facilities
Use another external library or the standard C/C++ library

A number of factors affect this decision:

Using our own code is a maintenance burden
Using the host localization code may not support the functionality used by QLocale or other Qt code
The standard C/C++ code options only support basic number formatting and not advanced options as in ICU/WinRT/OSX
The Win32 API only supports standard C/C++ library formatting
The ICU C API only supports int32, int64 and double, it does not support unsigned int64 until ICU 52
The standard C/C++ libraries do not integrate with ICU or Qt's own CLDR data and do not provide easy ways to override the settings used

As some platforms cannot support the minimum Qt localization requirements, we must continue to offer an internal version or use an external library that takes the required symbols. It may not make sense to only use this code for selected platforms and host code on others, so it could be used on all platforms and improved to include the required features. Alternatively we could clean up our existing code to match the new C++11 std:: api calls and provide a thin wrapper around whichever is appropriate to use on a given platform.

Currency Localization

Current QLocale support is as follows:

Host system data is used for Mac and Windows System Locales
Qt's copy of CLDR data is used for system locales on all other platforms, and for all custom locales on all platforms
No currency parsing is supported
On Mac and Windows sytem locales currency formatting uses the host facilities
On all other platforms and for custom locales Qt's own code is used
The Qt api requires support for int64, unit64 and double

Parsing support could be added.

Date/Time Localization

Current QLocale support is as follows:

Host system data is used for Mac and Windows System Locales
Qt's copy of CLDR data is used for system locales on all other platforms, and for all custom locales on all platforms
All date parsing is done by Qt's own code, no host or standard library code is used
All custom date formatting (dd-MMM-yyyy etc) is done by Qt's own code, no host or standard library code is used
Fixed format date formatting (LongDate, ShortDate, etc) is done by the host on Mac and Windows, and Qt's own code on all other platforms and custom locales

Ideally Qt would not have to implement it's own date/time localization code as it is complex and hard to get exactly right for all cases and so imposes a maintenance burden and possibly a performance burden. Unlike number localization, it may be possible to use the host facilities for all parsing and formatting, except for CLDR which will still require internal Qt routines.

A number of problems exist with the Qt code:

No support for non-Gregorian Calendar Systems, but the host data and formatting may return names and symbols for these
Time zone names are not properly supported
Not all standard format codes or formats supported
Advanced formatting options not supported
The format codes are a weird hybrid of Windows and Unicode
No easy way to format time with default format but no seconds, or with 24-hour time instead of 12-hour time

Given that most platforms use the Unicode standard format codes, it may make sense to convert all internal code to directly work with this standard and to store the CLDR data in this format instead of converting it first to Qt format. This will improve performance and reduce discrepancies. The only format conversions then required will be on Win32, and when the api is called with Qt format codes.

User Customization on Linux

A major issue that needs solving is user customizations on UNIX desktop platforms. This is of particular interest to KDE developers.

Currently if a UNIX user wishes to customize their locale settings, they are restricted to changing the locale code used for a group of functionality. For example, if a en_GB user wants to always use the ISO date format instead of the default, they need to set LC_TIME to a locale code that has an ISO date format, but otherwise has all the other settings the same as en_GB. This can be hard to find, and may not even exist. Such changes also do not take effect globally until the user has logged out and back in again.

ICU also offers no way for a user to define any customizations, although the app programmer is free to override any settings they want to. The ideal solution would be for ICU to implement a standard way to read in user customizations, but it currently doesn't offer this, or indeed respect user customizations on any other platform like Windows or Mac.

One solution would be for QLocale to read in a file with user-level customizations from a standard location for that desktop or system. This file could be in the CLDR JSON formats so would be in a universal standard and readable by QtCore. When the user uses the desktop locale config program to change a setting, this file would be written out, along with a custom POSIX file for other toolkits to use.

Platform Support

While we cannot use ICU directly on all platforms, all platforms except Win32 extended (Windows 7 and Windows Embedded 2013) use ICU or CLDR data as a base and so have a consistent feature set, object model and api that we can design to. Careful API design using new classes for the new features, an API that can be queried for supported features, and sensible fallback options where a feature is unsupported on minor platforms will allow for different levels of support on different platform versions while allowing the latest versions to make most new features available.

Choosing the supported lowest common set of advanced features is slightly tricky depending on whether we choose the lowest deployment version of every platform supported (currently RHEL supporting ICU 4.2), or choose the most widely installed version (say OSX 10.8 supporting ICU 49), or choose the ideal version we want (say ICU 52) and implement bridging code in the interim.

As at Qt5.7 the minimum deployment platforms are:

Platform	Reference	Official	Community	ICU
Windows	7 or Vista?	7 or Vista?	7 or Vista?	Win32 Extended only
Windows Embedded	Compact 2013	Compact 2013	Compact 2013	Win32 Extended only
Windows Phone	8.1	8.1?		Own API
Windows RT	8.1	8.1?		Own API
Mac OSX	10.8	10.8	10.8?	49 via Own API
Mac iOS	8.0	5.0?		? via Own API
Linux Ubuntu	14.04	14.04	11.10?	4.4 or 52
Linux RHEL	6.6	6.6	?	4.2
Linux OpenSuse	13.1	13.1	?	51
Android	4.1 (API Level 16)	4.1 (API Level 16)		4.8 via Own API
QNX			SDP 6.6	?

ICU

ICU ships standard on all modern Linux and BSD distros, and is shipped standard with QNX. With POSIX locale functions being clearly deficient (i.e. no calendar system support, poor collation, etc) then this is the preferred back-end for use on all Unix-like platforms, including embedded and QNX. One limiting factor is that the C++ API is not stable between versions so the C API must be used instead, and this lacks many of the features of the C++ API.

The defining platform here is RHEL 6.6 which only uses ICU 4.2, but it could be argued that RHEL is mostly a server distro and so localization is perhaps not as high a profile requirement there, so a later target of ICU 51 (OpenSUSE 13.1) could be used with a degraded feature-set on RHEL.

Handling of user-level overrides could occur at app level, but a platform wide solution should be sought (e.g. so KDE can define desktop-wide overrides).

Linux	ICU
RHEL 6.6	4.2
Ubuntu 11.10 LTS (Oneiric)	4.4
Ubuntu 12.04 LTS (Precise)	4.8
Ubuntu 14.04 LTS (Trusty)	52
OpenSuse 13.1	51
OpenSuse 13.2	53
QNX 6.6.0	?

Mac OS X / iOS

Mac uses ICU for localization, but does not ship the ICU headers and prohibits apps that link to ICU from the App Store. Instead we must use the native Mac API which in reality is a thin wrapper around ICU with a simplified API and some Mac convenience methods added. The Mac API also uses any user level overrides which using ICU directly would not. One problem is that the wrapper API's may not have been updated to provide access to new features in each ICU release used.

From Qt 5.7 onwards OSX 10.8 and ICU 49 are the lowest supported version.

OSX	ICU
10.6	4.0
10.7	4.6
10.8	49
10.9	51
10.10	53
10.11	55

Reference: http://opensource.apple.com/

Android

Android uses ICU for localization, but only ships the Java version of the library and data files. We will use the Java Android api via JNI.

From Qt 5.7 onwards Android 4.1 and ICU 4.8 are the lowest supported version.

Version	Codename	API Level	ICU
2.3.3	Gingerbread	10	4.4
3.0	Honeycomb	11	4.4
4.0	Ice Cream Sandwich	14	4.6
4.1	Jelly Bean	16	4.8
4.3	Jelly Bean	18	50
4.4	Kit Kat	19	51
5.0	Lollipop	21	53

Reference: http://developer.android.com/reference/java/util/Locale.html

Win32

Windows Vista, Windows 7 and Windows Embedded 2013 continue to use the Win32 API for localization functions, supplemented with some new functions using Locale Names in place of LCID's and improved calendar and custom locale data access. Unfortunately, the core number and date formatter calls have not changed and so remain the lowest common denominator.

Windows Vista/7 API:
Windows Embedded Compact 7 API: https://msdn.microsoft.com/en-US/library/gg155954%28v=winembedded.70%29.aspx and https://msdn.microsoft.com/en-US/library/ee491599%28v=winembedded.70%29.aspx
Windows Embedded 2013 API: https://msdn.microsoft.com/en-US/library/gg155954.aspx and https://msdn.microsoft.com/en-US/library/ee491599.aspx

Windows Runtime

WinRT and Windows Phone provide advanced localization functions clearly based on CLDR data and broadly comparable to ICU. An initial review of the WinRT api indicates all required features are exposed, but this needs to be fully documented.

API Docs: https://msdn.microsoft.com/en-us/library/windows/apps/windows.globalization.aspx

Embedded / CLDR / Fallback

Some embedded platforms may prefer not to ship ICU, or may only have very simple localization requirements. It will be required to provide a fall-back implementation for these platforms. This could be the existing QLocale code and database, but that could be a substantial maintenance burden. The alternative is to ship a simple C-locale back-end or a pass-through to POSIX. One option is to provide good documentation on building the minimal ICU required to support the embedded platform's locales.

Solution Design

NOTE: This section is constantly changing in response to research and code experiments and may not be settled until enough code is written to be submitted for review!

Custom Locales

Only the custom locales provided by the host system will be supported by default. On platforms that do not provide a full set of custom locales (i.e. Win32) then the option will be provided to include CLDR data or use ICU instead. The CLDR locales included will also be made configurable. Combined with new platform backends this will allow the dropping the CLDR data from most Qt builds, but will require retaining our own parse/format code for when CLDR is used.

Platform Backends

New platform backends will be written using the ICU design pattern of separate classes for locale code, number formatter, and date formatter. The implementation will be the same as QTimeZone, with a common public class with multiple switchable private backends, The backends will be implemented as QSharedData private classes derived from a common base class. The base class will implement a minimal C locale. Derived backends will include CLDR, ICU, Win32, WinRT, Android, and Mac. Compile time switches will determine which get built, with multiple backends supported at once, i.e. Win32 can use host services for system locale and optionally CLDR or ICU for a full set of custom locales. The cached global default QLocalePrivate will be replaced by the cached new private classes, with QLocalePrivate being a thin wrapper referring to these and performing any translation required.

It is yet to be determined if QLocale will also use the platform backends for the actual formatting or if Qt's internal code will continue to be used.

Public API

The public API will have 3 parts. The existing QLocale read-only API will be retained and extended with the minimum new API required for calendar, grouping, and multi-byte symbol support. A new minimal ICU-style read-only API will be provided to replace QLocale level use in the future. This minimal API will be guaranteed to work on all platforms. The read-only ICU-style API will be extended with derived custom formatter classes to allow customization or format options and the creation of specialised formatters. This extended API will not be guaranteed to work on all platforms and will depend on the host system level of support or choosing to build with ICU. If a platform does not support a feature, this will silently degrade or calling it will have no effect.

For example, QLocale will provide the existing default decimal number parse/format methods, QNumberFormatter will provide a similar decimal style formatter with default-only parse/format methods, and QCustomNumberFormatter will allow modification of the default format options and using other formatter styles like scientific, ordinal and spell-out. The spell-out formatter will not be supported on Win32 and so requesting it will simply return a standard decimal formatter instead. Changing rounding rules or padding will also not be supported on Win32 and changing these settings will have no effect. This degraded custom functionality may not be acceptable in the core library so may need to be moved to a separate library, but this will lead to considerable code duplication and a very small library and so is not recommended.

Note that the private backend classes will implement the full custom formatter in a single class which will be used by all 3 public APIs.

KDE User Customizations

The private default globals that are cached will now have setter methods on them. This should allow the platform plugin for KDE to override individual settings at application start-up.

Development Plan

New API and backend:

Create base classes for new ICU-style API, implementing a basic C locale, and using the existing Qt code routines
Implement an ICU-based backend using ICU parse/format calls, when built with Win32/Mac ensure queries host for system locale settings
Implement a CLDR-based backend using base class parse/format calls
Implement a Mac Cocoa backend using the Mac parse/format calls
Implement a Win32 backend using the Win32 parse/format calls, where features missing degrade gracefully
Implement a WinRT backend using the WinRT parse/format calls
Implement an Android JNI backend using the Android parse/format calls
Have a compile switch to choose -system-locale as 'platform' (i.e. mac, win32, winrt, icu), 'icu' (to force ICU), or 'cldr' (to force CLDR)
Have a compile switch to choose -custom-locale of 'platform', 'icu' or 'cldr'
Both compile switches default to 'system'

Note that the WinRT and Android changes can be implemented later as other backends will be available, but it will cause either bloat (Android using CLDR instead) or loss of functionality (WinRT using Win32 instead),

CLDR Data Import Script:

Modify to list all available locales and calendar systems available
Modify to accept a list of locales and calendar systems to be imported
By default include all locales and calendar systems
Modify to export Unicode date/time formats, not converted to Qt format
Modify to export all required formats, codes and names
Modify to write to new CLDR backend

QLocale:

The local matching and listing code will now be platform based rather than CLDR based, but may need to retain some intelligence
Change QLocalePrivate to refer to the new private backends, either the global defaults or custom locales.
Switch to always using Unicode date format codes internally, only translate in QLocale public API
Add full number grouping and multi-byte symbol support to Qt number parse/format code
Add a few extra date formatting options to date format/parse and public api
Add ISO codes support to public api based on new backends
Add calendar system support using the host facilities with an optional CLDR based fallback (this would bloat the CLDR data, so optional and configurable)

QCalendar

New class based on the common pattern, embedding calendar system and time zone
Perhaps a better name?

QTime Zone

Update QTimeZone support to the new minimum platform features

API Design

The new API will be implemented as a set of new classes completely separate to the existing QLocale class:

QLocaleCode
QNumberFormatter / QCustomNumberFormatter
QCurrencyFormatter / QCustomCurrencyFormatter
QDateTimeFormatter / QCustomDateTimeFomatter
QCalendar
QTimeZone

There are a number of very strong reasons for this split instead of a monolithic backend and API:

This is the design pattern used by ICU, OSX, Windows, and Java that all devs are already familiar with
It is more efficient as ICU splits the locale resource files that are loaded by the Number and DateTime formatters, so a monolithic class would take longer to load
It closely matches the traditional Unix LC_* envvars allowing for separate text, number, currency, time, collation and measurement locales, making it easier and more efficient to implement on Unix
It represents a clear break with the old API and format codes, making it clearer to devs that behaviour and format codes have changed
It prevents API bloat by having a single formatter api for different formatter types rather than multiple calls with prefixes or extra enums
It allows different levels of feature support for different formatter types on different platforms or versions while keeping the same simple api across them all

The formatter classes will provide separate classes for the default system formatters and custom formatters. The default formatter classes will not allow any customisation or overrides, allowing them to remain thread-safe. If temporary overrides were allowed in the api calls then the backend ICU/Mac/WinRT formatter would need to be cloned first to keep it thread-safe, which is a performance hit so should be discouraged. The custom classes should be documented as being more efficient to use for custom needs by retaining them. The custom formatters will be able to be set as the default formatters should the app wish, but at risk of accidentally rendering them no longer thread-safe.

QLocale

The current QLocale will remain mostly unchanged, devs wishing to use the new features must be encouraged to use the new API instead. The main API change will be to integrate calendar systems, number grouping, and multi-byte symbols. No other changes should be exposed to devs.

An alternative is to provide access to the common shared formatters that are standard across all platforms, i.e. QLocale is still the sum of all the localization options, including those like measurement systems that may not have anywhere else to live. It also matches the UNIX LC_* envvars quite well:

QLocale().number().toString(123);
QLocale().currency().toString(123.45);
QLocale().datetime().toString(myDate);
QLocale().calendar().month(myDate) - (convenience pointer to datetime().calendar)
QLocale().timeZone().isDaylightTime(myDate) - (convenience pointer to datetime().calendar)
QLocale().measurementSystem()
etc

In this way the old api can be deprecated immediately.

The backend will see major changes as detailed above. QLocale will essentially be a simplified wrapper around the new API.

QLocaleCode

A new class will encapsulate a locale code and its metadata. Will also provide static access to lists of available locales and ISO codes. May be tricky as every platform seems to have a different idea of what a locale exactly is.

Unicode Standard:

class QLocaleCode
{
public:
    QLocaleCode();
    QLocaleCode(const QString &locale);
    QLocaleCode(QLocale::Language language, QLocale::Country country);
    QLocaleCode(QLocale::Language language, QLocale::Script script, QLocale::Country country);
    QLocaleCode(const QLocaleCode &other);
    ~QLocaleCode();

    QLocaleCode &operator=(const QLocaleCode &other);
    QLocaleCode &operator=(QLocaleCode &&other);

    void swap(QLocaleCode &other);

    bool operator==(const QLocaleCode &other) const;
    bool operator!=(const QLocaleCode &other) const;

    bool isValid() const;

    QString locale() const;
    QString posixLocale() const;
    QString bcp47Locale() const;
    QString localeDisplayName(const QLocaleCode &inLocale = QLocaleCode()) const;

    QLocale::Language language() const;
    QString languageCode() const;
    QString languageDisplayName(const QLocaleCode &inLocale = QLocaleCode()) const;

    QLocale::Script script() const;
    QString scriptCode() const;
    QString scriptDisplayName(const QLocaleCode &inLocale = QLocaleCode()) const;

    QLocale::Country region() const;
    QString regionCode() const;
    QString regionDisplayName(const QLocaleCode &inLocale = QLocaleCode()) const;

    QString variant() const;
    QString variantDisplayName(const QLocaleCode &inLocale = QLocaleCode()) const;

    QString keywordValue(const QString &keyword) const;
    QString keywordDisplayName(const QString &keyword, const QLocaleCode &inLocale = QLocaleCode()) const;
    QString keywordValueDisplayName(const QString &keyword, const QString &value, const QLocaleCode &inLocale = QLocaleCode()) const;

    Qt::LayoutDirection characterOrientation() const;
    Qt::LayoutDirection lineOrientation() const;

    static QString default();
    static void setDefault(const QLocaleCode &locale);

    static QLocaleCode system();
    static QLocaleCode c();

    static QList<QLocaleCode> availableLocaleCodes();
    static QList<QLocaleCode> matchingLocaleCodes(QLocale::Language language, QLocale::Script script, QLocale::Country country);
};

QLocale

http://doc.qt.io/qt-5/qlocale.html

Create or find a locale:

QLocale()
QLocale(const QString & name)
QLocale(Language language, Country country = AnyCountry)
QLocale(Language language, Script script, Country country)
matchingLocales(QLocale::Language language, QLocale::Script script, QLocale::Country country)
setDefault(const QLocale & locale)
system()
c()

Metadata for a locale:

name()
bcp47Name()
language()
country()
script()
nativeCountryName()
nativeLanguageName()
countryToString()
languageToString()
scriptToString()

Text localization:

quoteString()
textDirection()
createSeparatedList(const QStringList & list)

Misc locale stuff:

uiLanguages()
measurementSystem()

ICU

Note: Get list of available locales from each resource type, i.e. available number locales, etc.

Mac

Win32

WinRT

Android

http://developer.android.com/reference/java/util/Locale.html

QNumberFormatter

The base class QNumberFormatter will expose the default decimal number formatter for the default locale only. No settings can be overridden and no alternative styles can be instantiated. This will ensure the class is thread-safe and can be used by QLocale.

The derived class QCustomNumberFormatter will allow for the instantiation of alternative style formatters such as scientific, percentage or spell-out. A static method will return the list of styles available on the current platform. It will also allow for overriding of individual settings.

The derived class QPatternNumberFormatter will allow for the instantiation of a formatter with a given format pattern. It will also allow the overriding of the initial pattern with a new pattern, and allow overriding of other settings not defined by the pattern. This must be a separate class to QCustomNumberFormatter due to ICU only supporting setting the pattern with a style of UNUM_PATTERN_DECIMAL. Note this is supported on ICU and Mac, but Win32, WinRT and Android support is yet to be determined. This class may be dropped as a result.

Rule-based formatters will not be made available, standard rule-sets should be exposed as styles (such as Ordinal and Spell-Out does), with custom rule-sets unlikely to be able to be supported cross-platform. In any event, a separate QRuleNumberFormatter class would be created were cross-platfrom support possible.

Note that while ICU and Mac number formatters support currency formatting in the same class, much of the API does not apply to currency formatting and has no effect, so a separate QCurrencyFormatter class will be implemented to provide a less confusing API. See the following section for more details.

The private backend class will be a single shared implementation for all three classes thinly wrapping the ICU UNUM object and Mac NSNumberFormatter object, etc.

QLocale Integration

QLocale requires the following features to be available for backwards compatible behaviour.

Parse/format must support:

Int64
UInt64
Double

The QLocale::NumberOption flags must be supported:

OmitGroupSeparator - format number without separator (UNUM_GROUPING_USED, usesGroupingSeparator)
RejectGroupSeparator - parse number without separator (UNUM_GROUPING_USED, usesGroupingSeparator)
OmitLeadingZeroInExponent format exponent without leading zero (ICU ?)
RejectLeadingZeroInExponent parse exponent without leading zero (ICU ?)

The QLocale::FloatingPointPrecisionOption flags must be supported:

FloatingPointShortest - Find shortest representation

Double must support the following format codes:

e - format as [-]9.9e[+|-]999
E - format as [-]9.9E[+|-]999
f - format as [-]9.9
g - use e or f format, whichever is the most concise
G - use E or f format, whichever is the most concise

A precision is also specified with the argument format. For the 'e', 'E', and 'f' formats, the precision represents the number of digits after the decimal point. For the 'g' and 'G' formats, the precision represents the maximum number of significant digits (trailing zeroes are omitted).

API

The current API design is as follows:

class Q_CORE_EXPORT QLocale {
    // Pre-defined number styles
    // ICU UNumberFormatStyle
    // Mac NSNumberFormatterStyle
    // DecimalPatternStyle requires own class/api
    // RuleBasedStyle would need own class/api
    enum NumberStyle {
        // ICU and Mac
        DecimalStyle = 1,     // 10.00
        ScientificStyle,      // 1.0E2
        PercentageStyle,      // 10%
        SpelloutStyle,        // ten
        // ICU Only, in OSX Swift but not Cocoa/Carbon
        OrdinalStyle,         // 10th
        DurationStyle,        // 00:00:10
        LastNumberStyle = DurationStyle
    };
    Q_ENUM(NumberStyle)

    // String Symbols and attributes affecting how a number is formatted
    // Selected ICU UNumberFormatSymbol and UNumberTextAttribute
    // Mac not an enum, uses named method calls, also has nilSymbol and negativeInfinitySymbol
    // Also have MonetaryDecimalSeparatorSymbol and MonetaryGroupingSeparatorSymbol, but these should be
    // special cased inside QCurrencyFormatter
    enum NumberSymbol {
        // ICU and Mac
        DecimalSeparatorSymbol = 1,
        GroupingSeparatorSymbol,
        PercentSymbol,
        MinusSignSymbol,
        PlusSignSymbol,
        ExponentialSymbol,
        PerMillSymbol,
        InfinitySymbol,
        NaNSymbol,
        PositivePrefix,
        PositiveSuffix,
        NegativePrefix,
        NegativeSuffix,
        // ICU > 4.6 and Mac
        ZeroDigitSymbol
    };
    Q_ENUM(NumberSymbol)

    // Integer Attibutes affecting how a number is formatted
    // String, bool, enum, and double attributes are handled separately
    // ICU UNumberAttribute
    enum NumberAttribute {
        MaximumIntegerDigits = 1,
        MinimumIntegerDigits,
        MaximumFractionDigits,
        MinimumFractionDigits,
        MaximumSignificantDigits,
        MinimumSignificantDigits,
        Multiplier,
        PrimaryGroupingSize,
        SecondaryGroupingSize
    };
    Q_ENUM(NumberAttribute)

    // Integer Attibutes affecting how a number is formatted
    // String, bool, enum, and double attributes are handled separately
    // ICU UNumberAttribute
    enum NumberFlag {
        ParseLenient = 1,
        ParseToDecimalSeparator,
        FormatIntegerWithDecimalSeparator,
        FormatWithoutGroupingSeparator
    };
    Q_ENUM(NumberFlag)
    Q_DECLARE_FLAGS(NumberFlags, NumberFlag)

    // Decimal Format Mode
    // ICU/Mac usesSignificantDigits
    enum DecimalFormatMode {
        IntegerFractionDigitsMode = 1,
        SignificantDigitsMode
    };
    Q_ENUM(DecimalFormatMode)

    // Rounding modes
    // ICU UNumberFormatRoundingMode
    // Mac NSNumberFormatterRoundingMode
    enum RoundingMode {
        RoundCeiling = 1,
        RoundFloor,
        RoundDown,
        RoundUp,
        RoundHalfEven,
        RoundHalfDown,
        RoundHalfUp,
        // ICU only
        RoundUnnecessary
    };
    Q_ENUM(RoundingMode)

    // ICU UNumberFormatPadPosition
    // Mac NSNumberFormatterPadPosition
    enum PadPosition {
        DefaultPadPosition = 0,
        PadBeforePrefix,
        PadAfterPrefix,
        PadBeforeSuffix,
        PadAfterSuffix
    };
    Q_ENUM(PadPosition)
};

class Q_CORE_EXPORT QNumberFormatter
{
public:
    // Create default decimal formatter in default locale
    QNumberFormatter();
    ~QNumberFormatter();

    QNumberFormatter(const QNumberFormatter &other);
    QNumberFormatter &operator=(const QNumberFormatter &other);

    bool isValid() const;

    QLocaleCode locale() const;

    QLocale::NumberStyle style() const;

    QString positivePattern() const;
    QString negativePattern() const;

    QString symbol(QLocale::NumberSymbol symbol) const;
    qint32 attribute(QLocale::NumberAttribute attribute) const;
    bool flag(QLocale::NumberFlag flag) const;
    QLocale::NumberFlags flags() const;

    QLocale::DecimalFormatMode decimalFormatMode() const;

    // Number Rounding settings
    QLocaleRoundingMode roundingMode() const;
    double roundingIncrement() const;

    // Number Padding settings
    qint32 paddingWidth() const;
    QString padding() const;
    QLocale::PadPosition paddingPosition() const;

    // Format as strings
    QString toString(qint64 from) const;
    QString toString(quint64 from) const;
    QString toString(double from) const;

    // Parse from strings
    qint64 toInt64(const QString &from, bool *ok = Q_NULLPTR, qint32 *pos = Q_NULLPTR) const;
    quint64 toUInt64(const QString &from, bool *ok = Q_NULLPTR, qint32 *pos = Q_NULLPTR) const;
    double toDouble(const QString &from, bool *ok = Q_NULLPTR, qint32 *pos = Q_NULLPTR) const;

    // Available custom styles on this platform
    static QList<QLocale::NumberStyle> availableStyles();
};

class Q_CORE_EXPORT QCustomNumberFormatter : public QNumberFormatter
{
public:
    // Create named style in default or named locale
    QCustomNumberFormatter(QLocale::NumberStyle style, const QLocaleCode &localeCode = QLocaleCode());
    QCustomNumberFormatter(const QCustomNumberFormatter &other);
    ~QCustomNumberFormatter();

    QCustomNumberFormatter &operator=(const QCustomNumberFormatter &other);

    void setSymbol(QLocale::NumberSymbol symbol, const QString &value);
    void setAttribute(QLocale::NumberAttribute attribute, qint32 value);
    void setFlag(QLocale::NumberFlag flag, bool value = true);
    void setFlags(QLocale::NumberFlags flags);

    void setDecimalFormatMode(QLocale::DecimalFormatMode mode);
    void setIntegerFractionDigits(qint32 maximumInteger, qint32 minimumInteger,
                                  qint32 maximumFraction, qint32 minimumFraction);
    void setSignificantDigits(qint32 maximum, qint32 minimum);

    void setRounding(QLocale::RoundingMode mode, double increment = -1.0);
    void setPadding(qint32 width, const QString &padding, QLocale::PadPosition position = QLocale::DefaultPadPosition);
};

// Optional, if we want to support pattern formatters, not Win32?
// ICU style = UNUM_PATTERN_DECIMAL
class Q_CORE_EXPORT QPatternNumberFormatter : public QNumberFormatter
{
public:
    // Create pattern in default or named locale
    QPatternNumberFormatter(const QString &positive, const QLocaleCode &localeCode = QLocaleCode());
    QPatternNumberFormatter(const QString &positive, const QString & negative, const QLocaleCode &localeCode = QLocaleCode());
    QPatternNumberFormatter(const QPatternNumberFormatter &other);
    ~QPatternNumberFormatter();

    QPatternNumberFormatter &operator=(const QPatternNumberFormatter &other);

    void setPattern(const QString &positive, const QString &negative = QString());
    void setFlag(QLocale::NumberFlag flag, bool value = true);
    void setFlags(QLocale::NumberFlags flags);
    void setRoundingMode(QLocale::RoundingMode mode);
};

QCurrencyFormatter

The QCurrencyFormatter class will provide currency formatting and parsing. This class is provided separate to QNumberFormatter in spite of ICU and Mac integrating them into the same class, to simplify the api and save confusion over what options do and do not work in currency styles. Further testing is required to determine which options do work in currency styles.

class QLocale
{
    // Pre-defined number styles
    // ICU UNumberFormatStyle
    // Mac NSNumberFormatterStyle
    enum CurrencyStyle {
        // ICU and Mac
        CurrencySymbolStyle = 1,    // $10.00
        // ICU >= 4.8, Mac Swift but not Cocoa/Carbon
        CurrencyCodeStyle,          // USD10.00
        CurrencyNameStyle,          // 10 US dollars
        // ICU >= 53
        CurrencyAccountingStyle,    // ($10.00)
        // ICU >= 54
        CurrencyCashStyle,          // $10
        LastCurrencyStyle = CurrencyCashStyle
    };
    Q_ENUM(CurrencyStyle)

    enum CurrencySymbol {
        // ICU and Mac
        CurrencyStandardSymbol,
        CurrencyInternationalSymbol,
    };
    Q_ENUM(CurrencySymbol)
}

class Q_CORE_EXPORT QCurrencyFormatter
{
public:
    // Create default currency formatter in default locale
    QCurrencyFormatter();
    ~QCurrencyFormatter();

    QCurrencyFormatter(const QCurrencyFormatter &other);
    QCurrencyFormatter &operator=(const QCurrencyFormatter &other);

    bool isValid() const;

    QLocaleCode locale() const;

    QLocale::CurrencyStyle style() const;

    QString currencyCode() const;
    QString currencySymbol(QLocale::CurrencySymbol symbol) const;

    QString positivePattern() const;
    QString negativePattern() const;

    QString symbol(QLocale::NumberSymbol symbol) const;
    qint32 attribute(QLocale::NumberAttribute attribute) const;
    bool flag(QLocale::NumberFlag flag) const;
    QLocale::NumberFlags flags() const;

    QLocale::DecimalFormatMode decimalFormatMode() const;

    // Number Rounding settings
    RoundingMode roundingMode() const;
    double roundingIncrement() const;

    // Number Padding settings
    qint32 paddingWidth() const;
    QString padding() const;
    QLocale::PadPosition paddingPosition() const;

    // Uses locale default settings
    QString toString(double from) const;
    double toDouble(const QString &from, bool *ok = Q_NULLPTR, int *pos = Q_NULLPTR) const;

    // Format/Parse any currency using this locale settings
    // Format will use the correct number of decimal places for currency code given
    QString toString(double from, const QString &currencyCode) const;
    double toDouble(const QString &from, QString *currencyCode, bool *ok = Q_NULLPTR, int *pos = Q_NULLPTR) const;

    static QList<QLocale::CurrencyStyle> availableStyles();
};

class Q_CORE_EXPORT QCustomCurrencyFormatter : public QCurrencyFormatter
{
public:
    // Create named style in default or named locale
    QCustomCurrencyFormatter(QLocale::CurrencyStyle style, const QLocaleCode &localeCode = QLocaleCode());
    QCustomCurrencyFormatter(const QCustomCurrencyFormatter &other);
    ~QCustomCurrencyFormatter();

    QCustomCurrencyFormatter &operator=(const QCustomCurrencyFormatter &other);

    void setCurrencyCode(const QString & currencyCode);
    void setCurrencySymbol(QLocale::CurrencySymbol symbol, const QString &value);

    void setSymbol(QLocale::NumberSymbol symbol, const QString &value);
    void setAttribute(QLocale::NumberAttribute attribute, qint32 value);
    void setFlag(QLocale::NumberFlag flag, bool value = true);
    void setFlags(QLocale::NumberFlags flags);

    void setDecimalFormatMode(QLocale::DecimalFormatMode mode);
    void setIntegerFractionDigits(qint32 maximumInteger, qint32 minimumInteger,
                                  qint32 maximumFraction, qint32 minimumFraction);
    void setSignificantDigits(qint32 maximum, qint32 minimum);

    void setRounding(QLocale::RoundingMode mode, double increment = -1.0);
    void setPadding(qint32 width, const QString &padding, QLocale::PadPosition position = QLocale::DefaultPadPosition);
};

// Optional, if we want to support pattern formatters, not Win32? May only need QPatternNumberFormatter?
// ICU style = UNUM_PATTERN_DECIMAL
class Q_CORE_EXPORT QPatternCurrencyFormatter : public QCurrencyFormatter
{
public:
    // Create pattern in default or named locale
    QPatternCurrencyFormatter(const QString &positive, const QLocaleCode &localeCode = QLocaleCode());
    QPatternCurrencyFormatter(const QString &positive, const QString & negative, const QLocaleCode &localeCode = QLocaleCode());
    QPatternCurrencyFormatter(const QPatternCurrencyFormatter &other);
    ~QPatternCurrencyFormatter();

    QPatternCurrencyFormatter &operator=(const QPatternCurrencyFormatter &other);

    void setPattern(const QString &positive, const QString &negative = QString());
    void setFlag(QLocale::NumberFlag flag, bool value = true);
    void setFlags(QLocale::NumberFlags flags);
    void setRoundingMode(QLocale::RoundingMode mode);
};

QTimeZone

QTimeZone has been successfully implemented in Qt 5.2 using separate back-ends for each system but a common API. The design of this class will be copied for much of the new API implementation.

A number of new features are still required, including a QEvent for TimeZoneChanged and a QTimeZoneDatabase class to load TZ databases on any platform.

Old design details can be found at http://wiki.qt.io/Qt-5-QTimeZone.

QCalendar

QCalendar will follow the design of QTimeZone to wrap the system provided calendar calculators.

Not all platforms equally support the same set of calendar systems, although this is slowly converging thanks to the increasing use of CLDR and ICU. QCalendar will define an enum for all possible calendar systems, with an API call for availableCalendarSystems() to list the systems available on the host platform.

QCalendar will implement baseline support for as many calendars as possible so that even where a calendar system is not available on a host system it may still be used as an optional calculation class, but not in the formatter.

ICU-based systems integrate the Time Zone into the Calendar to allow for fully accurate conversions of date/times, although this is strictly not necessary. It should allow calendars like Hebrew and Islamic to start their days at sundown or any other time, but doesn't. The wrapper will need to follow this design.

ICU

Supported range in ICU 56: Julian day numbers of -0x7F000000 to +0x7F000000. This corresponds to years from ~5,800,000 BCE to ~5,800,000 CE. Was previously wider range.

Date/Time Components:

UCAL_ERA
UCAL_YEAR
UCAL_MONTH
UCAL_WEEK_OF_YEAR
UCAL_WEEK_OF_MONTH
UCAL_DAY_OF_YEAR
UCAL_DAY_OF_WEEK
UCAL_DAY_OF_WEEK_IN_MONTH
UCAL_AM_PM
UCAL_HOUR
UCAL_HOUR_OF_DAY
UCAL_MINUTE
UCAL_SECOND
UCAL_MILLISECOND
UCAL_ZONE_OFFSET
UCAL_DST_OFFSET
UCAL_YEAR_WOY
UCAL_DOW_LOCAL
UCAL_EXTENDED_YEAR
UCAL_JULIAN_DAY
UCAL_MILLISECONDS_IN_DAY
UCAL_IS_LEAP_MONTH
UCAL_FIELD_COUNT
UCAL_DAY_OF_MONTH

Other:

Min/Max values
Lenient
Gregorian change date
First day of week
Min days in first week
ICU 4.4 Weekday/Weekend
ICU 49 Ambiguous Wall Time
ICU 50 TZ Transitions
ICU 51 Get TZ ID
ICU 52 Windows TZ ID

Mac.

API Docs say enum deprecated since 10.10? OSX 10.11 System Preferences lists all below as available, plus Amete Alem, Umm al-Qura, Islamic Tabular. Perhaps uses locale keyword instead?

NSGregorianCalendar
NSBuddhistCalendar
NSChineseCalendar
NSHebrewCalendar
NSIslamicCalendar
NSIslamicCivilCalendar
NSJapaneseCalendar
NSRepublicOfChinaCalendar
NSPersianCalendar
NSIndianCalendar
NSISO8601Calendar - Doc says not implemented?

ICU C++: http://icu-project.org/apiref/icu4c/classicu_1_1Calendar.html

Calendar	ICU	Mac	Android	WinRT	Win32	KDE
Gregorian	gregorian
Chinese Lunar	chinese
Coptic	coptic
Dangi (Korean Lunar)	dangi (ICU 51)
Ethiopic	ethiopic
Ethiopic Al Amate	???
Indian National	indian
Jalali	persian
Hebrew	hebrew
Islamic Civil	islamic
Islamic Um-al-Qura	islamic-umalqura (ICU 52)
Islamic Tabular	islamic-tbla (ICU 52)
Japanese (Gregorian)	japanese
Taiwan (Gregorian)	taiwan
Thai (Gregorian)	buddist

Locale Support in Qt 5: Difference between revisions

Latest revision as of 09:41, 5 September 2017

Current Support

Solution Options

Localization Features

User and System Locales

Custom Locales

Locale Changes

Number Localization

Currency Localization

Date/Time Localization

User Customization on Linux

Platform Support

ICU

Mac OS X / iOS

Android

Win32

Windows Runtime

Embedded / CLDR / Fallback

Solution Design

Custom Locales

Platform Backends

Public API

KDE User Customizations

Development Plan

API Design

QLocale

QLocaleCode

QLocale

ICU

Mac

Win32

WinRT

Android

QNumberFormatter

QLocale Integration

API

QCurrencyFormatter

QTimeZone

QCalendar

Navigation menu

Search