Translation of IDs with flexible formatting and name matching.
For and introduction to translation, see Translation primer and Mapping primer.
Classes
|
Translate IDs to human-readable labels. |
|
Create a |
|
Metadata pertaining to how a |
Bases: Generic[NameType, SourceType, IdType]
Translate IDs to human-readable labels.
For an introduction to translation, see the Translation primer page.
The recommended way of initializing Translator instances is the from_config() method. For configuration
file details, please refer to the Translator Configuration Files page.
The Translator is the main entry point for all translation tasks. Simplified translation process steps:
The
mapmethod performs name-to-source mapping (seeDirectionalMapping).The
fetchmethod extracts IDs to translate and retrieves data (seeTranslationMap).Finally, the
translatemethod applies the translations and returns to the caller.
fetcher – A Fetcher or ready-to-use translations.
fmt – String Format specification for translations.
mapper – A Mapper instance for binding names to sources.
default_fmt – Alternative Format to use instead of fmt for fallback translation of unknown IDs.
default_fmt_placeholders – Shared and/or source-specific default placeholder values for unknown IDs. See
InheritedKeysDict.make() for details.
allow_name_inheritance – If True, enable name resolution fallback to the parent translatable when
translating with the attribute-option. Allows nameless pandas.Index instances to inherit the name of
a pandas.Series.
Notes
Untranslatable IDs will be None by default if neither default_fmt nor default_fmt_placeholders is given.
Adding the maximal_untranslated_fraction option to translate() will raise an exceptions if too many IDs
are left untranslated. Note however that this verifiction step may be expensive.
Examples
A minimal example. For a more complete use case, see the DVD Rental Database example. Assume that we have data for people and animals as in the table below:
people: animals:
id | name | gender id | name | is_nice
------+---------+-------- ----+--------+---------
1991 | Richard | Male 0 | Tarzan | false
1999 | Sofia | Female 1 | Morris | true
1904 | Fred | Male 2 | Simba | true
In most real cases we’d fetch this table from somewhere. In this case, however, there’s so little data that we
can simply enumerate the components needed for translation ourselves to create a MemoryFetcher.
>>> from rics.translation import Translator
>>> translation_data = {
... 'animals': {'id': [0, 1, 2], 'name': ['Tarzan', 'Morris', 'Simba'], 'is_nice': [False, True, True]},
... 'people': {'id': [1999, 1991, 1904], 'name': ['Sofia', 'Richard', 'Fred']},
... }
>>> translator = Translator(translation_data, fmt='{id}:{name}[, nice={is_nice}]')
>>> data = {'animals': [0, 2], 'people': [1991, 1999]}
>>> for key, translated_table in translator.translate(data).items():
>>> print(f'Translations for {repr(key)}:')
>>> for translated_id in translated_table:
>>> print(f' {repr(translated_id)}')
Translations for 'animals':
'0:Tarzan, nice=False'
'2:Simba, nice=True'
Translations for 'people':
'1991:Richard'
'1999:Sofia'
Handling unknown IDs.
>>> default_fmt_placeholders = dict(
... default={'is_nice': 'Maybe?', 'name': "Bob"},
... specific={'animals': {'name': 'Fido'}},
>>> )
>>> useless_database = {
... 'animals': {'id': [], 'name': []},
... 'people': {'id': [], 'name': []}
>>> }
>>> translator = Translator(useless_database, default_fmt_placeholders=default_fmt_placeholders,
... fmt='{id}:{name}[, nice={is_nice}]')
>>> data = {'animals': [0], 'people': [0]}
>>> for key, translated_table in translator.translate(data).items():
>>> print(f'Translations for {repr(key)}:')
>>> for translated_id in translated_table:
>>> print(f' {repr(translated_id)}')
Translations for 'animals':
'0:Fido, nice=Maybe?'
Translations for 'people':
'0:Bob, nice=Maybe?'
Since we didn’t give an explicit default_fmt_placeholders, the regular fmt is used instead. Formats can be
plain strings, in which case translation will never explicitly fail unless the name itself fails to map and
Mapper.unmapped_values_action is set to ActionLevel.RAISE.
Create a Translator from TOML inputs.
path – Path to a TOML file.
extra_fetchers – Path to TOML files defining additional fetchers. Useful for fetching from multiple sources or kinds of sources, for example locally stored files in conjunction with one or more databases. The fetchers are ranked by input order, with the fetcher defined in path (if any) being given the highest priority (rank 0).
clazz – Translator implementation to create. If a string is passed, the class is resolved using
get_by_full_name() if a string is given. Use cls if None.
A new Translator instance with a config_metadata attribute.
Return from_config() initialization metadata.
Make a copy of this Translator.
share_fetcher – If True, the returned instance use the same Fetcher.
overrides – Keyword arguments to use when instantiating the copy. Options that aren’t given will be taken
from the current instance. See the Translator class documentation for possible choices.
A copy of this Translator with overrides applied.
NotImplementedError – If share_fetcher=False.
Translate IDs to human-readable strings.
For an introduction to translation, see the Translation primer page.
translatable – A data structure to translate.
names – Explicit names to translate. Derive from translatable if None.
ignore_names – Names not to translate, or a predicate (str) -> bool.
inplace – If True, translate in-place and return None.
override_function –
A callable (name, fetcher.sources, ids) -> ... returning one of
None (use regular mapping logic)
a source to use, or
a split mapping {source: [ids_for_source..]}. This forces IDs to be fetched from
different sources in spite of being labelled with the same name.
maximal_untranslated_fraction – The maximum fraction of IDs for which translation may fail before an error is raised. 1=disabled. Ignored in reverse mode.
reverse – If True, perform translations back to IDs. Offline mode only.
attribute – If given, translate translatable.attribute instead. If inplace=False, the translated
attribute will be assigned to translatable using
setattr(translatable, attribute, <translated-attribute>).
A copy of translated copy of translatable if inplace=False, otherwise None.
UntranslatableTypeError – If type(translatable) cannot be translated.
AttributeError – If names are not given and cannot be derived from translatable.
MappingError – If any required (explicitly given) names fail to map to a source.
MappingError – If name-to-source mapping is ambiguous.
ValueError – If maximal_untranslated_fraction is not a valid fraction.
TooManyFailedTranslationsError – If translation fails for more than maximal_untranslated_fraction of IDs.
ConnectionStatusError – If reverse=True while the Translator is online.
UserMappingError – If override_function returns a source which is not known, and
self.mapper.unknown_user_override_action != 'ignore'.
Map names to translation sources.
translatable – A data structure to map names for.
names – Explicit names to translate. Derive from translatable if None.
ignore_names – Names not to translate, or a predicate (str) -> bool.
override_function –
A callable (name, fetcher.sources, ids) -> ... returning one of
None (use regular mapping logic)
a source to use, or
a split mapping {source: [ids_for_source..]}. This forces IDs to be fetched from
different sources in spite of being labelled with the same name.
A mapping of names to translation sources. Returns None if mapping failed.
AttributeError – If names are not given and cannot be derived from translatable.
MappingError – If any required (explicitly given) names fail to map to a source.
MappingError – If name-to-source mapping is ambiguous.
UserMappingError – If override_function returns a source which is not known, and
self.mapper.unknown_user_override_action != 'ignore'.
Returns raw match scores for name-to-source mapping. See map() for details.
Return translation sources.
Fetch translations.
A TranslationMap.
ConnectionStatusError – If disconnected from the fetcher, i.e. not online.
Return the Fetcher instance used to retrieve translations.
Return the Mapper instance used for name-to-source binding.
Return a TranslationMap of cached translations.
Load or create a persistent fetch_all-instance.
Warning
Experimental method; may change or disappear without warning.
Instances are created, stored and loaded as determined by a metadata file located in the given cache_dir. A
new Translator will be created if:
There is no ‘metadata’ file, or
the original Translator is too old (see max_age), or
the current configuration – as defined by (config_path, extra_fetchers, clazz) – has changed in such a
way that it is no longer equivalent to the
configuration used to create the original Translator.
Note
This method is not thread safe.
config_path – Path to a TOML file. See from_config() for details.
extra_fetchers – Path to TOML files defining additional fetchers. See from_config() for details.
cache_dir – Root directory where the cached translator and associated metadata is stored. Derive based on
config_path if None.
max_age – The maximum age of the cached Translator before it must be recreated. Pass max_age=0 to
force recreation.
clazz – Translator implementation to create. If a string is passed, the class is resolved using
get_by_full_name(). Use cls if None.
A new or cached Translator instance with a config_metadata attribute.
Restore a serialized Translator.
path – Path to a serialized Translator.
A Translator.
TypeError – If the object at path is not a Translator or a subtype thereof.
See also
The Translator.store() method.
Retrieve and store translations in memory.
translatable – Data from which IDs to fetch will be extracted. Fetch all IDs if None.
names – Explicit names to translate. Derive from translatable if None.
ignore_names – Names not to translate, or a predicate (str) -> bool.
delete_fetcher – If True, invoke Fetcher.close() and delete the fetcher after retrieving data. The
Translator will still function, but new data cannot be retrieved.
path – If given, serialize the Translator to disk after retrieving data.
Self, for chained assignment.
ForbiddenOperationError – If Fetcher.fetch_all() is disabled and translatable=None.
MappingError – If map() fails (only when translatable is given).
Notes
The Translator is guaranteed to be serializable() once offline. Fetchers often
aren’t as they require things like database connections to function.
See also
The Translator.restore() method.
Bases: Generic[NameType, SourceType, IdType]
Create a Translator from TOML inputs.
A callable (name, kwargs) -> AbstractFetcher. Overwrite attribute to customize.
A callable (kwargs) -> Mapper. Overwrite attribute to customize.
Create a Translator from a TOML file.
Resolve desired Translator type.
Bases: object
Metadata pertaining to how a Translator instance was initialized from TOML configuration.
Check if this ConfigMetadata is equivalent to other.
Configs are equivalent if:
They have the same
ricsversion, andUse the same fully qualified class name, and
The main configuration files are equal after parsing, and
They have the same number of auxiliary (“extra”) fetcher configurations, and
All auxiliary fetcher configurations are equal after parsing.
other – Another ConfigMetadata instance.
Equivalence status.
Create ConfigMetadata from a JSON string s.
Modules
Integration for insertion and extraction of IDs and translations to and from various data structures. |
|
General errors for the translation suite. |
|
Factory functions for translation classes. |
|
Translation using external sources. |
|
Offline (in-memory) translation classes. |
|
Test implementations. |
|
Types used for translation. |