Translation using external sources.
Classes
|
Interface for fetching translations from an external source. |
|
Base class for retrieving translations from an external source. |
|
Fetch from memory. |
|
Fetcher which combines the results of other fetchers. |
|
Fetcher implementation using pandas |
|
Fetch data from a SQL source. |
Bases: ABC, Generic[SourceType, IdType]
Interface for fetching translations from an external source.
Flag indicating whether the fetch_all() operation is permitted.
Return connectivity status. If False, no new translations may be fetched.
Source names known to the Fetcher, such as cities or languages.
Placeholders for sources managed by the Fetcher.
A dict {source: [placeholders..]}.
Notes
Placeholders (and sources) are returned as they appear as they are known to the fetcher (without mapping).
Retrieve placeholder translations from the source.
ids_to_fetch – Tuples (source, ids) to fetch. If ids=None, retrieve data for as many IDs as possible.
placeholders – All desired placeholders in preferred order.
required – Placeholders that must be included in the response.
A mapping {source: PlaceholderTranslations} for translation.
UnknownPlaceholderError – For placeholder(s) that are unknown to the Fetcher.
UnknownSourceError – For sources(s) that are unknown to the Fetcher.
ForbiddenOperationError – If trying to fetch all IDs when not possible or permitted.
ImplementationError – For errors made by the inheriting implementation.
Notes
Placeholders are usually columns in relational database applications. These are the components which are
combined to create ID translations. See rics.translation.offline.Format documentation for details.
Fetch as much data as possible.
placeholders – All desired placeholders in preferred order.
required – Placeholders that must be included in the response.
A mapping {source: PlaceholderTranslations} for translation.
ForbiddenOperationError – If fetching all IDs is not possible or permitted.
UnknownPlaceholderError – For placeholder(s) that are unknown to the Fetcher.
ImplementationError – For errors made by the inheriting implementation.
Bases: Fetcher[SourceType, IdType]
Base class for retrieving translations from an external source.
Users who wish to define their own fetching logic should inherit this class, but there are implementations for
common uses cases. See PandasFetcher for a versatile base fetcher, or
SqlFetcher for a more specialized solution.
mapper – A Mapper instance used to adapt placeholder names in sources to wanted names, ie
the names of the placeholders that are in the translation Format being used.
allow_fetch_all – If False, an error will be raised when fetch_all() is called.
Map placeholder names to the actual names used in source.
source – The source to map placeholders for.
placeholders – Desired placeholders.
candidates – A subset of candidates (placeholder names) in source to map with placeholders. None
=retrieve using get_placeholders().
clear_cache – If True, force a full remap.
A dict {wanted_placeholder_name: actual_placeholder_name_in_source}, where
actual_placeholder_name_in_source will be None if the wanted placeholder could not be mapped to any of
the candidates available for the source.
Return the ID column for source.
Return the Mapper instance used for placeholder name mapping.
Raise an error if offline.
ConnectionStatusError – If not online.
Source names known to the Fetcher, such as cities or languages.
Placeholders for sources managed by the Fetcher.
A dict {source: [placeholders..]}.
Notes
Placeholders (and sources) are returned as they appear as they are known to the fetcher (without mapping).
Get placeholders for source.
Flag indicating whether the fetch_all() operation is permitted.
Retrieve placeholder translations from the source.
ids_to_fetch – Tuples (source, ids) to fetch. If ids=None, retrieve data for as many IDs as possible.
placeholders – All desired placeholders in preferred order.
required – Placeholders that must be included in the response.
A mapping {source: PlaceholderTranslations} for translation.
UnknownPlaceholderError – For placeholder(s) that are unknown to the Fetcher.
UnknownSourceError – For sources(s) that are unknown to the Fetcher.
ForbiddenOperationError – If trying to fetch all IDs when not possible or permitted.
ImplementationError – For errors made by the inheriting implementation.
Notes
Placeholders are usually columns in relational database applications. These are the components which are
combined to create ID translations. See rics.translation.offline.Format documentation for details.
Fetch as much data as possible.
placeholders – All desired placeholders in preferred order.
required – Placeholders that must be included in the response.
A mapping {source: PlaceholderTranslations} for translation.
ForbiddenOperationError – If fetching all IDs is not possible or permitted.
UnknownPlaceholderError – For placeholder(s) that are unknown to the Fetcher.
ImplementationError – For errors made by the inheriting implementation.
Retrieve placeholder translations from the source.
instruction – A single instruction for IDs to fetch. If IDs is None, the fetcher should retrieve data for
as many IDs as possible.
Placeholder translation elements.
UnknownPlaceholderError – If the placeholder is unknown to the fetcher.
Bases: AbstractFetcher[SourceType, IdType]
Fetch from memory.
data – A dict {source: PlaceholderTranslations} to fetch from.
Source names known to the Fetcher, such as cities or languages.
Placeholders for sources managed by the Fetcher.
A dict {source: [placeholders..]}.
Notes
Placeholders (and sources) are returned as they appear as they are known to the fetcher (without mapping).
Retrieve placeholder translations from the source.
instruction – A single instruction for IDs to fetch. If IDs is None, the fetcher should retrieve data for
as many IDs as possible.
Placeholder translation elements.
UnknownPlaceholderError – If the placeholder is unknown to the fetcher.
Bases: Fetcher[SourceType, IdType]
Fetcher which combines the results of other fetchers.
Flag indicating whether the fetch_all() operation is permitted.
Placeholders for sources managed by the Fetcher.
A dict {source: [placeholders..]}.
Notes
Placeholders (and sources) are returned as they appear as they are known to the fetcher (without mapping).
Return child fetchers.
Source names known to the Fetcher, such as cities or languages.
Retrieve placeholder translations from the source.
ids_to_fetch – Tuples (source, ids) to fetch. If ids=None, retrieve data for as many IDs as possible.
placeholders – All desired placeholders in preferred order.
required – Placeholders that must be included in the response.
A mapping {source: PlaceholderTranslations} for translation.
UnknownPlaceholderError – For placeholder(s) that are unknown to the Fetcher.
UnknownSourceError – For sources(s) that are unknown to the Fetcher.
ForbiddenOperationError – If trying to fetch all IDs when not possible or permitted.
ImplementationError – For errors made by the inheriting implementation.
Notes
Placeholders are usually columns in relational database applications. These are the components which are
combined to create ID translations. See rics.translation.offline.Format documentation for details.
Fetch as much data as possible.
placeholders – All desired placeholders in preferred order.
required – Placeholders that must be included in the response.
A mapping {source: PlaceholderTranslations} for translation.
ForbiddenOperationError – If fetching all IDs is not possible or permitted.
UnknownPlaceholderError – For placeholder(s) that are unknown to the Fetcher.
ImplementationError – For errors made by the inheriting implementation.
Return action to take when multiple fetchers return translations for the same source.
Return action to take when multiple fetchers claim the same source.
Bases: AbstractFetcher[str, IdType]
Fetcher implementation using pandas DataFrame s as the data format.
Fetch data from serialized DataFrame s. How this is done is determined by the read_function. This is typically
a Pandas function such as pandas.read_csv() or pandas.read_pickle(), but any function that accepts a
string source as the first argument and returns a data frame can be used.
read_function – A Pandas read-function.
read_path_format – A formatting string or a callable to apply to a source before passing them to read_function.
Must contain a source as its only placeholder. Example: data/{source}.pkl. Leave as-is if None.
read_function_args – Additional positional arguments for read_function.
read_function_kwargs – Additional keyword arguments for read_function.
See also
The official Pandas IO documentation
Read a DataFrame from a source path.
source_path – Path to serialized DataFrame.
A deserialized DataFrame`.
Search for source paths to pass to read_function using read_path_format.
A dict {source, path}.
IOError – If files cannot be read.
Placeholders for sources managed by the Fetcher.
A dict {source: [placeholders..]}.
Notes
Placeholders (and sources) are returned as they appear as they are known to the fetcher (without mapping).
Retrieve placeholder translations from the source.
instruction – A single instruction for IDs to fetch. If IDs is None, the fetcher should retrieve data for
as many IDs as possible.
Placeholder translation elements.
UnknownPlaceholderError – If the placeholder is unknown to the fetcher.
Bases: AbstractFetcher[str, IdType]
Fetch data from a SQL source. Requires SQLAlchemy.
connection_string – A SQLAlchemy connection string. Read from environment variable if connection_string starts
with ‘@’ followed by the name. Example: @TRANSLATION_DB_CONNECTION_STRING reads from
the TRANSLATION_DB_CONNECTION_STRING environment variable.
password – Password to insert into the connection string. Will be escaped to allow for special characters. If
given, the connection string must contain a password key, eg; dialect://user:{password}@host:port. Can
be an environment variable just like connection_string.
whitelist_tables – The only tables the SqlFetcher may access. Mutually exclusive with blacklist_tables.
blacklist_tables – The only tables the SqlFetcher may not access. Mutually exclusive with whitelist_tables.
include_views – If True, discover views as well.
fetch_all_limit – Maximum size of table to allow a fetch all-operation. 0=never allow. Ignore if None.
**kwargs – Primarily passed to super().__init__, then used as selection_filter_type() kwargs.
ValueError – If both whitelist_tables and blacklist_tables are given.
Notes
Inheriting classes may override on or more of the following methods to further customize operation:
get_approximate_table_size(); default isSELECT count(*) FROM table.
make_table_summary(); createsTableSummaryinstances.
selection_filter_type(); control what kind of filtering (if any) should be done when selecting IDs.
Fetch columns from a SQL database.
Placeholders for sources managed by the Fetcher.
A dict {source: [placeholders..]}.
Notes
Placeholders (and sources) are returned as they appear as they are known to the fetcher (without mapping).
Flag indicating whether the fetch_all() operation is permitted.
Parse a connection string. Read from environment if connection_string starts with ‘@’.
Create a table summary.
Return the approximate size of a table.
Called only by the make_table_summary() method during discovery. The default implementation performs a
count on the ID column, which may be expensive.
table – A table object.
id_column – The ID column in table.
An approximate size for table.
Determine the type of filter (WHERE-query) to use, if any.
In the descriptions below, len(table) refers to the TableSummary.size-attribute of table_summary.
Bare select implies fetching the entire table.
ids – IDs to fetch.
table_summary – A summary of the table that’s about to be queried.
fetch_all_below – Use bare select if len(ids) <= len(table).
fetch_all_above_ratio – Use bare select if len(ids) > len(table) * ratio.
fetch_in_below – Always use IN-clause when fetching less than fetch_in_below IDs.
fetch_between_over – Always use BETWEEN-clause when fetching more than fetch_between_over IDs.
fetch_between_max_overfetch_factor – If number of IDs to fetch is between fetch_in_below and
fetch_between_over, use this factor to choose between IN and BETWEEN clause.
One of ('in', 'between', None), where None means bare select (fetch the whole table).
Notes
Override this function to redefine SELECT filtering logic.
Bases: object
Brief description of a known table.
A flag indicating that the FETCH_ALL-operation is permitted for this table.
A flag indicating that the FETCH_ALL-operation is permitted for this table.
Return required and optional columns of the table.
Modules
Errors and warnings related to fethcing. |
|
Supporting functions for implementations. |
|
Types related to translation fetching. |