Translation primer#

Call diagram#

Green objects are Translator member functions. Red entities belong to the fetcher. Blue indicates a task that is delegated to an object owned by the Translator. The Translator either performs or coordinates the majority of tasks involved in translation. A notable exception to this rule is the placeholder mapping subprocess, which is handled by AbstractFetcher.map_placeholders.

A PandasFetcher is used in the example below, meaning that sources are resolved by searching for files in a given directory, and actually fetching translations means reading files in this directory.

Real applications typically use something like a SQL database instead as the source. Underlying concepts remain the same, no matter how translation data is retrieved.

Translatable data#

We’re translating a “Bite report” a misfortunate petting zoo, shown below.

The first columns indicates who was bitten (a human), the second who bit them (an animal). Since bites are a frequent occurrence, the zoo uses integers IDs instead of plaintext for their bite reports to save space. The Translator doesn’t work on files (see #154), so we’ll use pandas to create a DataFrame that we can translate.

from pandas import DataFrame, read_csv
bite_report: DataFrame = read_csv('biting-victims-2019-05-11.csv')

The Translator knows what a DataFrame is, and will assume that the columns are names to translate.

Translation sources#

The zoo provides reference tables which allows us to make sense of the data. These tables are stored as regular CSV files and contain some basic information about the humans and animals that are referenced in the (for now) unintelligible bite report.

To access these tables, the Translator needs a Fetcher that can read and interpret CSV files. The PandasFetcher is built to perform such tasks.

from rics.translation.fetching import PandasFetcher
fetcher = PandasFetcher(
    read_function=read_csv,
    # Look for .csv-files in the 'sources' sub folder of the current working directory
    read_path_format='./sources/{}.csv'
)

This fetcher will look for CSV files in the sources sub folder of the current working directory, using pandas.read_csv() to deserialize them. Source names will be filenames without the .csv-suffix.

Name-to-source mapping#

The rics.mapping namespace modules are used to perform name-to-source mapping. By default, names and sources must match exactly which is rarely the case in practice. In our case, there are two names that should be matched to one source each.

• Mapping human_id → humans. Mappings like these are common and may be solved using the built-in like_database_table() heuristic.

  from rics.mapping import HeuristicScore
  score_function = HeuristicScore('equality', heuristics=['like_database_table'])
  

• Mapping bitten_by → animals. This is an impossible mapping without high-level understanding of the context. Using and override is the best solution in this case.

  overrides = {'bitten_by': 'animals'}
  

We’re now ready to create the Mapper instance.

from rics.mapping import Mapper
mapper = Mapper(score_function, overrides=overrides)

Translation format#

We must now decide what we want our report to look like once it’s translated. First, we note that the first two columns, 'id' and 'name', are the same for humans and animals. The 'humans' source also has a unique 'title' column (or placeholder). The 'animals' source has a unique 'species' placeholder.

We would like the translations to include as much information as possible, and as such we will use a flexible Format that includes two optional placeholders.

translation_format = '[{title}. ]{name} (id={id})[ the {species}]'

The use of optional blocks (placeholders and string literals surrounded by angle brackets [..]) allows us to use the same translation for humans and animals.

Placeholder mapping#

Analogous to name-to-source mapping, placeholder mapping binds the wanted placeholders of the translation Format to the actual placeholders found in the source.

All placeholder names also match exactly, except for the 'animal_id' placeholder in the 'animals' source. The easiest solution is to use an override. However, as this kind of naming is common, a more generic solution makes sense. There’s no suitable built-in function for this, so we’ll have to create our own. The result is shown in the snippet below.

def smurf_column_heuristic(value, candidates, context):
    """Heuristic for matching columns that use the "smurf" convention."""
    return (
        # Handles plural form that ends with or without an s.
        f'{context[:-1]}_{value}' if context[-1] == 's' else f'{context}_{value}',
        candidates,  # unchanged
    )

smurf_score = HeuristicScore('equality', heuristics=[smurf_column_heuristic])

Placeholder mapping is the responsibility of the Fetcher. The reason for this is that the required mappings are often specific to a single source collection (such as a database). Having separate mappers makes fetching configuration easier to maintain for applications that use multiple fetchers.

# Amend the fetcher we created earlier.
fetcher = PandasFetcher(
   read_csv, read_path_format='./{}.csv',
   mapper=Mapper(smurf_score),  # Add the mapper.
)

With placeholder mapping in place, all the remains is to create the Translator.

Putting it all together#

from rics.translation import Translator
translator = Translator(fetcher, fmt=translation_format, mapper=mapper)
translated_bite_report = translator.translate(bite_report)

Unless inplace=True is passed translate(), always returns a copy.

Translated data#

Staying true to his reputation, Tarzan the cat has claimed the most victims.

Notebooks#

Implementations may be found in the following notebooks: