target_extraction.allen.token_indexers package

Submodules

target_extraction.allen.token_indexers.custom_elmo_indexer module

class target_extraction.allen.token_indexers.custom_elmo_indexer.CustomELMoTokenCharactersIndexer(namespace='elmo_characters', tokens_to_add=None, token_min_padding_length=0)

Bases: allennlp.data.token_indexers.token_indexer.TokenIndexer

Convert a token to an array of character ids to compute ELMo representations. # Parameters namespace : str, optional (default=`elmo_characters`) tokens_to_add : Dict[str, int], optional (default=`None`)

If not None, then provides a mapping of special tokens to character ids. When using pre-trained models, then the character id must be less then 261, and we recommend using un-used ids (e.g. 1-32).

token_min_padding_lengthint, optional (default=`0`)

See TokenIndexer.

as_padded_tensor_dict(tokens, padding_lengths)

This method pads a list of tokens given the input padding lengths (which could actually truncate things, depending on settings) and returns that padded list of input tokens as a Dict[str, torch.Tensor]. This is a dictionary because there should be one key per argument that the TokenEmbedder corresponding to this class expects in its forward() method (where the argument name in the TokenEmbedder needs to make the key in this dictionary).

The base class implements the case when all you want to do is create a padded LongTensor for every list in the tokens dictionary. If your TokenIndexer needs more complex logic than that, you need to override this method.

Return type

Dict[str, Tensor]

count_vocab_items(token, counter)

The Vocabulary needs to assign indices to whatever strings we see in the training data (possibly doing some frequency filtering and using an OOV, or out of vocabulary, token). This method takes a token and a dictionary of counts and increments counts for whatever vocabulary items are present in the token. If this is a single token ID representation, the vocabulary item is likely the token itself. If this is a token characters representation, the vocabulary items are all of the characters in the token.

get_empty_token_list()

Returns an already indexed version of an empty token list. This is typically just an empty list for whatever keys are used in the indexer.

Return type

Dict[str, List[Any]]

tokens_to_indices(tokens, vocabulary)

Takes a list of tokens and converts them to an IndexedTokenList. This could be just an ID for each token from the vocabulary. Or it could split each token into characters and return one ID per character. Or (for instance, in the case of byte-pair encoding) there might not be a clean mapping from individual tokens to indices, and the IndexedTokenList could be a complex data structure.

Return type

Dict[str, List[List[int]]]

class target_extraction.allen.token_indexers.custom_elmo_indexer.ELMoCharacterMapper(tokens_to_add=None)

Bases: object

Maps individual tokens to sequences of character ids, compatible with ELMo. To be consistent with previously trained models, we include it here as special of existing character indexers. We allow to add optional additional special tokens with designated character ids with tokens_to_add.

beginning_of_sentence_character = 256

beginning_of_sentence_characters = [258, 256, 259, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260]

beginning_of_word_character = 258

bos_token = '<S>'

convert_word_to_char_ids(word)

Return type

List[int]

end_of_sentence_character = 257

end_of_sentence_characters = [258, 257, 259, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260, 260]

end_of_word_character = 259

eos_token = '</S>'

max_word_length = 50

padding_character = 260

Module contents