Queries

corpus object

corpus – contain utt objects, not filenames to avoid this in e.g. vsm tagger / dt proc.s

for utt_name in speech_corpus:

utterance = Utterance(utt_name)

text and speech in copurs

in vsmtagger.py:

Todo

Add the text from the unspoken parts

others

• separate train and voice
• letter vsm uses cross-word contexts...
• lowercasing should be separated from safetexting

Document whole module like this:

naive.naive_util.add_htk_header(datafile, floats_per_frame, frameshift_ms)

Add HTK header (for user-specified format – 9) to some data in-place

From the HTKBook, p.69:

nSamples – number of samples in file (4-byte integer) sampPeriod – sample period in 100ns units (4-byte integer) sampSize – number of bytes per sample (2-byte integer) parmKind -– a code indicating the sample kind (2-byte integer)

naive.naive_util.all_entries_of_type(sequence, test_type)

If all elements of sequence are of type test_type, return True, else False.

naive.naive_util.config_list(value)

ConfigObj handles reading of string lists without validation, but in the case of 1-item lists, strings will of course be returned. This function does type checking and conversion for this case.

naive.naive_util.final_attribute_name(xpath)

Find the final text element of an xpath which we will assume is the name of an attribute.

TODO: find a better and less error-prone way to do this!

naive.naive_util.find_extension(fname)

Return everything in a string after the last dot

naive.naive_util.fix_data_type(data)

Turn the data into int if possible, then a float, else a unicode

naive.naive_util.flatten_mapping(mapping, sort_by=False, reverse_sort=False)

Turn dict of dicts to list of (key, value) pairs (with unicode keys and values).

Each subdict must contain the same keys -

these (together with lemma_name) will make up header entries.

TODO: assert tab character not inside keys / values

naive.naive_util.get_basename(fname)

Remove path and extenstion

naive.naive_util.get_htk_filelength(datafile)

parse the header of datafile, then return legnth of data according to header in seconds

naive.naive_util.htk_to_ms(htk_time)

Convert time in HTK (100 ns) units to ms

naive.naive_util.int_to_alphabetic(number)

Convert non-negative integer to base 26 representation using uppercase A-Z as symbols. Can use this instead of numbers in feature delimiters because:

– gives shorter full context model names (esp. with many features) – trivially, split-context-balanced.py expects delimiters to contain no digits

naive.naive_util.make_htk_wildcards(n)

HTK wildcards allow item sets like {/feature:?/,*/feature:1?/*} to express “feature < 20” in question definitions. For a given integer max value n, return a list of strings with HTK wildcards matching non-negative integers less than n.

E.g.: make_htk_wildcards(236) gives: [‘?’, ‘??’, ‘1??’, ‘20?’, ‘21?’, ‘22?’, ‘230’, ‘231’, ‘232’, ‘233’, ‘234’, ‘235’]

naive.naive_util.ms_to_htk(ms_time)

Convert time in ms to HTK (100 ns) units

naive.naive_util.read_feature_lexicon(input_fname, dims_to_keep=0)

Assumed format: lemma feature1 feature2 feature3 ... per line Features are numeric.

default dims_to_keep = 0 means keep all

naive.naive_util.read_htk_header(datafile)

Read HTK header of datafile, return ...

From the HTKBook, p.69:

nSamples – number of samples in file (4-byte integer) sampPeriod – sample period in 100ns units (4-byte integer) sampSize – number of bytes per sample (2-byte integer) parmKind -– a code indicating the sample kind (2-byte integer)

naive.naive_util.read_htk_label(fname)

Read HTK label, assume: “start end phone word”, where word is optional. Convert times from HTK units to MS

naive.naive_util.read_htk_state_label(fname)

Read HTK label with state alignment

return [word, [seg, [s1, start, end], [s2, start , end] ... ]]]

naive.naive_util.read_table(fname, n_entries=2)

Function for reading config files, context files etc. Strip comments (#) and empty lines.

naive.naive_util.remove_extension(fname)

Remove everything in a string after the last dot, and the dot itself

naive.naive_util.section_to_config(section)

Take a section from a ConfigObj and make it into new ConfigObj

naive.naive_util.unflatten_mapping(mapping)

Reverse flatten_mapping. Take dict-like object (e.g. config section), assume utf-8 coded

naive.naive_util.unicode_character_to_safetext(char)

work one out. The substitute should be safe to use with applications of interest (e.g. in HTK modelnames), and a perhaps over-cautious subset of ASCII is used for this (uppercase A-Z).

TODO: [make this explanation complete]

To enable

reverse mapping, multicharacter safetexts are delimited with _.

naive.naive_util.unique_append(old_list, new_list)

Add items from new_list to end of old_list if those items are not already in old list – returned list will have unique entries. Preserve order (which is why we can’t do this quicker with dicts).

naive.naive_util.write_bash_config(dict_like, fname)

Write keys & values in dict_like (e.g. ConfigObj) to file to be read as bash config. ConfigObj only writes .ini style files. Basically, remove space around = and add double quotes.

naive.naive_util.write_r_datafile(data, fname)

Take data, in the form of a list of lists like: :

[(0, u’response’, ‘True’), (1, u’token_text’, ‘_COMMA_’) [...]

I.e. feature number, feature name, feature value for each feature on a line. Feature names must be same on each line. Write data file for R where first line is header with feature names, and each line contains feature values for one data point.

naive.naive_util.writelist(data, filename, uni=False)

The default for writing utf-8 is False. This is important because the default should be to write ascii compatible files (for compatibility with HTK etc.)

Document functions like this:

naive.naive_util.unicode_character_to_safetext(char)

work one out. The substitute should be safe to use with applications of interest (e.g. in HTK modelnames), and a perhaps over-cautious subset of ASCII is used for this (uppercase A-Z).

TODO: [make this explanation complete]

To enable

reverse mapping, multicharacter safetexts are delimited with _.

utt mod

main.Utterance.Element()

makeelement(self, _tag, attrib=None, nsmap=None, **_extra)

Creates a new element associated with this parser.

class main.Utterance.Utterance(string, data_type=None, speech_file=None, utterance_location=None, speech_location=None, check_single_text_line=True)

-self.data holds xml structure of the utterance.

Warning

external data? see add_external_data

There are 3 modes by which utterances can be initialised:

• txt mode, in which string argument treated as file containing text of utterance
• utt``mode, in which ``string argument treated as file containing existing XML structure of utterance
• str``mode, in which ``string argument treated as text of the utterance to be initiliased.

Parameters:

• string – if string is the name of an existing file, choose initialisation mode txt or utt based on its extension. If not a filename, use str mode.
• data_type – use this to manually specify mode: must be txt, utt or str.
• speech_file – if given, is assumed to point to file containing natural waveform of utterance – waveform attribute added to utterance structure.
• utterance_location – XXX
• speech_location – XXX
• check_single_text_line – XXX

Warning

UPDATE f utt_location is not None, utt filename is assumed to be relative to this, and only partial path is stored in utt structure.

Warning

UPDATE If speech_location is not None, speech_file is assumed to be relative to this, and only partial path is stored in utt structure.

add_child(child_node)

[Reroute to self.data]

all_nodes()

Get nodes in document order

append(node)

[Reroute to self.data]

archive(fname=None)

Store an archived version of an utterance that will not be overwritten, and also a PDF of a visualisation.

dump_attribute_value_data(regex_string)

Find values for all attributes (matching the supplied regex) from any node of utterance. Do not unique the values (instances not types).

Return e.g. {“attrib1”: [“val1”, “val2”], “attrib2”: [“val1”, “val2”, “val3”]}

dump_features(target_nodes, context_list, return_dict=False)

For each utterance node matching target_nodes, get values for the list of contexts at that node.

get(key)

[Reroute to self.data] Get attribute key’s value at root node of utterance structure.

get_dirname(file_type)

Get the default name for a filetype directory from an utterance’s “utterance_filename”. If utterance_filename is <PATH>/utt/name.utt the dirname for type lab will be <PATH>/lab/. Make the directory if it does not exist already.

get_filename(file_type)

Get the default name for a filetype from an utterance’s utterance_filename. If utterance_filename is <PATH>/utt/name.utt the filename for type lab will be <PATH>/lab/name.lab.

get_utterance_filename()

Get the absolute path of the utt file where this structure is stored. Absolute paths are not stored directly in the structure so that files are portable.

has_attribute(attribute)

[Reroute to self.data]

has_attributes()

Return list of attribute names stored in utterance.

has_external_data(resource_name)

If utt file is <PATH>/utt/name.utt and resource_name is lab, check if <PATH>/lab/name.lab exists as a file.

insert(index, node)

[Reroute to self.data]

iterdescendants()

[Reroute to self.data]

make_from_string(string, speech_file=None)

Instantiate an utterance from the a string representing the utterance’s text

pretty_print()

[Reroute to self.data]

remove(*args)

[Reroute to self.data]

save(fname=None)

Save utterance structure as XML to file.

Parameters:fname – write here if specified, otherwise use utterance_location if it is set.

set(key, value)

[Reroute to self.data] Set attribute key’s value at root node of utterance structure.

text = None

Todo

ever really necessary to check single line when init’ing utterance from text?

visualise(image_file, force_rows=True, full_label=True, highlight_nodes=None)

Use GraphViz to make an image of utterance structure (extension specifies image type).

Parameters:

• force_rows – put nodes with same tag on same vertical level – true by default
• full_label – plot all node attributes – true by default, otherwise, just safetext

xpath(path)

[Reroute to self.data]

class main.Utterance.UtteranceElement

Specialised Element class for utterances, has safe_xpath method. See here: http://lxml.de/1.3/element_classes.html on using custom Element classes

ElementBase(*children, attrib=None, nsmap=None, **_extra)

add_child(child_node)

Add child to utterance node.

get_context_dict(context_list)

Get dict of features made by concatenatting the output of calls to xpath on this node. Return dict like {feature_name: feature_value, ... }

get_context_vector(context_list)

Get values for list of contexts at an utterance node.

Parameters:context_list – e.g.: [(‘name1’, ‘xpath1’), (‘name2’, ‘xpath2’), ... ] Returns:vector of features made by concatenating the output of calls to xpath

on this node. Return list of items like ((name1, value1), (name2, value2), ... )

has_attribute(attribute)

has_children()

Does this UtteranceElement have any child nodes?

pretty_print()

remove_children()

Remove any child nodes this UtteranceElement has

safe_xpath(path)

Provide padding for e.g. end-of-sentence contexts if xpath doesn’t find anything. In order to handle different padding types (e.g. mean vector for VSM features).

The default padding is _NA_ – this will be used for e.g. end-of-sentence phone contexts.

For count based features, xpath gives 0 in sentence-edge positions, which is fine.

utterance()

Get the top-level utterance node of an UtteranceElement.

main.Utterance.to_string(utt)

Warning

USED?

TODO list

Todo

Is there a more proper way to do this?

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/STORAGE/modules/corpus_utterance.rst, line 25.)

Todo

ever really necessary to check single line when init’ing utterance from text?

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/scripts/main/Utterance.py:docstring of main.Utterance.Utterance.text, line 1.)

Todo

Add the text from the unspoken parts

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/STORAGE/queries.rst, line 20.)

Todo

ever really necessary to check single line when init’ing utterance from text?

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/scripts/main/Utterance.py:docstring of main.Utterance.Utterance.text, line 1.)

Todo

Add some notes on corpus here – holds collections of utterances and text.

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/initial_voice.rst, line 52.)

Todo

tidy this up a bit

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/initial_voice.rst, line 63.)

Todo

Come back and make this recipe functional

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/refinements.rst, line 159.)

Todo

Add details

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/refinements.rst, line 173.)

Todo

Add information

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/refinements.rst, line 186.)

Todo

Add information

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/refinements.rst, line 201.)

Todo

Add information

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/refinements.rst, line 216.)

Todo

Add information

(The original entry is located in /Users/owatts/repos/ossian_git_clean/Ossian/doc/source/refinements.rst, line 229.)