---

---

SOFTWARES

• hpylm: Hierarchical Pitman-Yor Process Language Models (in C++)
• pldlm: Power Law Discounting Language Models (using hpylm)
• hdp: Hierarchical Dirichlet Processes (in C++)
• asd: Automatic Syllable Detection (in C)


• See licence and acknowledge information below
• Warning: the softwares have not been thoroughly tested, and are still on development. It is not surprising that if you find bugs. If you do, please feel free to contact me by s.f.huang AT ed.ac.uk. Thanks!

1. hpylm - hierarchical Pitman-Yor process language model

Download

• Program: hpylm
64bit version
• Python Script: runHPYLM
• Source Codes: download

Sample Data

Note: the following sample data (amicorpus.train.data.gz, amicorpus.test.data.gz) are extracted from the AMI Meeting Corpus. All copyrights of the data are belonging to the AMI project. Please check the web page for licence issues. The vocabulary file (amicorpus.vocab) were extracted by the AMIASR team.

• Training Data: amicorpus.train.data.gz
• Testing Data: amicorpus.test.data.gz
• Vocab File: amicorpus.vocab

Examples

• command options for hpylm:

  [cwd@localhost]: hpylm -help
  

• command options for runHPYLM:

  [cwd@localhost]: runHPYLM -h
  

• train an HPYLM using hpylm:

  [cwd@localhost]: hpylm -debug 0 -order 3 -vocab amicorpus.vocab -pydiscount
  -numiter 100 -numsamp 10 -numgap 5 -text amicorpus.train.data.gz -lm amicorpus.hpylm.arpa.gz -write amicorpus.hpymodel -ppl amicorpus.test.data.gz
  writing 50002 1-grams
  writing 154116 2-grams
  writing 416426 3-grams
  PPL on file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
  0 zeroprobs, logprob= -328749 ppl= 105.737 ppl1= 137.998
  
  [cwd@localhost]: ngram -lm amicorpus.hpylm.arpa.gz -ppl amicorpus.test.data.gz
  file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
  0 zeroprobs, logprob= -328749 ppl= 105.737 ppl1= 137.998
  
  [cwd@localhost]: ngram-count -order 3 -vocab amicorpus.vocab -ukndiscount -interpolate -text amicorpus.train.data.gz -lm amicorpus.iknlm.arpa.gz -gt3min 1
  writing 50002 1-grams
  writing 154116 2-grams
  writing 416426 3-grams
  
  [cwd@localhost]: ngram -lm amicorpus.iknlm.arpa.gz -ppl amicorpus.test.data.gz
  file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
  0 zeroprobs, logprob= -333752 ppl= 113.511 ppl1= 148.745
  
  [cwd@localhost]: ngram-count -order 3 -vocab amicorpus.vocab -kndiscount -interpolate -text amicorpus.train.data.gz -lm amicorpus.mknlm.arpa.gz -gt3min 1
  writing 50002 1-grams
  writing 154116 2-grams
  writing 416426 3-grams
  
  [cwd@localhost]: ngram -lm amicorpus.mknlm.arpa.gz -ppl amicorpus.test.data.gz
  file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
  0 zeroprobs, logprob= -331990 ppl= 110.709 ppl1= 144.867
  

• train an HPYLM using runHPYLM in single mode (single machine, single job):

  [cwd@localhost]: runHPYLM --workdir . --taskid single --prog hpylm --mode 0 --vocab amicorpus.vocab --text amicorpus.train.data.gz --ppl amicorpus.test.data.gz  --niters 100 --nsamps 10 --lm amicorpus.hpylm.single.arpa.gz --hpy amicorpus.hpymodel.single --debug 0
  writing 50002 1-grams
  writing 154116 2-grams
  writing 416426 3-grams
  PPLpit on file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
  0 zeroprobs, logprob= -328752 ppl= 105.742 ppl1= 138.004
  
  [cwd@localhost]: ngram -lm ./single/models/amicorpus.hpylm.single.arpa.gz -ppl amicorpus.test.data.gz
  file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
  0 zeroprobs, logprob= -328752 ppl= 105.742 ppl1= 138.004
  

• train an HPYLM using runHPYLM in sequential mode (single machine, multiple jobs):

  [cwd@localhost]: ngram-count -text amicorpus.train.data.gz -gt3min 1 -write amicorpus.train.count.n123
  
  [cwd@localhost]: runHPYLM --workdir . --taskid sequential --prog hpylm --mode 1 --vocab amicorpus.vocab --count amicorpus.train.count.n123 --ppl amicorpus.test.data.gz --nparts 4 --nprocs 4 --niters 100 --lm amicorpus.hpylm.sequential.arpa.gz --hpy amicorpus.hpymodel.sequential --debug 0
  

• train an HPYLM using runHPYLM in parallel mode (multiple machines, multiple jobs) (GridEngine, i.e., qsub, required):

  [cwd@localhost]: runHPYLM --workdir . --taskid parallel --prog hpylm --mode 2 --vocab amicorpus.vocab --count amicorpus.train.count.n123 --nparts 8 --ppl amicorpus.test.data.gz  --niters 100 --lm amicorpus.hpylm.parallel.arpa.gz --hpy amicorpus.hpymodel.parallel --debug 0 
  

Citation

If you are using this program, you may want to put one of the following papers as a citation:

• Songfang Huang and Steve Renals, Hierarchical Pitman-Yor Language Models for ASR in Meetings, In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU'07), pages 124-129, Kyoto, Japan, December 2007.
[ bibtex | pdf | poster | ASRU'07 ]
• Songfang Huang and Steve Renals, A Parallel Training Algorithm for Hierarchical Pitman-Yor Process Language Models, In Proc. Interspeech'09, pages 2695-2698, Brighton, UK, September 2009.
[ bibtex | pdf | poster | INTERSPEECH'09 ]
• Songfang Huang and Steve Renals, Hierarchical Bayesian Language Models for Conversational Speech Recognition, accepted by IEEE Transactions on Audio, Speech and Language Processing, 2009.
[ bibtex | pdf | IEEE Trans. ASLP ]

NEW: 2. pldlm - power law discounting language model (PLDLM) using hpylm:

Examples

1. with strength parameters theta:

[cwd@localhost]: hpylm -order 3 -vocab amicorpus.vocab -text amicorpus.train.data.gz -read-with-mincounts -ebdiscount -gt1min 1 -gt2min 1 -gt3min 1 -lm amicorpus.pldlm.arpa.gz -debug 1 -ppl amicorpus.test.data.gz -eb-use-theta
writing 50002 1-grams
writing 154116 2-grams
writing 416426 3-grams

[ebdiscount] PPL on file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
0 zeroprobs, logprob= -331250 ppl= 109.555 ppl1= 143.271

2. without strength parameters theta:

[cwd@localhost]: hpylm -order 3 -vocab amicorpus.vocab -text amicorpus.train.data.gz -read-with-mincounts -ebdiscount -gt1min 1 -gt2min 1 -gt3min 1 -lm amicorpus.pldlm.arpa.gz -debug 1 -ppl amicorpus.test.data.gz
writing 50002 1-grams
writing 154116 2-grams
writing 416426 3-grams

[ebdiscount] PPL on file amicorpus.test.data.gz: 8777 sentences, 154831 words, 1201 OOVs
0 zeroprobs, logprob= -331529 ppl= 109.988 ppl1= 143.869

Note

• the PLDLM has slightly lower perplexity (109.6) than the IKNLM (113.5) and the MKNLM (110.7).
• the PLDLM works even better than the IKNLM and the MKNLM especially if the domain is somewhat mismatched. So just try it!
• we have experimentally evaluated the PLDLM in terms of perplexity and word error rate on multiparty meetings - basically it works!
• you can manually set strength parameter theta by using -theta1, -theta2, -theta3, ...
• hpylm accepts input from count files, simply change -text to -read-count.

Citation

If you are using this program, you may want to put the following paper as a citation:

• Songfang Huang and Steve Renals, Power Law Discounting for N-Gram Language Models, In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'10), pages xxxx-xxxx, Dallas, Texas, USA, March 2010.
[ bibtex | pdf | slides | ICASSP'10 ]

3. hdp - hierarchical Dirichlet process

Download

• Program: hdp
64bit version
• Source Codes: download

Sample Data

Note: the following sample data (amicorpus.train.data, amicorpus.test.data.gz) are extracted from the AMI Meeting Corpus. All copyrights of the data are belonging to the AMI project. Please check the web page for licence issues.

• Training Data: amicorpus.train.doc.gz
• Testing Data: amicorpus.test.doc.gz
• Config File: config config.role

Examples

• command options for hdp:

  [cwd@localhost]: hdp -help
  [Command]: hdp -help
  Usage of command "options"
   -help-hdp:         print help information for HDP options
   -rand:             the random seed
                  Default value: 0
   -lda:              recover lda in HDP
   -numtopic:         initial number of topics in HDP
                  Default value: 1
   -numiter:          number of iterations to burn in HDP
                  Default value: 500
   -numsamp:          number of samples to collect in HDP
                  Default value: 10
   -numspace:         number of space samples in HDP
                  Default value: 5
   -train:            train file name for HDP
   -test:             test file name for HDP
   -config:           test file name for HDP
   -aa:               Dirichlet prior for HH
                  Default value: 0.05
   -prior-count:      use the prior count instead of aa
   -beta:             beta for adaptive LM
                  Default value: 0.5
   -read-hdp:         write the HDP model to file
   -write-hdp:        write the HDP model to file
   -test-hdp:         evaluate HDP in ppl
   -infer-role:       infer roles using the HDP
   -use-lm-vocab:     use the same vocab with baseline ngram
   -hdp-vocab:        vocab file for the HDP model
   -norm:             use a normalized LM for adaptation
   -prior-count-file: text file for prior counts for H in HDP
   -ngram-count-file: text file for prior counts for ungram to adapt from
   -top:              number of top classes to adapt LM
                  Default value: 0
   -write-likelihood: write the likelihoods for training and testing
   -write-hdp-vocab:  write hdp vocab to file
   -use-adapt:        adapting LM in HDP
   -use-mix:          mixing LM in HDP
   -role-ngram:       estimate a role ngram for each of the four roles
   the default action is to do nothing
   -help:             Print this message
  

• Data format:

  [cwd@localhost]: zcat amicorpus.train.doc.gz | head
   PM   KICKOFF MEETING PROJECT TWENTY MINUTES KIND SURE LAURA PROJECT MANAGER INTRODUCE 
  
   PM   DESIGNING REMOTE CONTROL RECORD ACTUALLY DAVID ANDREW CRAIG ARRIVED DESIGN REMOTE CONTROL SUPPOSED ORIGINAL TRENDY USER FRIENDLY KIND STAGES DESIGN SURE GUYS RECEIVED EMAILS 
  
   ID   PROJECT ANNOUNCEMENT PROJECT DESIGNING REMOTE CONTROL 
  
   PM   INDIVIDUAL MEETING REPEAT PROCESS TIMES POINT WHITEBOARD DRAW FAVOURITE ANIMAL SUM FAVOURITE CHARACTERISTICS 
  
  if no role information available, just use  SENTENCE 
  

• Config file format:

  [cwd@localhost]: zcat config.role
  ROOT BASE
  PM ROOT
  ID ROOT
  UI ROOT
  ME ROOT
  
  for default case, run hdp without -config argument.
  

• Train a topic model of two levels using hdp:

  [cwd@localhost]: hdp -numtopic 20 -numiter 200 -train amicorpus.train.doc.gz -write-hdp amicorpus.2level.hdpmodel.gz -test amicorpus.test.doc.gz -aa 0.5 -debug 1 2>&1 | tee log
  

• Train a topic model of three levels (role-HDP) using hdp:

  [cwd@localhost]: hdp -numtopic 20 -numiter 200 -train amicorpus.train.doc.gz -write-hdp amicorpus.3level.hdpmodel.gz -test amicorpus.test.doc.gz -aa 0.5 -config config.role -debug 1 2>&1 | tee log
  

Citation

If you are using this program, you may want to put the following paper as a citation:

• Songfang Huang and Steve Renals, Modeling Topic and Role Information in Meetings using the Hierarchical Dirichlet Process, In A. Popescu-Belis and R. Stiefelhagen, editors, Machine Learning for Multimodal Interaction V, volume 5237 of Lecture Notes in Computer Science, pages 215-225, Springer, 2008.
[ bibtex | pdf | slides | video | DOI | MLMI'08 | LNCS'5237 ]

Licence

This software was implemented on top of the SRILM toolkit from SRI. All those codes from the SRILM are credited to SRI.

• SRILM toolkit

Acknowledges

• Steve Renals's supervision and supports
• Wolfson Microelectronics Scholarship
• European IST Programme Project FP6-506811 - AMI
• European IST Programme Project FP6-033812 - AMIDA
• Yee Whye Teh's Matlab software for double-checking our implementation.

---

---