Opened at 10:13.
Present: Børre, Ciprian, Maja, Sjur, Thomas, Tomi, Trond
Absent: none
Agenda accepted as is.
Oahpa! in Swedish - lexicon or interface, or both?
No money from Norgesuniversitetet, unfortunately.
Meeting memos can be found at http://giellatekno.uit.no/ped/index.html#Meeting+memos
TODO
- Register oahpa.no ( Trond )
- Generate fin/xml/{nouns|verbs|adjectives}.xml, and implement the new Leksa dropdown menu ( Ciprian ).
- Finding a volunteer to translate the sme Leksa lexicon to Swedish ( XXX )
- From the start: sjd_oahpa Leksa in deu and eng as well
- clock and date for Numra ( Trond, Ciprian ).
- Numra for Skolt Sámi ( Trond, Ciprian )
- implement the Bodø-Oahpa ( Ciprian, Trond, Lene )
- email notification when the server goes down ( Ciprian )
HFST issues
Sjur had discussions with Krister Linden on friday, KL working towards version hfst-3. The previous versions as follows: hfst-1 = a working version, hfst-2 = portable to other platforms. hfst-3 is planned jan 1st-ish. The main focus of hfst-3: abstraction and redesign in order to facilitate support for new transducer formalisms. Examples: support FOMA and/or other transducer formats. The point is to see hfst as an umbrella library framwork, where different compilers and libraries can be plugged in. This will give bigger flexibility. Hfst-3 will also be focusing upon speed. After this part (jan 1st) the focus will be upon implementing applications, e.g. spell checking
HFST installation issues:
To check out from svn:
svn co https://hfst.svn.sourceforge.net/svnroot/hfst/trunk hfstsvn
Remove all old Fink (/sw/) and MacPorts (/opt/) installations, then compile under Snow Leopard.
VislCG 3
Everyone should have this one installed as well.
Missing installations:
- Tomi
- Thomas ( Børre )
- Maja ( Børre )
Makefile + tag simplification
Present process:
- tailor the source files (remove lines containing SUB, GG, whatever)
- compile the tmp source files
Goal:
- compile the source files
- tailor the resulting transducer (subtract the strings containing SUB, GG, whatever)
Also replace xfst script files with -e "action" statements.
Old version:
iclock.fst: $(TARGET)/bin/iclock-$(TARGET).fst
$(TARGET)/bin/iclock-$(TARGET).fst: $(TARGET)/bin/clock-$(TARGET).fst
@echo "*** iclock-$(TARGET).fst ***"
@printf "load < $(TARGET)/bin/clock-$(TARGET).fst \n\
invert net \n\
save stack $@ \n\
quit \n" >> tmp/iclock-script-$(TARGET)
$(XFST) < tmp/iclock-script-$(TARGET)
@rm -f tmp/iclock-script-$(TARGET)
New version:
iclock.fst: $(TARGET)/bin/iclock-$(TARGET).fst
$(TARGET)/bin/iclock-$(TARGET).fst: $(TARGET)/bin/clock-$(TARGET).fst
@echo
@echo "*** $(notdir $@) ***"
@echo
@$(XFST) -e "load < $<" \
-e "invert net" \
-e "save stack $@" \
-stop
TODO:
- discuss and decide upon tag format ( linguists )
- convert comment tags to real lexc tags when the discussion is over ( Tomi )
General list
To accomodate future enhancements in different directions (in rough order of importance):
- test bench for all parts of our language technology efforts
- test bench enhanced, but not yet complet
- set up the Leopard Server features for collaborative support:
- permanent chat rooms
- stored (and indexed) chat transcripts of the chat rooms
- iCal server / group calendars
- wiki
- wiki? on G5 (is part of Leopard Server) or other web-based documentation
- improve Forrest stability and i18n support ( the divvun crashes)
-
Sjur has been working on better i18n and pdf rendering
-
Børre has some ideas for getting back to serving static html files
- reorganise the documentation:
- differ between target groups
- get better grouping
- decide what to write in forrest and what in wiki (cf. Apertiumand http://xixona.dlsi.ua.es/apertium/) for a similar split)
- update/add missing parts
- migrate lexc lexicons to XML, splitting the task
- Name lexica (the Name project)
- Dictionaries (already in XML, task is to integrate them)
- At least migrate the lexc open POSes (Komi as a pilot case)
- change the look of the documentation web
- corpus content moved to Max Planck repositories? Norsk språkbank?
- update infrastructure to allow content-restricted spellers for special target groups
TODO:
- install the HFST tools ( Børre )
- upgrade the XServe to Snow Leopard Server ( Børre )
- ask Lene, Linda to install Mikogo ( Trond )
- restructure and clean the script catalogue, using subdirs to categorise the scripts ( Ciprian )
- please act as suggested, cf comments in the news group
- infrastructure remake: ( Børre, Ciprian, Saara, Sjur, Tomi, Trond )
- more modularised make / build infra (prepare for smn, sms, sjd, others)
- merge gt, kt and st into one
- modularised preprocess and spellrelax
- alternatives to make:
- make a test-all target that runs all tests we have ( Ciprian, Sjur, Trond )
- delayed until we have restructured the make/build process
- define and document testing routines ( Ciprian, Sjur, Trond )
- delayed until we have restructured the make/build process
North Sámi
(nothing new, see proofing bugs below)
Lule Sámi
(nothing new, see proofing bugs below)
South Sámi
Three types of problem areas:
- missing closed POSes: errors in transducer, or in PL conversion
- missing words found in the smanob dictionary: 110 verbs, 80 adjectives, all core sma words ( Cip : this statistics is NOT complete, as far as I know Trond has forgoten to take the Swedish verb file into account!)
- missing list from corpus
We also seem to have problems with derivations:
jealadehtedh jieliedehtedh jieledh+V+Der1+Der/dehte+V+Inf
galmadehtedh galmedehtedh galmedidh+V+Der2+Der/ehte+V+Inf
TODO:
- Prepare text´s about normativity issue to SGL/SGM ( Maja, Thomas )
- adjectives ( Maja with Thomas, Trond, Sjur )
- two competing naming conventions of continuation lexicons
- One naming goes ATTRSUFF-PREDSUFF-STEMTYPE
- One follows the sme convention of naming key adjectives
- There are duplicate lexica
- The comparative issue open here and there
- The ATTRSUFF-PREDSUFF-STEMTYPE lexica now go to EVENCOMP and ODDCOMP only.
- work meeting on loan words and proper nouns, this week - tomorrow? ( Maja, Ove (?), Sjur, Thomas, Trond )
Tomi has played with couchdbas a replacement for eXist in risten.no and general dictionary-related work. Seems much lighter and easier to work with.
Instead of building our own webforms and back-end update scripts, use XForms with a premade connection to our xml db. Orbeon XForms is such a tool (open source).
From the meeting with the terminology and IT teams last week:
- no major rework on the present search interface now
- no work on the editing section; instead:
- add existing lists of sanctioned terminology as separate term entities
- add a dictionary if we can make one with sufficient quality
This means the following tasks:
- find already approved lists, in paper or electronic form ( term team )
- convert paper lists to electronic lists ( term team )
- convert lists to standard XML ( Sjur, Tomi )
- add prepared lists to risten.no ( Sjur, Tomi )
TODO:
- send eXist log files to Ciprian ( Sjur )
- fix i18n bug in risten.no/G5 (so they will work without the proper locale request) ( Sjur )
- fix bugs in lexc2xml; add comments to the log element ( Saara )
- finish first version of the editing ( Sjur )
- test editing of the xml files. If ok, then: ( Sjur, Thomas, Trond )
- make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) ( Sjur, Saara )
- convert propernoun-($lang)-lex.txt to a derived file from common xml files ( Sjur, Tomi, Saara )
- implement data synchronisation between risten.noand the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
- start to use the xml file as source file
- clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) ( Thomas, linguists )
- merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset ( linguists )
- publish the name lexicon on risten.no ( Sjur )
- add missing parallel names for placenames ( linguists )
- add informative links between first names like Niillas and Nils ( linguists )
Dictionaries
- sma:nob-swe
- over 400 multiplied entries found (these are coused by a not so clean work with the data)
- done: deleted automatically about 300 of them
- ongoing: Maja-Lisa is checking the entries on correctness
- ongoing: Maja-Lisa is translating the missing words from her own diploma thesis into nob
- ongoing: synchronize data from dictionary with that from lexC files so that we are able to analyze everything from the dictonary ( Cip : this should be valuable for ALL dictionaries)
- fkv:nob and nob:fkv
- started to work on them: waiting for Verena to get rid of multiplied entries
- kom:fin-eng
- moved the original kom-lex.xml to the inc-dir and froze it
- split it by pos into the working_file dir, the ONLY place to work with the dictionary entries
- now, the lexC files are generated via XSLT sheets, no perl scripts
- adjusted the Makefile
- prepared the pipeline for compiling the mac dict
- todo: make a pipeline for StarDict also (as far as I know, Jaska has) a Linux machine
TODO:
- set up risten.no on eXist/XServe (as a beta version site) ( Sjur )
- set up required infra for smenob on risten.no/XServe ( Sjur )
- Continue the dictionary infrastructure discussion ( Ciprian, Sjur, Trond )
- end user documentation (how to download and install) ( Ciprian, Trond )
- Contact Davvi Girji about cooperation on electronic dictionaries ( Sjur )
- developing the mobile phone version of smenob:
- Komi
- take out the doublets to a separate file ( Ciprian )
- merge the doublets ( Jaska, Trond )
- Completing the automaton to some state ( Trond, Jaska, Paula )
- compile Komi dictionary ( Cirian )
- compile Kven dictionary ( Cirian )
South Sámi
TODO:
- Go through the phonrules file ( Thomas )
- Pick a text for proofreading ( Thomas )
- Proofread a text with tre eror$error notation ( Maja )
- difftest for fst and PL speller ( Sjur )
- Compile a new version by Friday ( Tomi )
-
cd $GTHOME/prooftools/; sudo make mac-dmg
- Look for external beta testers?
- David and Jovsset will get the speller, for other people we must wait.
- fix the installer build issues ( Sjur )
- correct easter egg to say "South Sámi", not "North Sámi" ( Sjur )
- fix sma installer bugs ( Sjur )
HFST-based proofing tools
Testing
Spelling Error Markup
TODO:
- test new and nested error markup ( Sjur )
- nesting still needs to be tested, depends on new ccat feature
Speller testing
TODO:
- test the error type selection feature in ccat ( Sjur )
Testing open-source Norwegian spellers
Sjur has invited the open-source group to test their spell-checker using our test bench. The response has been positive, we'll see what happens.
We should go to their developer meetings, and present our work and how to work with language technology.
Speller bugs
List of bugs returned from Polderland:
Tag reordering for abbreviations have caused a lot of problems:
smj:
hr.
hr. hr+ABBR+Acc
cand.philol.
cand.philol. cand.philol+ABBR+N+Acc
Per
Per Per+N+Prop+Mal+Sg+Attr
sme:
hr.
hr. hr+N+ABBR+Acc
Per
Per Per+N+Prop+Mal+Sg+Attr
Open issues based on test results:
sme
Version: Davvisámi, version 1.2, 2009-09-18
- 399 - missing numerals (plural forms) - still OPEN
- 425 - X not recognised; single letters were left out - still OPEN
- 435 - roman numbers - inflection of single letter numbers rejected, as well as some complex numbers (but is ok in smj ) - still OPEN
- we should pregenerate all numbers once and for all, and store them in a separate lexicon file
- 461 - REGRESSION: missing suggestion (sáhkki)
- 508 - REGRESSION: accepts smj entries (most likely abbreviation missing)
- 520 - REGRESSION: r9 and š9 not defined (abbr. missing)
- 595 - prefix+name without hyphen ( ovdaLot instead of ovda-Lot ) - still OPEN
- 603 - suomabealdi accepted - still OPEN
- 606 - compound-tags LEXICON VUOHTA - still OPEN
- 613 - short gen. as second compound part - still OPEN
- 619 - numerals and pronouns to NAMÁK and SASJ fails - vihttasoarttat remaining - still OPEN
- 629 - a taking part in compounding without hyphen - still OPEN
- only open case has word A-finálaid compounded
- 647 - numerals+NOUN - still OPEN , open case has uppercase letters
- 648 - unmotivated suggestions with numeral+noun - still OPEN
- 661 - REGRESSION: abbr. not recognized
- 709 - sámedikkeválga accepted - OPEN
- 728 - vowel shortening GenCmp+Left-tagged - still OPEN
- 779 - caseforms of pronoun okatahat - still OPEN
- 785 - does not recognize alphabet-abbr+noun - OPEN
- 802 - NEW: multiword propernouns
- 803 - NEW: FINJU- words accepted single-handed
- 804 - NEW: guovttilogát, njealjilogát
- 805 - NEW: Nouns+acronyms
smj
Version: Julevsáme, version 1.2, 2009-09-20
- 435 - roman number - single letter numbers now recognised
- we should pre-generate all numbers once and for all, and store them in a separate lexicon file
- please note that inflection of single letter numerals is fine in smj , as opposed to sme
- 482 - polardutkamin not recognized - FIXED
- 496 - REGRESSION: unrecognised clitics
- 556 - non-existent word accepted - FIXED
- 594 - lågenanguoktáj not recognized - still OPEN
- 595 - REGRESSION: prefix+name as split comp without hyphen
- 596 - C-giellan is not accepted - still OPEN
- 600 - Gen+hyph compound - FIXED
- 627 - prefix + hyhpen does not get accepted - FIXED
- 647 - numerals+NOUN - still OPEN , open case has uppercase letters
- 648 - unmotivated suggestions with numeral+noun - still OPEN
- 650 - REGRESSION: noun prefix+name compound without hyphen
- 652 - UPPERCASE-typos only get acronym-suggestions - still OPEN
- 692 - numeral-variants - all but one fixed ( gáktsalågenantjuotakta ), but still OPEN
- 744 - REGRESSION: numerals + clitic
- 803 - NEW: VINJU- words accepted single-handed
- 805 - NEW: Nouns+acronyms
TODO:
- document how compounding is controlled in the PLX conversion ( Tomi )
Hyphenator bugs
Open issues based on test results :
sme
Lexicon version: Davvisámi, version 1.2, 2009-09-18
No known issues!
smj
Lexicon version: Julevsáme, version 1.2, 2009-09-20
- 670 - Hard hyphen replaced with soft hyphen: 10-biejvvásattja (the word is not rec.; Bug #711) - still OPEN
sma
Command to test the hyphenator:
preprocess dev/corp/pressemelding.txt | lookup bin/hyph-sma.fst | cut -f2 | \
lookup bin/hyphrules-sma.fst | grep -v '^$' | cut -f2 | uniq | see
TODO:
- fix PL hyphenator errors ( Tomi )
- almost done - one smj bug left
Installer changes
TODO:
- test InDesign installer ( Sjur )
User documentation
TODO:
- InDesign documentation ( Sjur )
- Norwegian translation received from Davvi Girji
1.2 release
Content:
- several smj bug fixes
- lexicalisations
- InDesign Mac & Win
- new OOo beta
- improved installers, at least for Mac, preferably also for Windows
Autumn planning
- speller test project -> Sjur, Børre, Thomas, X
- speech synthesis -> Sjur, Trond, BA as a starter
- dictionaries: smanob, komfineng, fkvnob -> Ciprian, Sjur, Trond
- risten.no -> Ciprian, Sjur, Tomi
- Barents project follow-up meeting
- sme and smj proofing tools, next version -> Thomas, Tomi, Sjur
- HFST-based proofing tools -> Sjur
Text to speech
TODO:
- refine syntax / dependency rules ( Biret Ánne )
- start public tender process ( Sjur )
- find appropriate voices ( Biret Ánne )
MT and CAT
TODO:
- make A-ITE compile and run on Windows ( Ciprian )
Open source
The repository is properly closed/open now, and the availability of the source code should be announced.
TODO:
- announce the availability of our repository on relevant linguistic lists ( Sjur )
Christmas vacation / non-working days
SD: 24.12 and 31.12 are half days, and one of them off. One additional day in between is fully off.
UiT: all days in between are half days.
| Who |
Days off |
| Børre |
off 30th and 31st |
| Ciprian |
no vacation planned |
| Maja |
XXX |
| Sjur |
no vacation planned, non-working days later |
| Thomas |
no vacation planned, non-working days later |
| Tomi |
no vacation planned |
| Trond |
no vacation planned (Sindre rehearsing for revue) |
The next meeting is 21.12.2009, 9.30 Norwegian time.
The meeting was closed at 12:08.