480 likes | 670 Views
Levy Sheet Music Project and Optical Music Recognition introducing Gamut. Ichiro Fujinaga McGill University OCVE Workshop (May 2004). Contents. Levy Project Optical Music Recognition Gamera / Gamut Guido / NoteAbility Other Projects. Lester S. Levy Collection. Lester S. Levy Collection.
E N D
Levy Sheet Music Projectand Optical Music RecognitionintroducingGamut Ichiro Fujinaga McGill University OCVE Workshop (May 2004) OCVE 2004 Fujinaga
Contents • Levy Project • Optical Music Recognition • Gamera / Gamut • Guido / NoteAbility • Other Projects OCVE 2004 Fujinaga
Lester S. Levy Collection OCVE 2004 Fujinaga
Lester S. Levy Collection • North American sheet music (1780–1960) • Digitized 29,000 pieces • including “The Star-Spangle Banner” and “Yankee Doodle” • Database of: • text index records • images of music (8bit gray) • lyrics (first lines of verse and chorus) • colour images of cover sheets (32bit) OCVE 2004 Fujinaga
Digital Workflow Management • Reduce the manual intervention for large-scale digitization projects • Creation of data repository (text, image, sound) • Optical Music Recognition (OMR) • Gamera / Gamut • XML-based metadata • composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher • cross-references for various forms of names, pseudonyms • authoritative versions of names and subject terms • Music and lyric search engines • Music analysis toolkit OCVE 2004 Fujinaga
Optical Music Recognition (OMR) • Open-source adaptive OMR system in development since 1984 • Staff recognition and removal • Run-length coding • Projections • Lyric removal • Stems and notehead removal • Music symbol classifier (trainable) • Score reconstruction OCVE 2004 Fujinaga
OMR: Classifier • Connected-component analysis • Feature extraction, e.g.: • Width, height, aspect ratio • Number of holes • Central moments • k-nearest neighbor classifier • Genetic algorithm OCVE 2004 Fujinaga
Overall Architecture for OMR Image File Staff removal Segmentation Recognition K-NN Classifier Output Symbol Name Optimization Genetic Algorithm K-nn Classifier Knowledge Base Feature Vectors Best Weight Vector Off-line OCVE 2004 Fujinaga
The problem • Suitable OCR for lyrics not found • Commercial OCR systems are often inadequate for non-standard documents • The market for specialized recognition of historical documents is very small • Researchers performing document recognition often “re-invent” the basic image processing wheel OCVE 2004 Fujinaga
The solution • Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications • Generalize OMR for structured documents OCVE 2004 Fujinaga
Created Gamera • Framework for creation of structured document recognition system • Designed for domain experts • Image processing tools (filters, binarizations, etc.) • Document segmentation and analysis • Symbol segmentation and classification • Feature extraction and selection • Classifier selection and combiners • Syntactical and semantic analysis Generalized Algorithms and Methods for Enhancement and Restoration of Archives OCVE 2004 Fujinaga
Features of Gamera • Portability (Unix, Windows, Mac) • Extensibility (Python and C++ plugins) • Easy-to-use (experts and programmers) • Open source • Graphic User Interface • Interactive / Batchable (scripts) OCVE 2004 Fujinaga
Architecture of Gamera Scripting Environment (Python) Graphic User Interface (wxWindows) Plugins (Python) Plugins (C++) GAMERA Core (C++) OCVE 2004 Fujinaga
Example of C++ Plugin // Number of pixels in matrix Class area(PluginFunction): self_type = ImageType([ALL]) return_type = FloatVector(“area”, 1) #include “gamera.hpp” using namespace Gamera; template <class T> feature_t area(T &m) { return feature_t(m.nrows() * m.ncols()); } OCVE 2004 Fujinaga
Example of Python function // This filters a list of CC objects from gamera.core import * def filter_wide(ccs, max_width): tmp = [] for x in ccs: if x.ncols() > max_width: x.fill_matrix(0) else: tmp.append(x) return tmp OCVE 2004 Fujinaga
Gamera: Interface(screenshot in Linux) OCVE 2004 Fujinaga
Gamera: Interface(screenshot in Linux) OCVE 2004 Fujinaga
Histogram(screenshot in Linux) OCVE 2004 Fujinaga
Thresholding(screenshot in Linux) OCVE 2004 Fujinaga
Thresholding(screenshot in Linux) OCVE 2004 Fujinaga
OMR reborn as Gamut • Built within the Gamera framwork • Designed by domain experts • Lyric separtation and recognition • Staffline removal routine • Stems and notehead removal • Music symbol segmentation and classification (trainable) • Score reconstruction • Guido (NoteAbility) • Coming soon… MusicXML (Finale, Sibelius) Gamera-basedAdaptiveMusicUnderstandingTools OCVE 2004 Fujinaga
Staff removal: Lute tablature OCVE 2004 Fujinaga
Classifier: Lute(screenshot in Linux) OCVE 2004 Fujinaga
Staff removal: Neums OCVE 2004 Fujinaga
Classifier: Neums(screenshot in Linux) OCVE 2004 Fujinaga
Greek example OCVE 2004 Fujinaga
“A formal language for score-level representation” Plain text: readable, platform independent Extensible and flexible Adequate representation NoteServer: Web/Windows GUIDO/XML NoteAbility (K. Hamel) GUIDO Music Notation FormatH. Hoos, K. Renz, J. Kilian OCVE 2004 Fujinaga
GUIDO: An example { [ \beamsOff | \clef<"treble"> \key<"D"> f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8. g*1/16 | c#2*1/4. b1*1/8 a*1/4. g*1/8 | | e#*1/2 f#*1/4 f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8 g | c#2*1/4. b1*1/8 a*1/4. c#*1/8 ], . . . OCVE 2004 Fujinaga
NoteAbility Demo OCVE 2004 Fujinaga
Summary of Gamera • Gamera allows rapid development of domain-specific document recognition applications • Domain experts can customize and control all aspects of the recognition process • Includes an easy-to-use interactive environment for experimentation • Beta version available on sourceforge.net for Linux, OS X, and Windows OCVE 2004 Fujinaga
Recent Developments • Sheet Music Consortium • OAI (Open Archive Initiative) • Library of Congress • Chopin Early Editions • MODS (Metadata Object Description Schema) • METS (Metadata Encoding and Transmission Standard) • University of Maine OCVE 2004 Fujinaga
Sheet Music Consortium OCVE 2004 Fujinaga
Sheet Music Consortium OCVE 2004 Fujinaga
Library of Congress OCVE 2004 Fujinaga
Chopin Early Editions OCVE 2004 Fujinaga
Chopin Early Editions OCVE 2004 Fujinaga
Maine Music Box OCVE 2004 Fujinaga
TowardsDistributed Digital Music Archives and Libraries(DDMAL) • Open Standards • Open Source • Open File Formats (non-binary) • XML (MODS, METS) • Web Services • UDDI (Universal Description, Discovery, and Integration) • SOAP (Simple Object Access Protocol ) • WSDL (Web Services Description Language) OCVE 2004 Fujinaga
Conclusions • Levy Project • Optical Music Recognition • Gamera /Gamut • Guido / NoteAbility • Other Projects • Distributed Digital Music Archives and Libraries (DDMAL) OCVE 2004 Fujinaga
Acknowledgments • Michael Droettboom (Gamera / Gamut) • Karl McMillan (Gamera) • Robert Ferguson (OS X port) • Keith Hamel (NoteAbility) • JHU Digital Knowledge Center • National Science Foundation • Québec Government • Levy Family OCVE 2004 Fujinaga
Projections X-projections Y-projections back OCVE 2004 Fujinaga
Chopin Early Editions OCVE 2004 Fujinaga
Chopin Early Editions OCVE 2004 Fujinaga
Chopin Early Editions OCVE 2004 Fujinaga
Chopin Early Editions OCVE 2004 Fujinaga