QOrchestra & FlowML

1. QOrchestra & FlowML High-level audio synthesis tools on top of SAOL Bert Schiettecatte, VUB

2. Introduction (1) This presentation: • Introduce an audio synthesis diagram format (FlowML) • Present a compiler from FlowML to SAOL (FlowML2SAOL) • Present a high-level authoring tool for SAOL (QOrchestra)

3. Introduction (2) The SA (Structured Audio) standard of Mpeg-4 audio consists of 5 elements: • SAOL (SA Orchestra Language): a signal-processing language for describing musical instruments • SASL (SA Score Language): a language for describing a performance using the instruments of a SAOL program

4. Introduction (3) • SASBF (SA Sample Bank Format): allows transmission of sample banks • Scheduler description: used to translate MIDI or SASL to events internally, in a decoder • Reference to MIDI: supported MIDI messages and their meaning

5. Available technology (1) Command-line encoders, decoders, profilers: • saolc: reference implementation of encoder/decoder • sfront: SAOL-to-C compiler with realtime I/O and some profiling support • sarun: compiles SAOL to a program for a virtual machine with a special instruction set, for efficient decoding

6. Available technology (2) Authoring tools: • QOrchestra: high-level synthesis diagram tool with SAOL export function, “non-realtime”, supports re-use of diagrams • CPS: low-level software synthesis tool with SAOL save function, “realtime” • VisOrc: Similar to CPS, with integrated sequencer

7. Available technology (3) IDEs: • SFX: SAOL editing environment with syntax highlighting, auditioning, some profiling support

8. QOrchestra (1) • Goal: experimental visual authoring tool for SAOL (GPL license), drawing space with primitive building blocks, connection tool, library for new blocks, … • Main idea: tool for people who don’t know SAOL, or don’t want to write SAOL code • Also a goal: case study of a complex diagram editor and object-oriented programming technology

9. QOrchestra (2) QOrchestra is NOT: • A low-level SAOL editor (e.g. CPS) • A complete SA authoring environment • A real-time software synthesis application • An IDE • Finished

10. QOrchestra (3)

11. FlowML (1) • QOrchestra project requirement: save, load and re-use synthesis diagrams • Available storage technology: XML (eXtensibleMarkup Language), a way to add meta-information to information (e.g. <author>Bert Schiettecatte</author>)

12. FlowML (2) • Tags for a specific document class are described in an XML DTD (document type definition) • XML technology is freely available, tested, reliable, very easy to use, … • FlowML = XML DTDs describing synthesis diagram formats on a high level + collection of synthesis constructs + synthesis environment-specific extension mechanism (~ C++ pragma’s)

13. FlowML (3) • Solved storage problems in QOrchestra: store diagrams and libraries as FlowML • Generating SAOL code from FlowML is relatively easy • Very easy to understand (stored as text file) • Could be a bridge between various software synthesis systems • Could be the right high-level tool for describing big synthesis systems (<> SAOL)

14. FlowML (4) • “Web-friendly”: easy to create browser plugins • Is not a programming language, but a high-level modelling language, in a way similar to UML (methodology for structuring OO designs) • FlowML is open and free • Extensions for future versions can be proposed and will be examined prior to becoming official (same license as HTML)

15. FlowML (5) FlowML diagram rules (informal): • Multiple outputs per block • Multiple connections per output • Multiple inputs per block • Only one connection per input • All signals are mono

16. FlowML (6) FlowML diagram rules (informal): • Every component instance has a unique ID used for referring • Every definition contains a description and a unique name used for instantiating • 2 types of signals: audio and control • Diagram/component I/O is done through special primitives defined in the standard

17. FlowML (7) Audio synthesis diagram example: Reverb

18. FlowML (8) FlowML code for the reverb diagram <?xml version='1.0'?> <GraphPrototype name="revbfilt"> <Description> Reverb (filtered delay). </Description> <Implementation> <Instance id="audio_in" type="audio_in" x="0" y="0"> <Connection to="in1" from="out" targetid="mixer"/> <Connection to="in1" from="out" targetid="xfade"/> </Instance> <Instance id="fback_in" type="control_in" x="0" y="0"> <Connection to="lvl2" from="out" targetid="mixer"/> </Instance> <Instance id="level" type="control_in" x="0" y="0"> <Connection to="lvl1" from="out" targetid="mixer"/> </Instance> <Instance id="mixer" type="mixer" x="0" y="0"> <Connection to="in" from="out" targetid="dly"/> </Instance> <Instance id="time_in" type="control_in" x="0" y="0"> <Connection to="time" from="out" targetid="dly"/> </Instance>

19. FlowML (9) FlowML code for the reverb diagram (cont’d) <Instance id="dly" type="delay" x="0" y="0"> <Connection to="in2" from="out" targetid="xfade"/> <Connection to="in2" from="out" targetid="mixer"/> </Instance> <Instance id="xfade" type="xfade" x="0" y="0"> <Connection to="in" from="out" targetid="audio_out"/> </Instance> <Instance id="drywet" type="control_in" x="0" y="0"> <Connection to="x" from="out" targetid="xfade"/> </Instance> <Instance id="audio_out" type="audio_out" x="0" y="0"/> </Implementation> </GraphPrototype>

20. SAOL Generator (1) • Originally intended to be a part of QOrchestra, but now a separate program • Reads FlowML synthesis diagrams and generates SAOL code with the behaviour of the diagram • Uses libraries of SAOL opcodes to simplify generated code: every block in the diagram translates to an opcode call

21. SAOL Generator (2) • Will support libraries containing FlowML diagrams and SAOL code (currently only FlowML) • Generated code will require preprocessing (e.g. #include) • Problem: separation authoring tool (including synthesizer) & code generator can cause differences in resulting sound

22. SAOL Generator (3) • Problem: Integrating software synthesizer/code generator might cause a problem: what about intellectual property?

23. SAOL Generator (4) Code generation: opcode declaration generateOpcDecl(Prototype, Program, DeclsDone) VarDecls = new List Code = new List InstsDone = new Set Params = getInputNames(P) Outputs = getOutputNames(P) foreach O in Outputs add(Results, generateOpcResult(Prototype, O, Null, Program, VarDecls, Code, DeclsDone, InstsDone)) add(Code, makeReturn(Results)) return makeOpcDecl(getRate(getElement(Outputs)), getName(P), Params, VarDecls, Code)

24. SAOL Generator (5) Code generation: opcode result (1/3) generateOpcResult(Prototype, N, C, Program, VarDecls, Code, DeclsDone, InstsDone) if isOutput(Prototype, N) X = getConnectionTo(N) return generateOpcResult(Prototype, getFrom(X), getFromConnector(X), Program, VarDecls, Code, DeclsDone, InstsDone) elseif isInput(Prototype, N) return makeVarRef(N) elseif contains(InstsDone, N) if hasMultipleOutputs(N) return makeArrayLookup(last(N), getNumber(N, C)) else return makeVarRef(last(N)) Output instance Input instance Instance already calculated

25. SAOL Generator (6) Code generation: opcode result (2/3) else add(InstsDone, N) if isGraphPrototype(N) and not contains(DeclsDone, N) add(DeclsDone, N) add(Program, generateOpcDecl(N, Program, DeclsDone)) Params = new List ConnectionsTo = getConnectionsTo(N) foreach X in ConnectionsTo add(Params, generateOpcResult(Prototype, getFrom(X), getFromConnector(X), Program, VarDecls, Code, DeclsDone, InstsDone)) A = makeOpcCall(N, Params) R = getRate(N, C) S = getNrOutputs(N) Instance in middle of diagram Satisfy instance inputs

26. SAOL Generator (7) Code generation: opcode result (3/3) if hasMultipleOutputs(N) or hasMultipleConnectionsPerOutput(N) add(Code, makeAssignment(last(N), next(N))) add(Code, makeAssignment(next(N), A) if hasMultipleOutputs(N) add(Vars, makeArrayDecl(last(N), R, S)) add(Vars, makeArrayDecl(next(N), R, S)) return makeArrayLookup(last(N), getNumber(N, C)) else if hasMultipleConnectionsPerOutput(N) add(Vars, makeVarDecl(last(N), R)) add(Vars, makeVarDecl(next(N), R)) return makeVarRef(last(N)) else return A

27. SAOL Generator (8) Generated SAOL code (reverb diagram) aopcode revbfilt(asig audio_in, ksig drywet, ksig fback_in, ksig level, ksig time_in) { asig dly_next, dly_last; dly_last = dly_next; dly_next = delay(mixer(audio_in, level, dly_last, fback_in), time_in); return xfade(drywet, audio_in, dly_last); }

28. The future • FlowML needs input to mature and become really interesting • Free software (e.g. FlowML parser, SAOL and CSound code generators, …) needs to be developed • Serious authoring tools are needed to make Mpeg-4 popular • Artists need to make a difficult transition

29. Conclusion (1) • FlowML might be interesting as a bridge between synthesis systems • FlowML hides much of the engineering side of audio synthesis and provides a high-level description format for musicians • FlowML allows to prototype a synthesizer quickly before optimizing it (e.g. compile to SAOL and hand-optimize)

30. Conclusion (2) • The popularity of SAOL will depend on its support in commercially available software synthesis applications

31. References and contact info • FlowML and QOrchestra http://www.bertschiettecatte.com/ • Mpeg-4 SA at the MLG, MIT Media Lab http://sound.media.mit.edu/mpeg4/ • Mpeg-4 SA book at Berkeley http://www.cs.berkeley.edu/~lazzaro/