1 / 20

Multimodal Plan Representation for Adaptable BML Scheduling

Multimodal Plan Representation for Adaptable BML Scheduling. Dennis Reidsma, Herwin van Welbergen, Job Zwiers. SAIBA. BML. BML behavior and sync points. The BML scheduling process. Scheduling the multimodal behavior plan. The behavior plan. BML

rian
Download Presentation

Multimodal Plan Representation for Adaptable BML Scheduling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal Plan Representation for Adaptable BML Scheduling Dennis Reidsma, Herwin van Welbergen, Job Zwiers

  2. SAIBA

  3. BML

  4. BML behavior and sync points

  5. The BML scheduling process • Scheduling the multimodal behavior plan

  6. The behavior plan • BML • Specifies behaviors and the constraints between them • The multimodal behavior plan • Specified on the basis of, e.g., communicative intent • Incrementally constructed from BML blocks • The motor plan • Executes the behavior plan on the embodiment of the virtual human, using sound, joint rotations, FAPs, … • Making the motor plan flexible • Achieve interpersonal coordination • Allow on the fly adjustments of ongoing behavior • But do not violate the specified constraints

  7. BML constraints • Explicit constraints • Before/after • At • Implicit constraints • Sync points should not occur before the start of their block • Behaviors should have >0 duration • The default sync points (e.g. start, ready, stroke_start, stroke, stroke_end, relax, end) must remain in order

  8. BML constraints • Realizer specific constraints • Due to technical limitation or • Theoretically motivated • Block level constraints • Merge: start block at current time • Append: start as soon as possible after all behaviors in the current plan (but not earlier than current time)

  9. Additional constraints in Elckerlyc • Elckerlycspecific constraints • Whitespace constraint: no unnecessary ‘whitespace’ between behaviors • Append-after(X) block constraint

  10. Flexible plan representation • The (motor) plan • Describes the low level execution of motor behavior on the embodiment of the virtual human: joint rotations, MPEG4 FAP movement, sound, …etc • Tobeflexible, the plan representation must • maintain information about the relationbetween elements in the plan and the original BML expressionsfromwhichtheyoriginated • becapable of expressingall the constraints • becapable of beingmodified… • …in such a way that the (remaining) constraintsautomaticallystaysatisfied

  11. Elckerlyc’s plan representation • Central: the PegBoard • Each sync point is assigned to a TimePeg on this board • Sync points that are connected by an ‘at’ constraint share the same TimePeg • TimePegs can be moved, changing the time of the associated sync(s) • TimePegs provide local timing (irt the start of the BML block) • Each TimePeg is connected to a BMLBlockPeg • BMLBlockPegs provide global timing

  12. Elckerlyc’s plan representation • OffsetPeg: links to anotherTimePeg and moves withit, retaining a time offset • All ‘at’ constraintscanbeexpressed in Pegs, OffsetPegs en Blockpegs • After en beforeconstraintsrequire a specific pair of Pegs • Notimplementedyet • (but: trivial to design and implement)

  13. Example: solving BML to TimePegs • <bml id="bml1"> • <speech id="speech1"> • <text>As you can see on <sync id="s1"> this painting, ... • </text> • </speech> • <gesture id="point1" start="walk1:relax" type="POINT" • target="painting1" stroke="speech1:s1+0.5"/> • <head id="nod1" action="ROTATION" rotation="NOD" stroke="speech1:s1"/> • <locomotion id="walk1" target="painting1"/> • </bml>

  14. Managing block timing

  15. Architecture • Separated parsing, scheduling, execution • Central: the PegBoard • Engines handle unimodal motor plans • Sync points of PlanUnits are connected by TimePegs • Anticipators may make time adjustments through TimePegs

  16. Architecture • The scheduler communicates with a set of engines to set up the multimodal behavior plan • It knows for each BML behavior what Engine handles it • Engines have a standardized interface with functions to: • Add a BML behavior to the motor plan • Remove a BML behavior from the motor plan • Resolve unknown time constraints on a BML behavior given its known time constraints • Check which BML behavior in the motor plan are currently invalid • Note that communication with Engines is in terms of BML behaviors

  17. Managing adjustments of the plan • Interruption of a behavior • Will automatically shorten the block’s end • Satisfies block append constraint • Time adjustment of a sync point • Through its associated TimePeg • Will also move syncs that are connected to it with ‘at’ constraints • Might invalidate other constraints • Each Engine has functionality to check for this • Solution: drop behavior, drop constraint, .. • And communicate back to the behavior planner

  18. Conclusion • Conclusion • BML scheduling can be viewed as a constraint problem • Elckerlyc uses this view to maintain a flexible behavior plan representation • Allows micro (timing) adjustments • But maintains constraints • Enables applications that require tight mutual coordination between a user and a virtual human • These capabilities have been used in: Attentive Speaker, recent version of Virtual Trainer

  19. Thanks for your attention

  20. Further work • Further work • Capture before/after constraints • Multithreaded scheduling

More Related