Flamenco Image Browser: Using Metadata to Improve Image Search During Architectural Design

1. Flamenco Image Browser:Using Metadata to Improve Image Search During Architectural Design Ame Elliott Group for User Interface Research (GUIR) & Dept. of Architecture University of California, Berkeley

2. Problem Statement • Architects begin new projects by looking at images of prior projects • How are elementary schools put together? • What’s new in patio materials? • Current image search interfaces are not helpful for image browsing in the early phase of design • Rely on well-formulated textual queries • Ignore context of architecture

3. Scope of the Project • The Flamenco Image Browser is one customization of a toolkit for domain-specific search (the Flamenco projects) • The Flamenco Image Browser is one component of my dissertation Understanding Architects’ Work Practices (ethnography, task analysis, interviews) Prototyping Tools . Hardware: Digital Desk Software: Image Search UI Evaluation Quantitative Qualitative

4. Scope of the Project • The Flamenco Image Browser is one customization of a toolkit for domain-specific search (the Flamenco projects) • The Flamenco Image Browser is one component of my dissertation Understanding Architects’ Work Practices (ethnography, task analysis, interviews) Prototyping Tools . Hardware: Digital Desk Software: Image Search UI Evaluation Quantitative Qualitative

5. Flamenco Image Browser • Interface to a collection of architectural images using metadata about the images to support search • Metadata about the images provides • Query previews giving hints about possible next actions • Ways of broadening or refining search terms

6. Why Use Metadata? • Open questions about its utility • How to use metadata not well understood • Not an obvious thing to do for images • Contrasts with automated content analysis • Defies conventional wisdom • If a picture is worth 1,000 words, let the pictures speak for themselves • Professional architects have different needs than curators • The thumbnail is the whole document, not like text search using a title to stand in for a long article

7. Faceted Metadata Image: Architect: Period: Location: Concept: Planalto Palace Parti Communiste Francais Pantheon Oscar Neimeyer Oscar Neimeyer Jacques-Gabriel Soufflot 20th Century 20th Century 17th & 18th C. Brasilia Paris Paris Stone Curvilinear Stone

8. Faceted Metadata Image: Architect: Period: Location: Concept: Planalto Palace Parti Communiste Francais Pantheon Oscar Neimeyer Oscar Neimeyer Jaques-Gabriel Soufflot 20th Century 20th Century 17th & 18th C. Brasilia Paris Paris Stone Curvilinear Stone

9. Evaluation Methodology • Solicit feedback from architects to determine if faceted metadata is helpful and how to present it • Not evaluating if the current metadata in the system is the right metadata • Lo-Fi evaluation of paper prototype • 1 hour one-on-one with 3 professional architects • Walk-through interactions on a paper computer, users think-aloud (audio recorded), questions about the experience • Informal study of live version • 1 hour one-on-one with 9 architects /grad students, 2 tasks (audio recorded) and a survey

11. 100 images of France in the 1,000 results for stone 734 images of France in the entire collection

12. Results of Lo-Fi Study • The difference between searching within results and changing to a new search confused users • The “breadcrumb” did not help users orient themselves when browsing results • Home >> Gehry, Frank >> Berlin Home >> Berlin • Users were surprised at the outcome and could not anticipate which they were getting (refine or expand). • Image search is exploratory, not reductive • Specific textual queries not important • “If I knew what I wanted to type in the box I’d just go get it myself.” • Expanding search more important than refining is different than

13. Goals for Study of Live Web Site • Make iterative changes to the alternatives to identify useful aspects of the UI – not statistical analysis • Learn to what extent the metadata is useful for searching • How much text is too much? • What kinds of queries will the users do? • Explore how to clarify searching within results vs. starting a new search

14. Informal Study of Live Version • Comparison- a form-based UI to the same collection of images • Task 1 – Free form search • Participants told they are helping to test an image search engine • Asked to talk about a project they’re working on and something they’d like to find • Let them go at it and try to find images they’re interested in – they can ask any questions they want but no formal instructions given • Task 2 – Treasure hunt • Participants given paper copies of 3 images (but no metadata) and told to find them (range from easy to hard) • Exit survey

17. About the System • Web browser used to view Cold Fusion generated pages, queries made to a mySQL database • 36,000 images from the UC Berkeley Architecture Slide library • 5 facets of metadata about each image • Image title, architect, period, location, and concept • Shallow hierarchy for results in architect, period, and location • Concept facet is from the Getty Art and Architecture thesaurus - a 40,000 term hierarchical thesaurus of descriptors of the built environment – this hierarchy not tested in the live version Building Materials Masonry Finishing Materials Stonework Brickwork Shingles

18. Results: Metadata is Helpful • Very positive feedback about Flamenco • All 9 participants named the metadata in the search results area as their favorite aspect of Flamenco • Metadata was successful at giving hints about where to go next • Perceived as useful “These are places I can go from here.”

19. Results: More Metadata Please • Participants asked for more metadata • Although there were complaints about the contents of the metadata, users still wanted more • Longer lists of options (more hints) • Users wanted more control to make very specific searches • Half the participants requested the ability to control order of results with metadata • Juxtapose visible images 2 different ways: • Overview (one image from each project) vs. like together ( all images of a project next to each other) • Different than ranking for text retrieval (precision, recall), but ordering does matter

20. Results: Complaints • The UI was not successful at clarifying searching within results vs. starting a new search • Only 2 of the 9 participants understood the distinction without discussion – but they want to do both • The 1/3 of the participants who couldn’t find a treasure hunt image felt that Flamenco was slow • Corroborates findings that perceived system speed is about finding what you want (Spool ‘00)

21. Enthusiasm for Metadata • Contradicted my suspicions that a sledgehammer and pick- through-the-rubble approach would be preferable • No-one thought there was too much text • Adding more text preferable to adding another row of images

22. Conclusions • Metadata is useful for exploratory tasks • Good at giving hints about where to go next • Architects want to use metadata to get more control • Of results display • Build complex queries • Sometimes the right word is worth more than many pictures

23. Contributions • Increase understanding of what makes a good search interface • Provide insights about how image search differs from text search

24. Acknowledgements • Thanks to • Marti Hearst my advisor • Andy Edmonds for excellent coding • Andy Chou for help with the database • the Berkeley Museum Informatics Project for access to the image collection • the Flamenco project team for search ideas • and the members of GUIR for helpful feedback

25. Thanks!

29. Related Work • “Berrypicking” model of information seeking (Bates) • Information scent (Pirolli) • Query previews (Plaisant & Shneiderman et al) • QBIC & Blobworld (content analysis)

30. Architects’ Image Use Practices • Architects rely on personal image collections and libraries of print media • Images emerge as relevant due to their juxtaposition with other images