testing the effectiveness of digital storytelling

1. testing the effectiveness of digital storytelling Peter Spyns – Flemish dept. of Economy, ScienceandInnovation

2. Agenda • initialproblem on introducing digital storytelling forfundingagencies • organisingan experiment on digital story telling in a call for project funding • aim • policy context • set-up • outcomes • comments • discussion • outlook • conclusions

3. initialsituation • con’s of digital storytelling • potentialadditionalthreshold • time • technical set up • adequate content • good scenario, style • favoursalready “strong”/experiencedsubmitters • pro’s • one image is worth a 1000 words (Frederick R. Barnard in 1921) • not all submitters are excellent writers • for some themes or topics, words fall short • problem: how to convince external funding/facilitating agencies/organisations that digital storytelling has an added value for them and their stakeholders and proposal submitters ?

4. aim of the experiment • organise a “real life” context to apply digital storytelling • find out howpopular digital storytelling already is • investigate in a serious way the effect of digital storytelling • collect evidence-basedarguments to convincefundingorganisations

5. context of the experiment • EWI call forproposals on socialinnovation • what: definition of socialinnovation • SI aims in a structural way at meeting a societalneed, such as tacklingmultidimensionalsocietalproblems of the most vulnerablegroups in society; • SI is about a new or significantlyimproved product, service, process, marketing method or organisational model. •  adequate topic • how: call document excerpt (translated): • OPTIONAL: • Submitters are allowed to add, as an extra support to theirproposal, a (digital) animation, video or selfreferencingpresentation. Bydoingso, submitterscanvisualisebettersomeaspectsof theirproposal (e.g., innovative features, way of putting to use the resultsaimed at, etc.). • Submitterswho want to make advantage of thispossibility are invited to contact Peter.Spyns@ewi.vlaanderen.be to arrange to file transfert that has to bedonebefore the deadline.

6. practical set-up of the experiment • collecting the proposalsand – ifany – the associated digital story • distributing the proposalamongst the reviewers without the digital story • regularreviewingprocess (without the digital story) • uploading digital stories to separate (andprotected) webpage • after the regular review, each expert with a proposalsupportedby a digital story is invited to also have a look at the digital story andsendan e-mail withcommentsabout • whether or not the digital story adds new information • whether or notthis new information makeshim/her change his/her opinion on one or more of the evaluation criteria • whether on notthis leads to a change in score forone or more of the evaluation criteria • ifso, pleasegive a short explanationwhy • a statistical test candeterminewhether or not a change in expert behaviour is significant or not

7. outcomes of the experiment • on 105 eligibleproposals, 11 added a digital story (10,5%) • leadingtheoretically to 23 reviews by 19 different reviewers • 3 proposalswith a digital story wereaccepted (vs. 17 without) • in practice, 4 reviewersrespondedwith 6 reviews •  example: Blue Assist movie (a best practice of FP7 INNOSERV project)

8. commentsby the participatingreviewers • presentation made more clearwhat I guessed but didnotprivde real answerandhence, didnot lead to a change in score or opinion • movie confirmedmyinitialnegative opinion, but didnot lead to a change in score or opinion • the URL for the movie was in the proposalso I already had a look at it: itcontributedpositively to my scores (2x) • proposal was well writtenandcontainedphotosso the movie didnotreallyadd but ratherconfirmedmy opinion – hence, no change in score or opinion • movie merelyconfirmedmyinitial opinion, hence no change in score

9. discussion on the experiment • too few responses for a statistical analysis mostlydue to • lack of time (regular review was delayed) • bad timing (too close to holidays) • digital story already taken into account duringoriginal review • only 10% of the submittersadded a digital story • coulddepend on the natureand topic of the proposal • formanysubmittersit was their first proposal • in most cases, the digital story merelyconfirmed the initialimpression of the reviewer • no change of mind and scores • whenlooked at duringregularreviewing, it was considered as a positive source of additional information

10. outlook andsubsequent steps • participation of “the FlemishSocialInnovationFactory” • has to stimulatesocialentrepreneurshipandsocialinnovation in Flandersbya.o. • capturingideasfor project proposals • supportingsubmitters in elaboratingtheirideasinto project proposals • enriching the project proposal to enhanceitsacceptancepotential • guidingsocial entrepreneurs to setting up a “socialentreprise” • acts as a thematic front-end for the FlemishInnovation Agency (IWT) thatformallyapprovesand funds the submittedproposals • have discovered “themselves” digital storytelling • are interested in becominginvolved in InnoFunandtrying out digital story telling as part of theirregular way of working • in the long term, thatcouldimplythat the IWT accepts a digital story as (additional) part of a proposal • we hope to integratethem in the RIP

11. conclusions • scientificallyinconclusive experiment • more experiments are needed – maybe in a different setting • low participation (submittersandreviewers) • effectiveness of digital story telling is questionable • it is no source/driver of change in opinion or score • at best, it is an extra source to confirman opinion in the making • the “socialinnovationfactory” couldbe a promising route foradopting digital storytelling as a regular part of a project proposalpreparation procedure.

12. Thank you Department of Economy, Science and Innovation (EWI)Koning Albert II-laan 35 box 10, 1030 Brussels www.ewi-vlaanderen.be | info@ewi.vlaanderen.be