Workplace Digital Essential Skills in Rural Small Businesses Pilot Project

1. Workplace Digital Essential Skills in Rural Small Businesses Pilot Project SDRC – Digital Essential Skills Assessment Orientation Session, October 30, 2013, Campbellton, New Brunswick

2. Preliminary Steps • Identifying the occupations where participants would be trained in • Identifying the digital tasks participants would be trained to perform

3. Targeted Occupations -1 • Identified occupations where there were would be large numbers of lower literacy workers who use Information and Communications Technologies • Based to some extent on the National Occupational Classification system and the OLES website – identifying required skills relating to Information and Communication Technologies • Confirmed in the pre-ONA conducted in the summer that these were the “right” occupations

4. Targeted Occupations - 2 • Administrative (all sectors): general office, administrative, accounting, purchasing, and production clerks • Production (manufacturing sector): machine operators, assemblers, inspectors, material handlers, labourers, cleaners • Operations (service sector): cleaners, housekeepers, cooks, material handlers, labourers, patient service associates • Client service (all sectors): customer service clerks, receptionists, client service assistance, sales associates

5. Digital Tasks – Desired Attributes • Examined past research – external validity • Advised by Satya Brink, an expert in the field • Derived a framework: a continuum of tasks covering the range of digital activities workers perform in their job • Tasks are common across, but contextualized to, occupational groups • Confirmed in the pre-ONA with participating organizations

6. Digital Tasks • Use • Consult/read documents to the Web or other digital media • Use documents/databases on the web or other digital media • Complete forms on the Web or other digital media • Understand • Perform online commercial transactions • Search databases on the Web or other digital media • Seek, find choose information on the Web • Communicate/Create/Do • Use a calendar on the Web or other digital media • Communicate electronically with co-workers, suppliers, clients to coordinate workplace activities

7. Assessments - Purpose • Administrative purpose (pre-training assessment): to determine which digital tasks an employee lacks the skills to perform and therefore needs digital skills training for • Employer indicates what tasks the employee does and thus needs to be assessed in • Results contribute to “skills map” of the organization shown to employer • Research purpose (with post-training assessment): to measure the effectiveness of training in increasing digital skills levels • by comparing pre- to post-training levels

8. Assessments - Considerations • Pre & post training assessments should be very similar • Reason: to ensure measured gains in skills between pre- and post-training are due to the training and not the assessment instrument (easier or harder) • Risks: • potential for participant to remember the pre-test • can’t be totally certain skills gains are due to only the training (may have been other causes) • Common to all training evaluations where there is no control group (individuals who were not trained) • Should not “teach to the test” -assessment should not have exactly same content as the training • otherwise would be testing memory, rather than ability

9. Creating Assessments • Creating an assessment for each of the 8 digital tasks, with some tasks having up to 3 sub-tasks • This means a lot of assessments: for each occupational group each with 4 or 5 component detailed occupations, there are up to 80 cells for which assessments must be produced • However, once a “generic” assessment question set is produced for a digital task, it can fairly easily be contextualized across detailed occupations • By adapting positions and documents applicable to the particular occupation

10. Contextualizing to Occupations

11. Digital Skills Score • To measure skills gains, need to measure gradations of skill not just whether or not one has a skill • Skills gains = difference between pre and post-training skills scores • Skill score depends on 3 dimensions of complexity: closed/open and well/un-defined, and complexity of sub-tasks required for each task

12. Complexity – 2 Dimensions • Closed/open: • Closed problem has limited solutions: “Identify the number of employees that left the company in April” • Open problem has unlimited solutions: “Improve the system for waste disposal that meets municipal green policies” • Well/not (or un)-defined: • Well-defined problem is directive and clearly tells employee what they are supposed to do: “Find and download government form x”. • Undefined problem is less clear about what to do but more clear about the situation and what the employer needs: “Schedule the delivery of products to Calgary within the next 48 hours.”

13. Complexity of sub-tasks • Complexity of sub-tasks in different domains required to successfully carry out the task: • Prescriptiveness of instructions • Clarity of expected outcomes • Sophistication of technological applications • Complexity of digital info and needed info processing skills • Preciseness of task setting • Still working on details to formulate a skill score, but an example follows

14. Complexity Scale – Illustration CLOSED OPEN NOT DEFINED DEFINED 1 2 3 2 4 3 4 3 5

15. Example of an Assessment: Task 8 (Q1 & Q2), Coordinating activities with colleagues by email Question 1: You received an email from the bookkeeper asking about the arrival of some merchandise. Forward the email to your colleague Michel Laurent to ask him if he received the merchandise. Insert a subject and copy (CC) the bookkeeper on the email. The message should say: Hello Please let me know if you remember receiving the merchandise. (your name) Question 2: You received an email from the production logistics coordinator asking you to forward an attached notice about overtime policy changeto two colleagues, Eric Bernard and Gabriel Robert, and to notify them about the upcoming meeting (to discuss the new policy). You need to CC the coordinator on the email, and BCC the director general, Charles Dubois. Insert a subject and write this message: Hello Please read the attached policy on overtime hours. Note that there is a meeting about the policy on Thursday the 14th at 12:30PM. (your name)

16. Example of an Assessment: Task 8 (Q3 & Q4), Coordinating activities with colleagues by email Question 3: You received an email from the production logistics coordinator asking you to send your colleagues, Simon Fremont and Luc Thompson, the attached notice about the water service interruption. Forward the notice as an attachment to your colleagues. Then CC the sender, and BCC the director general, Charles Dubois. Indicate that the email is urgent. Insert a subject and write this email: Hello Please read the attached notice about the water service interruption. Make arrangements to avoid a delay in production. (your name) Archive both the received and forwarded emails in the Management folder of your email application. Question 4: You received an email from the production logistics coordinator asking you to send an attached pamphlet about a new piece of production equipment. You must forward the attachment to your colleagues, Jean-Claude Petit and Yves Lefebvre, and notify them there is an information session about the machine on Wednesday at noon. Make sure you CC the email sender and BCC the director general Charles Dubois. Also, indicate the message is urgent, and archive both emails in the Management folder.

17. Example of How an Assessment Might be Scored: Task 8, Q1 and Q2 Question 1 • Complexity Score: 3 (Closed/Open: 1, Defined/Not defined: 3) • Task description (domains): Simple and precise instructions (1), defined outcome (1), defined technological application (1) and information (1), and defined task setting (1), routine task with definite steps. Question 2 • Complexity Score: 4 (Closed/Open: 2; Defined/Not defined axis: 3) • Task description (domains): Simple task with more complex instructions (2), defined outcome (1), moderately complex technological applications (2), defined pieces of digital information (2), and defined task setting (1), routine task with definite steps.

18. Example of How an Assessment Might be Scored: Task 8, Q3 and Q4 Question 3 • Complexity Score: 4 (Closed/Open: 2, Defined/Not defined: 3) • Task description (domains): Simple task with more complex instructions (2), less precise outcome (2), moderately complex technological applications (2), a few defined and more complex pieces of digital information (2), and defined task setting (1), less routine task with less definite steps Question 4 • Complexity Score: 4 (Closed/Open: 2, Defined/Not defined axis: 3) • Task description (domains): Simple task with more complex instructions (2), less precise outcome (2), moderately complex technological applications (2), a few defined and more complex pieces of digital information (2), and less defined task setting (2), less routine task with less definite steps.

19. SRDC contacts • Norm Leckie: nleckie@srdc.org • Social Research and Demonstration Corporation: www.srdc.org