7. IS Audit Process Applied to Data Life Cycle Audit, e-Learning & Higher Education

1. 7. IS Audit Process Applied toData Life Cycle Audit, e-Learning & Higher Education Judy Yard & Jeff Brandon Extra Credit Project 7. IS Audit Process

2. Conducting your own Data Life Cycle Audit Using an Information Technology Advisory Group (ITAG) 7. IS Audit Process

3. ITAG Overview • Proven methodologies: • Business Technology Alignment • Strategic Planning • Business Intelligence • Technology Selection & Implementation • Technology Initiative Management • Business Continuation / Disaster Recovery Planning 7. IS Audit Process

4. What is the Data Life Cycle? • The Data Life Cycle are the phases in which data moves through the organization. The different phases include how the organization collects, stores, processes and disseminates their key data. • Key data defines the most critical or important data elements that are pertinent to supporting an internal organization’s specific business processes. 7. IS Audit Process

5. Why should you conduct a Data Life Cycle IS Audit? • Eliminate duplication of work effort and increase overall productivity. • Consolidate stranded information assets. • Generate business intelligence in order to better facilitate customer relationship management. • Support cross-departmental business functions. • Enhance one-on-one (personalized) marketing. 7. IS Audit Process

6. Why should you conduct a Data Life Cycle Audit? • Improve data maintenance and integrity. • Helps identify data flow dependencies for business continuation/disaster recovery planning. • Identify a complete enterprise application integration strategy. • Creates a vehicle to communicate technology needs to senior management 7. IS Audit Process

7. What does the AMS have to do with it? • AMS is core business (association) application which supports all four phases of the Data Life Cycle • Streamlines data collection • Primary data repository • Supports transaction processing • Base for reporting/business intelligence • Identifies data integration • Third party applications / data sources • Strategic Partners • Web 7. IS Audit Process

8. Enterprise Application Integration • AMR Research (2003) • 45% of companies require the same customer to place orders through separate systems • 60% have limited or no integration between their front- and back-office systems • 90% struggle with customer responsiveness, visibility, automation and data quality • 20% of customer data is centrally managed 7. IS Audit Process

9. How to Conduct a Data Life Cycle Audit • Identify primary business functions and associated processes and tasks for each • Identify departments that are associated with each business function • Document the Data Life Cycle for each business function (and process) • Data Collection • Data Repository • Business Process • Data Dissemination 7. IS Audit Process

10. Primary Business Functions • Membership • Committees / Boards • Meetings Management • Products / Publication Sales • Sponsorship • Education • Certification • Marketing • Legislative Tracking • Regulatory Tracking • Finance • Customer Service • Communications • Research • Foundation 7. IS Audit Process

11. Business Function Analysis • What are the primary responsibilities (business functions) for each department? • What other departments or business functions are directly related with this business function? • What are the routine tasks to perform the primary responsibilities? • What are the standard operating procedures? • How current is your business continuity / disaster recovery plan? 7. IS Audit Process

12. Business Functions • The same business function can be performed by multiple departments. • Processes and tasks related to a business function can cross over multiple departments. • A business function can have multiple processes and tasks associated with it. 7. IS Audit Process

13. Data Collection Methods • Web Forms • Email • Conversations • Third-Party Strategic Partners • Abstracts (Call for Speakers) • Membership Application • Subscriptions • Order Entry • Meetings Registration • Exhibit Registration • Surveys • External Societies or Chapters 7. IS Audit Process

14. Data Collection Analysis • What key data is required to support each business function? • What key data is collected during each data collection method identified for each business function? • What is the frequency of collected key data based upon each data collection method? • What key data is considered static, non-static or calculated? 7. IS Audit Process

15. Key Data Mapping 7. IS Audit Process

16. Data Storage Analysis • Where does the key data reside that supports each business function ( Word, Excel, etc.)? • What format is the data? • How does that data get accessed (standalone, networked, remote application)? • Based upon the interdependencies of key data between business functions, does each business function have access to the most current and accurate data source? • What are the key indexes for each data repository? • How much data is being stored? 7. IS Audit Process

17. Data Repository Map Online Meeting Registration Online Product Sales PAC Certification CRM/AMS Chapters Stranded Information Assets Press/ Media Legislative/ Regulatory FMS 7. IS Audit Process

18. Challenges with Stranded Information Assets • Technical Challenges • Multiple data sources, data types, locations • Data accessibility • Redundant data sources • Organizational Challenges • Managing information • Maintaining data integrity • Creating accurate and timely reports • Publishing to Members, Committees or Board 7. IS Audit Process

19. Process Analysis • What process is being supported based upon data collected by a different business function? • Are there manual processes which can be automated easily? • What processes are supported by web-based applications? • What external strategic partnerships do you have that support internal business functions – how does those partnerships impact the process, including data collection, storage and dissemination? 7. IS Audit Process

20. Business Process Map Meetings/ Education Products Exhibits Committee Management Customers Legislative / Regulatory Financial Reporting Certification 7. IS Audit Process

21. Data Dissemination Methods • Digital Content for Sale • List/Reports/Queries • Inter/Intra/Extra Net • Business Intelligence • Forecasting • New Product/Service • Enhance Customer Service • Knowledge Creation • Web Forms • PDA • Personalized Emails • Publications • File Transfer • Conversations 7. IS Audit Process

22. Data Dissemination Analysis • What are the different tools used for disseminating data (report writers, data mining tools, etc.)? • What format does key data get disseminated? • Reports • Publications • Web Content • PDA • External Strategic Partnerships • What other applications is data being disseminated to? 7. IS Audit Process

23. Data CollectionRepositoriesProcessesDissemination Data Life Cycle Map Repositories Processes Data Collection Dissemination Mail Store Mail Store Mail Store PDA PDA PDA Action Trigger Content Update • Trip Reports • CEU Verification • Issue Lobbyist AMS/ Subset Dynamic Update Web Forms Analytics • Meeting Registration • Product Purchase • Member Profile Temp DB/ Other Batch Update Integrity Reporting Data Entry • Survey • Membership Application 7. IS Audit Process Temp DB/ Other Batch Update Analytics Data Entry

24. Data Life Cycle Data Collection Repository Process Dissemination Continuous 7. IS Audit Process

25. Introduction to e-Learing Audit • Complexity of student experience • Problems facing evaluators • Different research methodologies • Audits promote evaluation • Holistic view – why? how? when? and what? of evaluation 7. IS Audit Process

26. Pressures • Develop transferable student abilities • Pressure within the worldwide web • Governmental and employer pressure • Requirement – learners to participate in a networked information society • Emphasis on student-centred learning 7. IS Audit Process

27. Evaluators • Experienced evaluators needed in production, usability and effectiveness of online delivery • The Internet and ‘blended learning’ with frontal learning elements now widespread 7. IS Audit Process

28. Promoted by government initiatives Cost effectiveness of e-Learning Permits tailored education Can develop in ways which impoverish education Time & production costs of development Students & tutors lacking skills to use Two Sides of e-Learning 7. IS Audit Process

29. The Impact of Quality Assurance Assessment (QAA) • Parallel to increased use of ICT (Interactive Computer Technology) is rising interest in quality assurance strategies • Lighter QAA touch & emphasis on stakeholders – students, employers, funders, society • Emphasis on outcomes rather than processes • QAA looking for an institution’s strategy 7. IS Audit Process

30. Why Evaluate the Learner’s Experience? • Subject review & institutional audits • Act as: educational ‘connoisseurs’ with approaches and tools to use in order to act as experts informing policy • Require: awareness and understanding of what is experienced • Require: personal judgement • Require: ability to articulate subtleties to a wider audience 7. IS Audit Process

31. Is the student experience exceptional? Are there defects in the system or delivery? Fit for purpose? Value for money or adds costs? Transformative/Value adding? Students’ perceptions? Effect on student numbers? Questions to Ask yourself 7. IS Audit Process

32. Metrics for e-Learning Evaluation • Efficiency of the teaching and learning process • Readability & usability of materials • Communication • Social relationships • Costs 7. IS Audit Process

33. Quality of the e-Learning Experience • Efficiency of the teaching and learning process • Accurate student data • Drop out rates • Student assessment results • Student satisfaction ratings • Training of tutors 7. IS Audit Process

34. Quality of the e-Learning Experience • Efficiency of the teaching and learning process To best understand the potential for e-Learning: • Start with 1-2 courses • Is this course needed by students? • Is the assessment electronic? • Collection of computer log data 7. IS Audit Process

35. Materials and Communication • 3. Communication • Communication with tutors & other students • Skills in using technology • Access from home/campus/3rd party • Group work 2. Readability & usability of materials • Peer review of materials • Reuse as learning objects • Online exercises • Study skills • e-Library and e-journal access 7. IS Audit Process

36. Social Relationships and Costs • 5. Costs • Costs per student of recruitment • Training of tutors • Costs of electronic access • Reuse of learning materials & learning objects 4.Social Relationships • Hours working online • Student motivation • Student support & interaction with other students 7. IS Audit Process

37. Approaches to Human-Computer Interaction • Learning theory is dominant – how students and their interaction with computers fit together • How students learn when using computers • Using the technology to teach in ways that are novel and unique to the environment 7. IS Audit Process

38. How are we going to Evaluate the e-Learning Experience? • Evaluation goals need to be clearly defined • Is this a formative or summative evaluation? • How should the data be gathered? • Should quantitative or qualitative methods be used? 7. IS Audit Process

39. How are we going to Evaluate the e-Learning Experience? Methods include: • Surveys • Observations • Semi-structured interviews • Focus groups • Cognitive walk-throughs • Qualitative studies can provide examples of usability difficulties 7. IS Audit Process

40. How are we going to Evaluate the e-Learning Experience? Surveying students’ views: • detects feelings of overload • informs necessary course adjustments Survey methods: • paper based with • telephone supplementary interviews • increased use of electronic surveys • focus groups 7. IS Audit Process

41. When to Evaluate the e-Learning Experience? • Dependent on whether the evaluation is formative or summative? • When is the student deemed to have completed the course? • Will formative evaluation during the course engage students more? • Harvesting evaluation information from student interaction with the course 7. IS Audit Process

42. What does the Evaluation provide? • Audio/video record of the computer screen, the user’s comments and keystrokes Baseline data to: • make adjustments, possibly ‘instant’ feedback • for external audit • shape e-learning strategy • input into associated resources – library and information literacy provision 7. IS Audit Process

43. Data Protection • Students and tutors need to be kept informed about how the data they provide is being used • Students and tutors retain the right to remove what they have provided at any time • Evaluators have to ensure secure access to the data • Evaluators must get consent especially if using the data in a different context 7. IS Audit Process

44. Conclusion • The University is interested in the quality of the student experience • To improve that experience needs understanding of student and staff needs • Here we have aimed to provide a framework in which to conduct evaluations • Current work on modelling courses will provide data on points of concern that arise in blended and e-learning 7. IS Audit Process

45. Preparing for an A-133 Audit Office for Sponsored Research Risk Management and Audit Services 7. IS Audit Process

46. Background and Requirements of A-133 “Reporting Package” submitted annually includes: - Financial statements - Schedule of expenditures of federal awards - Auditor’s reports on internal controls - Auditor’s reports on compliance - Schedule of findings and questioned costs - Summary schedule of prior audit findings - Corrective action plan - Data collection form 7. IS Audit Process

47. A-133 Audit Process A-133 audit process involves the following: • External groups: PricewaterhouseCoopers (PWC) and federal government • Internal groups: schools, OSR, RMAS, Office of General Counsel, VP Finance 7. IS Audit Process

48. Month July Aug. Sept. Schools A-133 PWC orientation meeting Provide support documentation for direct cost testing, department questionnaires to OSR PWC & RMAS perform direct cost & compliance testing, includes visits to grant administrators A-133 Schedule 7. IS Audit Process

49. Month Oct. Nov. Dec. Jan. Feb. Schools Provide responses for additional information from PWC, RMAS to OSR Provide responses to resolve initial findings Present final draft report to Sponsored Program Operating Committee (SPOC) PWC completes final report; Schools complete corrective action plan “Reporting Package” submitted A-133 Schedule, cont 7. IS Audit Process

50. A-133 Audit -Two Major Areas Internal controls Compliance testing 7. IS Audit Process