Transforming the Australian Social Science Data Archive

1. Transforming the Australian Social Science Data Archive Deborah Mitchell The Australian National University

2. Presentation Overview • Introduction to ASSDA • Changing environment • cultural attitudes to data sharing • growth and diversity in data creation • technological change • Response to change: the new ASSDA • The distributed data Archive as a grid environment

3. ASSDA in Brief ASSDA was set up in 1981 at ANU in Canberra Archive is housed in the Research School of Social Sciences The Archive holds some 1500 data sets, core holdings being survey based data Most notable holdings are national election studies; public opinion polls; social attitudes surveys The Archive also acts as a custodian for data from SE Asia, New Zealand and smaller Pacific nations

4. Operating environment For most of its history, ASSDA operated in a highly centralised fashion serving a small, fairly tight knit community of users - who were also willing depositors Up until 1995 the Archive also acted as the distribution point to the academic community for official data collections of the Australian Bureau Statistics (ABS) Complementarity between the holdings and functions of ASSDA: the general ABS data underpinning the specialised research surveys carried out by the academic community

5. What changed? • 1. Cultural shift in attitudes towards data • ABS withdrew open, free-of-charge access to its data and ‘commercialised’ its data distribution • This sent a powerful signal that data/information is valuable; academic community responded to this signal by hoarding its research collections • In the period 1995-2000, survey data deposition declined markedly.

6. What changed…ctd • 2. Growth and diversity in data collection • As the ABS gradually tightened access to official data and ‘priced’ itself out of the academic market, the academic community significantly stepped up its own primary data collection efforts • Qualitative methods were being embraced and creating large numbers of small area studies and new forms of data - linguistic; web pages; visual images were being shaped into digitised databases • The Archive was being called on to manage increasing amounts of data, in unfamiliar formats

7. What changed…ctd • 3. Technological change • Memory became cheap! • Proliferation of software & other analytical tools to deal with non-quantitative data • - Advent of grid-computing and e-Research • User expectations of what the Archive could/should be doing grew rapidly.

8. The ASSDA Response • 1.Cultural change • Re-establish a culture of data sharing within the academic community; open up the governance of the Archive to the research community. • 2. Growth & diversification in data • Look to a means of growing the Archive’s capacity and skill base in terms of handling both greater volume and type of data. • 3. Technological change • Harness new technologies, software tools in a staged a manageable fashion, with particular attention to user demands.

9. ASSDA Distributed Archive • Moving to a distributed archive solution is not an ‘obvious’ solution in the Australian context, as economies of scale still favour the original centralised model that we started with in the 1980s. • High-speed broad-band networks servicing the academic community across the Australian continent also favour a centralised solution. • However, we have decided to move to a distributed archive form in order to solve several problems: • Diversity in forms of data; • Growth in volume of data and knowledge about where these data are located; and • To deal with data hoarding.

10. The ASSDA Grid Solution NESSTAR Web V i ew ASSDA-UQ ASSDA-UNSW ASSDA-ANU

11. The ASSDA Plan • In summary: • The Archive will become a series of ‘nodes’ each of which will specialise in a particular area, determined by the research strength of each host institution. • In the first round of development, nodes will be established at the U. Queensland [qualitative data] and UNSW [administrative data]. • As shown in the previous slide, we will be leaning quite heavily on the capacities of the Nesstar software to deliver one interface to users.

12. The ASSDA Plan … ctd • The ANU-UQ-UNSW network will be set-up in 2005 to function in a grid fashion,with middleware designed at ANU to load-share catalogue enquiries and Nesstar on-line analyses between the three servers. • We expect that at each node, specialised analysis tools will be developed for data analysis, beyond Nesstar capacity. • Over time, we aim to establish at least one node in each state that will broaden and deepen the ASSDA holdings in a significant fashion.