100 likes | 110 Views
Big Data and Macroeconomic Accounts. Michael Davies, Division Head, Macroeconomic Statistics Division, Australian Bureau of Statistics September 2014. Background. There is a lot of hype around big data.
E N D
Big Data and Macroeconomic Accounts Michael Davies, Division Head, Macroeconomic Statistics Division, Australian Bureau of Statistics September 2014
Background • There is a lot of hype around big data. • Many are very excited - it seems to solve many of the problems of getting good data from households and businesses, saving a lot of trouble and expense.
First Reaction: Methodology • Initial reaction from methodologists who have spent their lives ensuring high quality survey and census data is to proceed carefully. • Elegant solutions are proposed to bring the data up to the quality of survey and census data.
ABS History • The ABS has a lot of experience with administrative data - Customs trade data; tax data for businesses and people; and financial data from the banking regulator. • The ABS is doing a lot of work on the combination of Government datasets e.g. health, welfare, employment. • Big demand for social micro data
Now • The new territory is transactional data from private businesses e.g. supermarket scanner data. • This is at a time when companies are being told that their data are their most valuable asset.
ABS Recent Experience • In the case of one big respondent - the boss saw benefits and said "make it happen" then lawyers got involved - took forever and we had to accept significant restrictions on what we can do with the data. • Can we make them hand over detailed transaction data under existing laws? Open to debate and interpretation of laws intended to cover paper forms collecting a limited amount of aggregate data.
The Way Ahead. • Lots of things to address in parallel - legal right to data/ contracts if buying - continuity of supply - interpretation of data - don't get hung up on methodology but give priority to measurement aspects. - consistency over time • Haste to save money does not help. • Take care stopping traditional data sources
The Way Ahead Cont. • Need thoughtful and considered approach balanced with meeting expectations and cost reductions.
Risks • More noise than signal • Disrupted supply (depending on commercial arrangements) • Bias (data manipulated by supplier) • High cost to provider
For you may palm upon us new for old:All, as they say, that glitters, is not gold. John Dryden, 1687, The Hind and the Panther.