Big Data EUDAT 2012 – Training Day

1. Big DataEUDAT 2012 – Training Day Adam Carter, EPCC EUDAT Training Task Leader

2. Data

3. Collect Data Share Organise Process Interpret Store Knowledge Paper

4. Collect Data Share Organise Interpret PIDs Process SimpleStore Data Staging Store Knowledge iRODS Paper Replication

5. Collect Workflows Dataflows Data Share Organise Interpret PIDs Process SimpleStore Data Staging Store Knowledge iRODS Paper Replication

6. Data Bytes Data Data Data km Data I/O ≫ Compute

7. Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data

8. Big Data - Problems

9. 1. Many Bytes Data takes a lot of space to store Data takes a long time to move Data takes a long time to process Data takes a long time to organise Data too large to be manually interpreted

10. 2. Distributed Data • You have to bring (some of) the data together to process it • You have to bring (some of) the data together to interpret it • Often implies: • Different ownership • Different formats • Different quality

11. 3. Data Intensive • Actually a feature of the application and the data size (and, arguably, the hardware) • Difficult to compute efficiently • Often implies big (in bytes) • Different definitions: • A lot of data on which to compute • A lot more I/O required than compute

12. Everyone can now do big science… Big Data - Rewards

13. Datascopes Theory Experiment Simulation Datascope “Observe” a large amount of data and look for patterns Data Intensive Research

14. Interdisciplinary Research Using others’ data brings more to your problem than you could have collected yourself May already have been pre-processed, possibly with input from experts in another field

15. More data Better statistics More likely to be data collected that can be used to test a theory

16. Of course, bigger doesn’t mean better As always, you should use the right tools for the job

17. Big Data – (Partial) Solutions

18. Data Infrastructure Share your data more easily Use others’ data more easily Store more data Store data more reliably Move data more quickly Process data more efficiently Interpret data more easily

19. Move the Compute to the Data • Virtualisation • Queries to relational databases • Distributed query processing • If you’ve got a large amount of “baseline” data that is accessed repeatedly • put compute power close to the data • consider mechanisms to allow others to perform arbitrary compute against your data at your site

20. Data Intensive Computing • Build and use computers built for data-intensive research • Slower, low-power processors • High performance I/O systems • “Amdahl-Balanced” • In some cases these machines might (locally) move the compute to the data

21. Organise Your Data • Metadata • PIDs • Catalogues, Indexing • Make your data addressable • Allows for caching, or using nearby replicas • Allows for retrieving only necessary data • Consider why you’re storing your data • Think big!

22. Conclusions • We can do a lot more with our data than we currently do • A problem shared is a problem halved • Use infrastructure and services provided by others • Consider your data as important as the paper • Infrastructure likely to help with this • Also need change of mindset re attribution, etc. • Big data offers opportunities • Think about what you could do with data or techniques from other disciplines • Think big!