GCC Tour

1. GCC Tour • For FNAL Saturday Morning Physics

2. GCC Entrance • Docent: • sign log book: one person per tour (the docent) signs in. • “We have a brief itinerary inside the GCC facility. We're going to visit a few rooms where some of Fermilab's biggest questions are tackled.” • Facts: • The facility of GCC is 10 years old, has 10,000 square feet of area with almost 7000 processing nodes. • this is a lights-out computing center. No one has a permanent office here and it runs entirely remotely, except for servicing. • Note safety issues: • watch your step and tell the person following you. • let people know the rooms have noise, but it's not unsafe (except for Comp Room C for >15min). There are earplugs for people to take. • Q: What kinds of physics questions require big computers? • Hints: • Should the world change how it makes and uses energy? • How many stars are there in a galaxy, in the universe, and how do we predict the evolution of the universe? • billions of stars per galaxy; billions of galaxies in the universe. • How many particles are in each bunch that collides in a particle accelerator, and how many bunches per second? • two trillion particles collide per second, where we try to understand what goes and what comes out. • What are some hard math problems that you’ve encountered? • A: Simulations (of particles and the universe) • A: Data Processing (event reconstruction) • A: Many-variable math: linear algebra, integration, etc. • Q: What is the basis for the modern computer? Hint: Benedict Cumberbatch • A: Alan Turing’s Universal Machine

3. Tape Room • Docent: • turn on light in tape room: can enter tape room without booties to turn on light; that’s as far as you can go. • Main monitor: • 3 robots to the left in front of room are shown in the top monitor in right large box • Tape Data: • 40K tapes hold 50K TB of data; each slot in a frame holds 5TB (10K slots); Transfer Rate is 240 mbps • Holds raw data and some data that’s been processed/reduced; other reduced data is at FCC: this avoids single-point failures and avoids unrecoverable loss of data. • Fire suppressant is water and then some powder • Equipment Organization: • 3 front frames are for CMS only; 1 frame in back is for general experiments • robots load data onto server to be accessed for analysis • Tapes need servicing about 3x per week • Cabling: yellow are fiber; purple are cat6 • Trivia: • SSDs would not be a good replacement, because they degrade much faster than tapes

4. Network Room A • Networking for Computer Room A • Speed: 10’s GB/s • right: connects to Tape Room A • left: connects to world. • Cabling • Ethernet (e.g., cat) and fibers are both used • Compare cables in and out • Wires into room from Tape Room • Wires out of room • cables get larger from room to room (except for the last Computer Room) • Note red bags are fire wall to prevent movement of a fire

5. Hallway to Computer Room A • Note raised floor: • Q: Why is there a raised floor • answer question in Computer Room A • Note Safety Issues again: • Warn about step at exit of Computer Room A • Remind people that the rooms will get warm and loud. Offer ear protection. • Q: Why will it get warm? • Q: What were the early computers? Who built the first one?

6. Computer Room A • 3K Dual- and Quad-core machines • run linux with 4GB of memory per machine. • not parallel between computers, only on each computer alone. • A: It get’s warm, because of all the computers • Computing Purpose: • CDF and DZero are the primary users: used it for search of Higgs and frontier physics • Also, general purpose computing: Computing Science Grid • Q: What’s Grid Computing? • A: parallel processing of large problems, that need to have many potential solutions checked or multiple calculations that have to be done at the same time, as in n-body simulations. • Infrastructure: • A: Raised floor is used for cooling • Power and network cabling is done above the machines. • Historical/Technological Notes: • Mark-1 (1944) - Used in manhattan project • ENIAC (1946) - Electronic Numerical Integrator and Computer, with 80 bytes at 5000 flops (floating point operations per second) • UNIVAC 1 (1951) - Census bureau ~2000 flops • Cray I (1976) - First vector processing 166 megaflops (floating point operations per second) • Fastest computer today: 33.86 petaflops (10^15 flops)

7. Electrical Room • Can supply 2MW of power to Computer Rooms A and B (power for Computer Room C is located within that room). • 3 UPS’s that switch to backup generators in case of emergency. If the power shuts off, they can be kept on long enough for a clean shutdown. • Fun fact: • ConEd services all feeds coming into site twice a year to fix weather-loosened connections to outside world. several trucks with energy sources have to be brought in to keep the building on during this site-wide outage.

8. Computer Room B • 2500+ Dual- and Quad-core computers: • 32 processors per computer • 64GB ram per computer. • Computing purpose: • Grid Computing • runs the FermiCloud is run in this room. • not parallel between computers, only on each computer alone. • Computer Room A and be is about the equivalent of 10K laptops??? • Millenium simluation • 10 billion particles, 1.4M cpu hrs over ~2000 cores (~1month)

9. Examples of computations • Millennium simulation video • 60 Million cpu hours ???

10. Network Room B • Networking for Computer Room B • Speed: >200 GB/s • phone line gets about 40MBps • Docent: • Cabling: • Go to right immediately: • show that there are even more wires coming into this room, than there were entering into Network Room A • Go to left: • show how many go out to Comp C (far far fewer) • Q: why would this be? • A: switches for Comp Room C are in with the computers in Comp Room C instead of outside. This is necessary to allow fully parallel computation across all the racks in Comp C. Otherwise it would take too long for information to go out of room and then back in to the other computers. Comp Room C is like Computers + Switches together, while the other Comp Rooms have their switches in the network rooms. • Safety • Q: Why is there a humidifier in this room? • A: Necessary to reduce sparking and destruction of circuits.

11. Computer Room C • 1000 normal multi-core computers + 3 racks of GPU computers • Inter-node connections are infiniband: allows complete parallel processing among all nodes. • a few racks near entrance are GPUs only. • the rack with lots of cables coming out like a cable-monster is the infiniband switch which allows info to pass very quickly (40GB/s) between machines. • Computing Purpose: • Runs Lattice QCD projects. • Q: What’s the purpose and use of Lattice QCD? • A: Describe Lattice QCD • Compute forces on a grid of points to approximate full solution of interacting quarks. (similar to n-body simulations in cosmology).

12. Exit • Q: How much energy used? • A: 2.25 megawatts at maximum (about 40% used as of 2014) • equivalent to ~35,000 60W lightbulbs or ~3,000 horsepower • Q: What is the energy bill? • A:Batavia residential utilities are about $0.10 kWh. If GCC operating at 2MW all year, this would total ~$1.75 million per year at market rates. Fermilab probably gets a discount... • Q: Why might data infrastructure companies (like Microsoft or Google) wish to re-locate their data storage and computing centers to the Nordic countries, like Iceland? • Hint: It’s not because of the low temperatures. • A: It’s because of the low-cost, low-carbon-footprint energy generation technologies, like hydropower.