190 likes | 338 Views
Usenet Training. May 4-7, 2004 Orlando, FL At Disney’s Coronado Spring Resort. Day Two. Building an Enterprise Usenet Environment. Questions to ask. How many concurrent NNTP reading sessions? I don't know What is the size of the user base? 5-10% or more may access news
E N D
Usenet Training May 4-7, 2004 Orlando, FL At Disney’s Coronado Spring Resort
Day Two Building an Enterprise Usenet Environment
Questions to ask • How many concurrent NNTP reading sessions? • I don't know • What is the size of the user base? • 5-10% or more may access news • How much retention? • What content? • Full feed, internal discussion, partial feed • Binaries/Text • 97-99% size from binaries • Full feed is 1.1 TB/day - doubles every 9 months • Redundancy desired • Full data replication • Disaster recovery • Failover • Feeding architecture to provide/receive • Authentication and classes of service • Performance • Honda, Audi, Porsche
Sample Architecture Criteria • 8,000 Concurrent Connections • 14 days binary, 180 days text • Two data centers • Full redundancy, outage of one DC can support 4000 connections. • Sell 10 full feeds, peer with 50 providers • Authentication by IP address
Calculations • Note about calculations • Change over time • Not hard and fast • Change with experience and new architectures
Storage • 14 days Binary and 180 days text • 17 TB • x2 with full redundancy • Adaptive spooling
Machine Class Choices • Problems we're solving • Totally random I/O • Significant network volume • We're not NASA • No Complex Number Theory modelling here • WOPR is misplaced • Disk mechanics • Slowest evolving technology • Physical limitations • Cost • Administration • Deployment
Machine Classes • 2000 connections per host - 4 hosts • 1000 connections per host - 8 hosts • 400 connections per host - 20 hosts • Machine Class dictates OS or vice versa • Summary • We're looking for an efficient I/O mover • Many paths, independent busses • Carl Lewis, not Andre the Giant.
Architecture Diagramming • Start with Tornado Front Ends • At least 100 GB storage • At least 1 "fast" spindle per 400 connections • At least 2 "slow" spindles per 300 connections • Sun 280R's, 2 CPU, 4 GB RAM, Sun 3510 attached array (shared) • Module to support 2000 connections
Tornado Back Ends • 4000 connections accessing articles • 1 BE with one array can be enough • BE's can be smaller than an FE • Direct spool sharing only • Not quite a Commodore 64… but close
Tornado Back Ends • Storage Array • Very rough estimate 1 drive per 75 connections. • Optimize for Random I/O • Benchmark Precautions • Usually sequential I/O, push for random access data. • Plan on performance 10-20% of spec benchmark or even less. • 48x72GB 10,000 RPM drives (3456 GB) • FCAL Switching • Writes – approx 120 mbps ~ 13 MB/sec • Reads • 4000 connections • Average? 1Mbps • Many “constant downloaders” • .11 MB/sec X 4000 = 450 MB/sec • 450 MB/sec x ~6 (truly random adjustment) • 2.6 GB/sec Switching Spec • Start with one 2GB/sec switch with plans to add another. • We only have 3 TB and we need 17?!
Enter Storm Cellar • 17TB total, 3 TB deployed - need 14 TB • 3 Storm cellars with 6 TB usable • Split cascade feed from Tornado Back Ends • 4U NAS Appliance 24x300GB drives (7.03 TB / ATA Drives) • Linux 2.6.2+
Feeding • 10 Full feeds @ 120 mbps ~ 3-5 feeds per box • 2 Peering Cyclones per site • 1 Master Cyclone • 1 Hurricane
What could we do with Linux? • Features we can use • FE adaptive caching • Cyclone Split Feeding • Newest Linux kernels • FCAL sharing probably not feasible • IP spool sharing with adaptive caching • Hardware choices • 1 Day’s worth of disk on each FE for cache • BE's have 4+ TB each
Local spool cache on Tornado FE’s serves recent articles that are actually read (8 copies – 16 for 2 Data Centers) Spools are aggregated by all FE’s Back Ends contain only partial data – spreading I/O over many BE’s. Linux Architecture
Quiz • Trace the path of locally posted articles. • Trace the path of articles received from peers • Where and how are articles numbered? • If a Front End is down, what happens? • If a Back End is down what happens? • If the storage array is out, what happens? • What do we do if the Master Cyclone goes down? • What do we do if the Hurricane goes down? • How do we add a feed from another data center to only fill in articles we’re missing? • How do we add a feed to only turn on if our Master Cyclone goes down? • Who should we peer with?
Questions Questions