230 likes | 392 Views
Recent Developments with UniBoard CASPER 2011 Workshop 10-14 October 2011 NCRA, Pune, India. Jonathan Hargreaves, JIVE. Introduction. The UniBoard Project The UniBoard Hardware Toolflow Control System Applications Future Plans. Radio Net FP7.
E N D
Recent Developments with UniBoard CASPER 2011 Workshop 10-14 October 2011 NCRA, Pune, India Jonathan Hargreaves, JIVE
Introduction • The UniBoard Project • The UniBoard Hardware • Toolflow • Control System • Applications • Future Plans CASPER Workshop, 10-14th October 2011, Pune, India
RadioNet FP7 • Several joint research activities of which one is UniBoard • 27 partners: all of the major radio astronomy facilities and the laboratories involved in technology development • Shanghai Observatory joined October 2009 • 10 M€ over 3 years • Started January 2009 Contract no. 227290 CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard Project • UniBoard is a Joint Research Activity led by JIVE • UniBoard is a generic digital board for Radio Astronomy applications • Hardware developed by ASTRON • Test firmware development done together by ASTRON/ JIVE The concept • Place as much processing power and I/O as reasonably possible on one board • Use10GbE interfaces for data I/O • Front to back symmetry – data can flow both ways • All FPGAs identical • As generic as possible - the same hardware can be used for many applications CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard Block Diagram • 8 Altera Stratix IV FPGAs, each with • 14Mbits on-chip SRAM • 1288 18x18 bit multipliers • 2 DDR3 interfaces, up to 64GB per board • 3 external 10GbE ports • 32 LVDS inputs to Back node only • A high speed transceiver mesh provides a • 24Gbps bi-directional connection between • Each Front node/Back node pair • A single chip switch provides a 1Gbps • Ethernet connection to each FPGA • Power supply is -48V, maximum load 400W CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard Hardware Dimension = 9HE x 340 x 2.4mm 14 layer PCB Can be rack mounted with a backplane … or placed in an enclosure CASPER Workshop, 10-14th October 2011, Pune, India
Uniboard Production Status Complete • First prototype delivered May 2010 • 8 boards produced this year and delivered to partners in May 2011 In Progress • 9 more boards in production, expected December 2011 CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard Tool Flow and IP • Use Altera’s Quartus for synthesis and fitting • Major infrastructure blocks (DDR3, 10GbE) licensed from Altera • Use an SVN repository to share our own IP and designs between partners • Many IP blocks from I2C interfaces to filterbanks adapted from the Lofar project • Most design input is VHDL/Verilog, exception is SOPC builder for the NIOS processor • Could use Simulink but we don’t currently • Use scripts to automate the toolflow • Use wrappers to make higher level designs independent of the target device • Simulation in ModelSim CASPER Workshop, 10-14th October 2011, Pune, India
Control System Nios2 CPU (Server) • There is no Master control FPGA • Instantiate a NIOS soft microcontroller in each FPGA • Setup and Control registers in VHDL blocks appear in the NIOS memory map • Simple designs can be controlled via the JTAG UART • More complex designs use the 1Gbps Ethernet port and unb_os Nios2 CPU (Server) UART 1G Eth FPGA FPGA Design Module UART JTAG Control 1G Eth Design Module Control Computer Windows/Linux (Client) UniBoard windows/unix Ethernet Switch Ethernet Switch CASPER Workshop, 10-14th October 2011, Pune, India
Control System – UNB_OS & Erlang UNB_OS • UNB_OS is a compact, 14kB embedded interrupt handling program which runs on the NIOS microcontroller • It executes UDP command packets received from the client • Because UDP packets can be lost, each command packet is acknowledged by its unique packet sequence number • Commands are available to read, write and modify any location in the NIOS address space ERLANG • The client side is written in ERLANG, a high level functional programming language (i.e. side-effect free) developed for monitoring and control of telecommunications hardware where reliability is essential • It is specifically designed for manipulating bits in registers in hardware • ERLANG’s concurrency allows you to control many UniBoards as easily as one • List based. No variables means fewer bugs • A client library has been developed to provide access to bits, words and blocks of words in the Nios address space via UDP/IP • Operations can be performed simultaneously on more than one FPGA CASPER Workshop, 10-14th October 2011, Pune, India
Applications Current and possible future applications include • Next generation EVN2015 and Apertifcorrelator (JIVE/ASTRON) • APERTIF beamformer (ASTRON) • Digital receiver (INAF + BORD) • Pulsar binning machine + RFI mitigation : (UMAN + UORL) • Shanghai observatory digital receiver (SHAO) • All dipoles Lofar station correlator(Oxford, ASTRON) • Next gen. Chinese e-VLBI correlator • prepSKAcorrelator effort CASPER Workshop, 10-14th October 2011, Pune, India
EVN Correlator CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard UniBoard dish WAN 10 GbE switch Digital Receiver One per station Correlator Sub-bands from one station spread over several UniBoards Up to 32 Stations EVN correlator at JIVE EVN Correlator 32 Stations, 2 polarizations 2, 4 or 8 bit sampling, VDIF format 64MHz per UniBoard, expandable by adding UniBoards CASPER Workshop, 10-14th October 2011, Pune, India
EVN Correlator Signal Flow pkt_rx Polyphase filter bank Interface FN packetize 10G ports Mixer Pre- filter FFT Phase shift Normalize Truncate To 9 bits Port buffer Port control Port validity LO Port control Delay & Phase Correction FN 1G Eth Store coeffs Evaluate models Phase LUT Nios BN control 1G Eth Nios Interface BN Corner Turner Correlator Port Sync De- packetize CT In Mux CE 0 Out Mux Validity acc 0 output Port In Mux1 CE 1 Out Mux control Port Port Validity acc 1 Packetize 10GbE CASPER Workshop, 10-14th October 2011, Pune, India
EVN Correlator – Scaling to 128MHz • Current design processes 64MHz per UniBoard, we aim for 128MHz • DSP elements are plentiful, on-chip SRAM is the limiting resource • At 266MHz the DSP elements meet timing easily, but feeding data in and out through large multiplexers is difficult • To use every multiplier on the chip we would need more of everything else: registers, SRAM and routing resources CASPER Workshop, 10-14th October 2011, Pune, India
Fitting the Correlator - BN • Correlator engine consists of 132 MAC cells • By default the design is flattened and hierarchy is lost • ‘LogicLock’ regions force the fitter to place related logic together • Placing each MAC cell’s multiplier, accumulator and SRAM in a LogicLock region improved timing substantially • Overuse of LogicLock regions made timing worse and caused resource conflicts CASPER Workshop, 10-14th October 2011, Pune, India
Fitting the Correlator - FN • SOPC system placed to the side out of the way of more critical design elements • Timing of FFTs improved when they were placed in tightly packed LogicLock regions CASPER Workshop, 10-14th October 2011, Pune, India
Digital Receiver • Decimation in time FFT algorithm: • separate data flow for parallel decimated samples • Tuneable filters to select arbitrary portions of input band • Output as VDIF frames over UDP stream CASPER Workshop, 10-14th October 2011, Pune, India
APERTIF Requirements Beam former: • 12 Westerbork 25 m dishes each with a Focal Plane Array • 60 dual polarization antennas per telescope • 400 MHz RF input bandwidth • 300 MHz beam output bandwidth • 37 beams Correlator: • 12 dual polarization FPA telescopes • 37 beams, so in total 11000 visibilities CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard for APERTIF Beamformer CASPER Workshop, 10-14th October 2011, Pune, India
UniBoard for APERTIF Correlator CASPER Workshop, 10-14th October 2011, Pune, India
SKA Mid-frequency Correlator 256 telescopes single pixel feeds 1024 MHz instantaneous bandwidth 2 pols 8 bits representation 7.5 kHz maximum resolution 0.1 s minimum dump time Feasible… 384 Boards ~170kW ~3.8Meuro CASPER Workshop, 10-14th October 2011, Pune, India
Next – UniBoard2 • Possible Joint Research Activity in RadioNet3, follow-up of current project, start date 2012 (if approved by EC) (looking good) • Received strong support from RadioNet community • Same basic idea, development of generic hardware complemented by a number of applications • Consolidate and build on expertise obtained through UniBoard project • Strong emphasis on power efficiency (green computing) • Production-ready in 2015/2016 • Complete re-design, using the next generation 28 nm FPGAs, possibly one generation beyond that (some slack in start date of project) • Non-leaded components • Possible use of 40GE, 100GE • Investigation into effects of hard-copy and partial hard-copy • Tuning of algorithms and firmware design to minimize power consumption • Balancing of system parameters and performance to minimize power consumption • Standardized interfaces and coding conventions to facilitate sharing and re-use of firmware blocks among developers of different applications CASPER Workshop, 10-14th October 2011, Pune, India