1 / 24

Implementation of the Python Bytecode Compiler

Implementation of the Python Bytecode Compiler. Jeremy Hylton Google. What to expect from this talk. Intended for developers Explain key data structures and control flow Lots of code on slides. The New Bytecode Compiler. Rewrote compiler from scratch for 2.5 Emphasizes modularity

betsy
Download Presentation

Implementation of the Python Bytecode Compiler

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Implementation of the Python Bytecode Compiler Jeremy Hylton Google

  2. What to expect from this talk • Intended for developers • Explain key data structures and control flow • Lots of code on slides

  3. The New Bytecode Compiler • Rewrote compiler from scratch for 2.5 • Emphasizes modularity • Work was almost done for Python 2.4 • Still uses original parser, pgen • Traditional compiler abstractions • Abstract Syntax Tree (AST) • Basic blocks • Goals • Ease maintenance, extensibility • Expose AST to Python programs

  4. Compiler Architecture Source Text Tokenizer Tokens Parser Parse Tree AST Converter AST Code Generator Blocks __future__ Symbol Table Assembler bytecode bytecode Peephole Optimizer bytecode

  5. Compiler Organization

  6. Tokenize, Parse, AST • Simple, hand-coded tokenizer • Synthesizes INDENT and DEDENT tokens • pgen: parser generator • Input in Grammar/Grammar • Extended LL(1) grammar • ast conversion • Collapses parse tree into abstract form • Future: extend pgen to generator ast directly

  7. Grammar vs. Abstract Syntax compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | funcdef | … if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite] for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite] suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT test: and_test ('or' and_test)* | lambdef and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison comparison: expr (comp_op expr)* comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not‘ stmt = For(expr target, expr iter, stmt* body, stmt* orelse) | If(expr test, stmt* body, stmt* orelse) | … expr = BinOp(expr left, operator op, expr right) | Compare(expr left, cmpop* ops, expr* comparators) | Call(expr func, expr* args, keyword* keywords, expr? starargs, expr? kwargs) | …

  8. AST node types • Modules (mod) • Statements (stmt) • Expressions (expr) • Expressions allowed on LHS have context slot • Extras • Slots, comprehension, excepthandler, arguments • Operator types • FunctionDef is complex • Children in two namespaces

  9. Example Code L = [] for x inrange(10): if x > 5: L.append(x * 2) else: L.append(x + 2)

  10. Concrete Syntax Example (if_stmt, (1, 'if'), (test, (and_test, (not_test, (comparison, (expr, (xor_expr, (and_expr, (shift_expr, (arith_expr, (term, (factor, (power, (atom, (1, 'x')))))))))), (comp_op, (21, '>')), (expr, (xor_expr, (and_expr, (shift_expr, (arith_expr, (term, (factor, (power, (atom, (2, '5')))))))))))))), (11, ':'), …

  11. Abstract Syntax Example For(Name('x', Load), Call(Name('range', Load), [Num(10)]), [If(Compare(Name('x', Load), [Lt], [Num(5)]), [Call(Attribute(Name('L', Load), Name('append', Load)), [BinOp(Name('x', Load), Mult, Num(2))])] [Call(Attribute(Name('L', Load), Name('append', Load)), [BinOp(Name('x', Load), Add, Num(2))])])])

  12. 2 0 BUILD_LIST 0 3 STORE_FAST 1 (L) 3 6 SETUP_LOOP 71 (to 80) 9 LOAD_GLOBAL 1 (range) 12 LOAD_CONST 1 (10) 15 CALL_FUNCTION 1 18 GET_ITER >> 19 FOR_ITER 57 (to 79) 22 STORE_FAST 0 (x) 4 25 LOAD_FAST 0 (x) 28 LOAD_CONST 2 (5) 31 COMPARE_OP 4 (>) 34 JUMP_IF_FALSE 21 (to 58) 37 POP_TOP 5 38 LOAD_FAST 1 (L) 41 LOAD_ATTR 3 (append) 44 LOAD_FAST 0 (x) 47 LOAD_CONST 3 (2) 50 BINARY_MULTIPLY 51 CALL_FUNCTION 1 54 POP_TOP 55 JUMP_ABSOLUTE 19 >> 58 POP_TOP 7 59 LOAD_FAST 1 (L) 62 LOAD_ATTR 3 (append) 65 LOAD_FAST 0 (x) 68 LOAD_CONST 3 (2) 71 BINARY_ADD 72 CALL_FUNCTION 1 75 POP_TOP 76 JUMP_ABSOLUTE 19 >> 79 POP_BLOCK Our Goal: Bytecode

  13. Strategy for Compilation • Module-wide analysis • Check future statements • Build symbol table • For variable, is it local, global, free? • Makes two passes over block structure • Compile one function at a time • Generate basic blocks • Assemble bytecode • Optimize generated code (out of order) • Code object stored in parent’s constant pool

  14. Symbol Table • Collect basic facts about symbols, block • Variables assigned, used; params, global stmts • Check for import *, unqualified exec, yield • Other tricky details • Identify free, cell variables in second pass • Parent passes bound names down • Child passes free variables up • Implicit vs. explicit global vars

  15. Name operations • Five different load name opcodes • LOAD_FAST: array access for function locals • LOAD_GLOBAL: dict lookups for globals, builtins • LOAD_NAME: dict lookups for locals, globals • LOAD_DEREF: load free variable • LOAD_CLOSURE: loads cells to make closure • Cells • Separate allocation for mutable variable • Stored in flat closure list • Separately garbage collected

  16. Class namespaces classSpam: id = id(1) 1 0 LOAD_GLOBAL 0 (__name__) 3 STORE_NAME 1 (__module__) 2 6 LOAD_NAME 2 (id) 9 LOAD_CONST 1 (1) 12 CALL_FUNCTION 1 15 STORE_NAME 2 (id) 18 LOAD_LOCALS 19 RETURN_VALUE

  17. Closures defmake_adder(n): x = n defadder(y): return x + y return adder return make_adder defmake_adder(n): 2 0 LOAD_FAST 0 (n) 3 STORE_DEREF 0 (x) 3 6 LOAD_CLOSURE 0 (x) 9 LOAD_CONST 1 (<code>) 12 MAKE_CLOSURE 0 15 STORE_FAST 2 (adder) 5 18 LOAD_FAST 2 (adder) 21 RETURN_VALUE defadder(y): 4 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE

  18. Discriminated unions One for each AST type Struct for each option Constructor functions Literals Stored as PyObject* ast pass parses Identifiers Also PyObject* string typedef struct _stmt *stmt_ty; struct _stmt { enum { ..., For_kind=8, While_kind=9, If_kind=10, ... } kind; union { struct { expr_ty target; expr_ty iter; asdl_seq *body; asdl_seq *orelse; } For; struct { expr_ty test; asdl_seq *body; asdl_seq *orelse; } If; } int lineno; }; Code generation input

  19. Basic blocks Start with jump target Ends if there is a jump Function is graph of blocks Instructions Opcode + argument Jump targets are pointers Helper functions Create new blocks Add instr to current block struct instr { unsigned char i_opcode; int i_oparg; struct basicblock_ *i_target; int i_lineno; // plus some one-bit flags }; struct basicblock_ { int b_iused; int b_ialloc; struct instr *b_instr; struct basicblock_ *b_next; int b_startdepth; int b_offset; // several details elided }; Code generation output

  20. Code generation • One visitor function for each AST type • Switch on kind enum • Emit bytecodes • Return immediately on error • Heavy use of C macros • ADDOP(), ADDOP_JREL(), … • VISIT(), VISIT_SEQ(), … • Hides control flow

  21. Code generation example static int compiler_if(struct compiler *c, stmt_ty s) { basicblock *end, *next; if (!(end = compiler_new_block(c))) return 0; if (!(next = compiler_new_block(c))) return 0; VISIT(c, expr, s->v.If.test); ADDOP_JREL(c, JUMP_IF_FALSE, next); ADDOP(c, POP_TOP); VISIT_SEQ(c, stmt, s->v.If.body); ADDOP_JREL(c, JUMP_FORWARD, end); compiler_use_next_block(c, next); ADDOP(c, POP_TOP); if (s->v.If.orelse) VISIT_SEQ(c, stmt, s->v.If.orelse); compiler_use_next_block(c, end); return 1; }

  22. Assembler • Lots of fiddly details • Linearize code • Compute stack space needed • Compute line number table (lnotab) • Compute jump offsets • Call PyCode_New() • Peephole optimizer • Integrated at wrong end of assembler • Constant folding, simplify jumps

  23. AST transformation • Expose AST to Python programmers • Simplify analysis of programs • Generate code from modified AST • Example: • Implement with statement as AST transform • Ongoing work • BOF this afternoon at 3:15, Preston Trail

  24. Loose ends • compiler package • Should revise to support new AST types • Tricky compatibility issue • Revise pgen to generate AST directly • Develop toolkit for AST transforms • Extend analysis, e.g. PEP 267

More Related