Linear-time Encodable/Decodable Error-Correcting Codes

1. Linear-time Encodable/Decodable Error-Correcting Codes Some results from two papers of M. Sipser and D. Spielman1 1Much of the technical wording within is either taken verbatim or paraphrased from [SS96; S96]; see references.

2. Coding for Low-Power Communication Across Noisy Channels • Static on the line / Game of “Operator” (Spielman’s lecture). • A coding solution: “A” as in “Apple”, “B” as in “Baseball”, …”Z” as in Zebra.

3. Messages (live in low-d space) minimum distance bound (Encode) Codewords(live in high-d space)= Message Bits + Check Bits + (Decode) Diagram for codeword space inspired by digital signature lecture at MIT earlier this year

4. Motivation • Recall one question answered in professor Spielman’s lecture: • Q1: Do there exist families of codes which as block length grows realize: • constant information rate / • bounded relative minimum distance • A: Yes! (e.g. Justesen’s explicit construction) • Q2: Can we construct such a family of codes with linear time algorithms for encoding AND decoding a constant fraction of error? • A: Yes again!

5. Goals • Describe randomized and explicit constructions of families of asymptotically gooderror-correcting codes • Describe linear-time encoding and decoding algorithms for these codes • Sketch proofs of correctness for the decoding algorithms • Motivate why these algorithms are in fact linear-time

6. Sights and sounds along the way… • Linear-time decodable error-correcting codes (built from expander graphs and existing code) • Randomized construction • Explicit construction • Linear-time encodable/error-reducible Error-reduction codes (also from expander graphs and existing code) • Explicit construction • Linear-time encodable/decodable error-correcting codes (from error-reduction codes)

7. …and mathematical flavors • Graph theory • Clever correspondences between graphs and codes,graphs and graphs. • Expander and regularity properties of underlying graphs used to establish • minimum distance bounds • correctness of decoding algorithms • Regularity also used to show rate, linear-time • Probability • The “probabilistic method,” or, a way to prove existence theorems non-constructively • How to identify properties of randomly constructed objects

8. Models of Computation for Linear-Time1 • Cost of Memory access / Bit-wise operations (as a function of argument size n): • Uniform cost model => 1 time unit • Logarithmic cost model => log n time units • Machines for sequential decoding algorithms • Graph inputted as a set of pointers; each vertex indexes a list of pointers to its neighbors2 • Pointer Machine (constant degree directed graph) • RAM • Boolean circuits for parallel decoding algorithms • Size (= # wires) • Depth (= longest path length in circuit digraph) 1 [SS96, p1710-11] 2 [SS96, p1713]

9. Important things to recall • Linear Codes: codewords form a vector space • Gilbert-Varshamov bound: “Good” linear codes exist for sufficiently long block lengths • Graphs G = (V, E) • Expander graphs (Mohamed’s lecture) • All minority subsets of vertices have many (external) neighbors (large “boundary”): Expansion factor = • Polynomial-time constructible infinite families of k-regular Ramanujan graphs (Mohamed’s lecture)1 • Realize maximum possible “spectral gap” (~expansion) • Explicit construction of Lubotsky-Philips-Sarnak/Margulis using Cayley graphs2 1 [LL00 p1] • 1 SS [1, p1712]

10. Example: A (2,3)-regular graph: • “Unbalanced”, (c,d)-regular bipartite graphs • Regularity important for showing linear-time; also for counting arguments used to prove correctness of decoding algorithms. Six 2-regular Four 3-regular vertices vertices • Degree ratio c/d determines bipartite ratio (edge count)

11. Defining expansion for (c,d)-regular graphs • Only consider subsets of the c-regular vertices (LHS) Definition1: A (c,d)-regular graph is a -expander if every subset of at most an -fraction of the c-regular vertices expands by a factor of at least . 1 [SS96, p1711]

12. Building an Expander Code:C(B,S) • Necessary Ingredients • a (c,d)-regularexpander graph (c<d) • a good block-length error-correcting code (existence by the Gilbert-Varshamov bound) • A correspondence between the graph and the codewords of .

13. c d n variables (c-regular) constraints (d-regular)

14. Corresponding (c-regular) vertices (d-regular) vertices variablesconstraints

15. The expander code1 : • A constraint is satisfied if its d neighboring variables form a codeword of the input code S. • The expander code C(B,S) is the code of block length whose codewords are the settings on the variables such that all constraints are satisfied simultaneously. • The (linear) code C(B,S) is the vector subspace which solves the set of all linear constraints • The constraints collectively determine a check matrix H (from which generator matrix G can be derived) 1 [SS96, p1712]

16. C(B,S) is asymptotically good1 • For: • Graph B with good enough expansion • Code S with bounded rate and minimum distance • Conclusion: • C(B,S) is a code of (related) bounded rate and minimum distance, for all block lengths n. Theorem1: Let be a -expander and an error-correcting code of block length , rate , and minimum relative distance . Then C(B,S) has rate at least and minimum relative distance at least . (note that these quantities are independent of the code’s block length ). 1[SS96, p1712]

17. Proof1: Rate Bound (Relies on regularity & rate of S) • Count the total number of linear restrictions imposed on the subspace of codewords • Assume all restrictions are mutually independent • Dimension of solution space = # degrees of freedom left out of block length n • Corresponds to the # message bits in code => value for rate 1 [SS96, p1712]

18. Proof (continued): Min Distance Bound • Sufficient to show that no codeword w exists below a certain weight. • Contradiction by regularity and expansion: • Regularity fixes the number of edges leaving the subset of all 1 (non-zero) variables in w • Expansion guarantees these edges enter a lot of constraints • => Average #edges/constraint low • => there must be some constraint with few 1 variables as neighbors • By minimum distance bound of underlying code S, entire word can’t be a codeword of C(B,S).

19. Encoding and Decoding C(B,S) • Encoding • quadratic time algorithm (multiply message by generator matrix G) • Decoding • Brute force • Linear time algorithms • Challenge: Correctness proof requires constructing a graph B with expansiongreater than c/2; no explicit constructions are known for such graphs.

20. Constructing a (c,d)-regular graph with very good expansion (>c/2) • Explicit constructions: Best known (1996) can give expansion c/2, but no greater: • (Lubotsky-Phillips-Sarnak, Margulis 1988)1: Ramanujan graphs: (p+1)-regular, second-largest eigenvalue • (Kahale 1992)2: Lubotsky et al.’s Ramanujan graphs have quality expansion, but spectral gap can not certify expansion greater than c/2 • Random(c,d)-regular graphs will almost always have greater expansion • Food for thought: How could this random construction be done? • 1 [SS96, p1712] • 2[SS96, p1713]

21. Random (c,d)-regular graphs: How good is their expansion? • For B a randomly chosen (c,d)-regular bipartite graph between n variables and (c/d)n constraints, for all , with exponentially high probability any set of variables will have at least neighbors1. Pf: Bound probability that the # neighbors of any subset is far from its expected # neighbors 1[SS96, p1721]

22. Decoding a RandomExpander Code C(B,S) • Flip variable(s) in more unsatisfied than satisfiedconstraints until none remain • Sequential Algorithm • Parallel Algorithm • Proof of correctness (sequential algorithm): • Relies on expansion and regularity properties of graph B, as well as nature of code S. Sufficient to show: • As long as there are some corruptvariables (but # below threshold), then somevariable will be flipped. • =>Number of unsatisfiedconstraintsstrictly decreases • Linear time relies only on regularity of graphB. • Total # (unsatisfied) constraintslinear in block length • Local graph structure unchanged as block length grows; fixed code S=> constant local cost

23. Remarks on Construction • Expansion factor required: for efficient decoding of one example code • Bound on rate assumed all constraints imposed on variables were independent • Potential applications for codes with some redundancy in constraints(see Spielman’s thesis1, Ch.5 PCP’s and “checkable” codes) 1http://www-math.mit.edu/~spielman/Research/thesis.html

24. Explicit Construction of Expander Code C(B,S) • To construct the code C(B,S), we will take • B the edge-vertex incidence graph of a d-regular Ramanujan graph G (Lubotsky et al: A dense family of goodexpander graphs)1 • S a good block length d code 1[S96, p1726]

25. Edge-vertex incidence graphs • Definition: Let G be a graph with edge set E and vertex set V. The edge-vertex incidence graphB(G) of G is the bipartite graph with vertex set and edge set: • Thus the edge-vertex incidence graph B(G) of our d-regular Ramanujan graph G will be a (2,d)-regular bipartite graph, to which we correspond dn/2 variables and n constraints.

26. Edges E of G Vertices V of G 2 d variables (2-regular) constraints (d-regular)

27. Corresponding EdgesE of G VerticesV of G (2-regular) vertices (d-regular) vertices variablesconstraints

28. Parallel Decoding for C(B,S)1(where S has block length d, min rel dist ) • For each constraint, if the variables in that constraint differ from a codeword of S in at most places, then send a “flip” message to each variable that differs. • In parallel, flip the value of every variable which receives at least one flip message. 1[SS96, p1716]

29. Main Theorem1 • There exist polynomial-time constructible families of asymptotically good expander codes in which a constant fraction of error can be corrected in a circuit of size and depth . • The action of this circuit can be simulated in linear time on a Pointer Machine or a RAM under the uniform cost model. (No natural sequential decoding algorithm is known). 1 [SS96, p1716]

30. Proof Sketch • Outline similar to the proof above for the random expander case • One important difference: • Above, it was the known expansion factor of graph B which was used to prove • Minimum distance bound • Correctness of decoding algorithm • Here, instead apply a result of Alon-Chung to upper bound the number of edges contained in a subgraph of a graph G with a given spectral gap. • Convenient, since our edge-vertex incidence graph B(G) is built from a Ramanujan graph G (Lubotsky) that is qualified in terms of spectral gap, not expansion .

31. (Alon-Chung)1: Let G be a d-regular graph on n vertices with second largest eigenvalue . Let X be a subset of the vertices of G of size . Then, the number of edges in the subgraph induced by X in G is at most 1 [SS96, p1712]

32. Could use Alon-Chung bound to quantify the expansion factor of the edge-vertex incidence graph B(G) • But a direct application of Alon-Chung itself works well in proof • => each decoding round removes a constant fraction* of error from input words within threshold of nearest codeword. *(fraction <1 for sufficiently large choices of block length d of S) • => after O(log n) rounds, no error remains.

33. Circuit for Parallel Decoding1 • Each parallel decoding round can be implemented by a circuit of linear size and constant depth • constraint satisfaction: constant # of layers of XOR’s • variable “flips”: majority of constant # of inputs • => Total of O(log n) rounds needs a circuit of size O(n log n), depth O(log n). 1 [SS96, p1715]

34. Linear Time of Sequential Simulation (Uniform Cost Model) • During each round, compile a list of all constraints which might be unsatisfied at the beginning of next round by including: • All constraints of previous round’s list which actually were unsatisfied • All constraintsneighboring at least one variable which received a “flip” message in current round • Total amount of work performed by algorithm is linear in the combined length of these lists: • Each round: Check which constraints from input list are actually unsatisfied;dodecodes on S; send “flips” • Regularity, fixed code S=> constant local cost • Total length of these lists is linear in block length n of the code C(B,S). • # variable errors decreases by constant ratio each round • Only constraints containing these variables appear in lists • => convergent geometric series, first term O(n)

35. Linear Time of Sequential Simulation (Logarithmic Cost Model) Challenge: • Need O(n) bit additions and O(n) memory accesses • Cost per access is O(log n) • Total cost O(n log n) Solution: • Retrieve an O(log n) byte per access, only need to make O(n/log n) accesses; total cost O(n) • Encode each byte with a good linear error-correcting code • Algorithm will correct a constant fraction of error in overall code unless there is a constant fraction of error in the encoding of a constant fraction of bytes.

36. Expander Codes Wrap-Up1 • Error distance from a correct codeword was corresponded to a potential1 (of unsatisfiedconstraints) in a graph whose expander and regularity properties guarantee the success of a linear-time decoding algorithm which steadily decreases this potential to zero. • Quadratic time encoding / linear time decoding • Superior speed of sequential decoding on randomexpanders • Can tune parallel decoding algorithm for better performance 1[SS96, p1718-20]

37. Expander Codes Wrap-Up • Average-case error-correction results in experiments much better than bounds predict (worst-case). • Introducing random errors in variable flips can further improve performance. • For randomexpanders, degree 5 at variables seemed to give best results.

38. Linear-time Encodable/Decodable Error-Correcting Codes • Expanders leveraged to produce linear time encodable/decodable error-reduction codes • Construction involving a recursive assembly of smaller error-reduction & error-correcting codes gives linear time error-correcting codes. • Decoding algorithm, proof of its correctness are the same flavor as those already encountered. • Randomized / Explicit constructions • Explicit case shown here is a generalization of the randomized one

39. Historical Context1 • Prior constructions did not always use such well-defined models of computation or deal with asymptotic complexity • Juntesen and Sarwate used efficient implementation of polynomial GCD algorithm to get O(n log n log log n) encoding, O(n log2 n log log n) decoding for RS and Goppa codes. • Gelfand, Dobrushin, and Pinsker gave randomized constructions for asymptotically good codes with O(n) encoding; but no polynomial-time algorithm for decoding is known. 1 [S96, p1723]

40. Main Theorem1 • There exists a polynomial-time constructible family of error-correcting codes of rate ¼ that have linear-time encoding algorithms and linear-time decoding algorithms for a constant fraction of error. • The encoding can be performed by linear-size circuits of logarithmic depth and decoding can be performed by circuits of size O(n log n) and O(log n) depth, and can be simulated sequentially in linear time. [S96, p1729]

41. Error-Reduction Codes • Definition: A code C is an error-reduction code if there is an algorithm ER which, given an input word w with total # errors below threshold, can reduce the # of message bit errors in w to a constant fraction of the # of check bit errors originally present in w. • Note: for zerocheck bit errors, implies error-correction of all errors in message bits. • Can construct such codes from good expanders

42. Building the Error-Reduction Code: R(B,S) • Necessary Ingredients • B a (c,d)-regularexpander graph (c<d) • S a good block length d error-correcting code with k message bits (rate = k/d) • A correspondence between the graph B and the codewords of R(B,S).

43. c d check bit clusters (d-regular) variables (c-regular)

44. Corresponding (c-regular) vertices (d-regular) vertices message bitscheck bit clusters

45. The error-reduction code R(B,S) 1 • The error-reduction code R(B,S) will have n message bits and (c/d)nk check bits. • The n message bits are identified with the n c-regular vertices. • A clusterCi of k check bits is identified with each of the (c/d)n d-regular vertices. • The check bits of cluster Ci are defined to be the check bits of that codeword in S which has the d neighbors of cluster Ci as its message bits • R(B,S) can be encoded in linear time, (constant time for generating each check bitcluster; linear # of clusters) 1 [S96, p1726]

46. Parallel Error-Reduction round of R(B,S)1(where is the minimum distance of S) • In parallel, for each cluster, if the check bits in that cluster and the associated message bits are within relative distance of a codeword, then send a “flip” signal to every message bit that differs from the corresponding bit in the codeword. • In parallel, every message bit that receives at least one “flip” message flips its value. [S96,p1728]

47. Quality of an Error-Reduction Round on R(B,S) • Take B as the edge-vertex incidence graph of a d-regular Ramanujan graph G (Lubotsky et al: A dense family of goodexpander graphs)1 • Can show every error reduction round removes a constant fraction* of error from the message bits of input words which are within threshold of nearest codeword. *(fraction <1 for sufficiently large choices of block length d of S) 1[S96, p1726]

48. Proof of Quality of Error-Reduction • Analogous to that seen above for error-correction of explicitly constructed expander codes • Relies on the same graph properties: • Regularity (and fixed code S) • theorem of Alon-Chung (~expansion)

49. Linear Time of Error-Reduction on R(B,S) • Argument analogous to that for linear-time simulation of parallel error-correction for the explicitly constructed expander codes above • For good enough eigenvalue separation (~expansion) of graph B & good enough min distance of code S, after O(log n) rounds, error in message bits will be reduced to within threshold required by recursive decoding of overall error-correcting code Ck.

50. Recursive Definition of the Error-Correcting Codes Ck