A System to Detect Complex Motion of Nearby Vehicles on Freeways

A System to Detect Complex Motion of Nearby Vehicles on Freeways C. Y. Fang Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. P. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. S. Fuh Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R. O. C. S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C. violet@ice.ntnu.edu.tw

Outline • Introduction • Dynamic visual model (DVM) • Detect complex motion of nearby vehicles • Feature extraction • Attention map partition • Collection of classification results • Temporal integral process • Experimental Results • Conclusions violet@ice.ntnu.edu.tw

Introduction -- VDAS • System to detect motion of nearby vehicles: a Vision-based driver assistance system (VDAS) • Advantages: • High resolution • Rich information • Difficulties of VDAS • Weather and illumination • Daytime and nighttime • Vehicle motion and camera vibration violet@ice.ntnu.edu.tw

Introduction -- DVM • DVM: dynamic visual model • A computational model for visual analysis using video sequence as input data • Two ways to develop a visual model • Biological principles • Engineering principles • Artificial neural networks violet@ice.ntnu.edu.tw

Improved DVM • Two problems: • The motions of vehicles may occur anywhere on the road. • Training a CART neural network to recognize various complex motions is quite difficult. • Solutions: • Feature extraction • Attention map partition • Collection of classification results • Temporal integral process violet@ice.ntnu.edu.tw

Video images Data transduction Sensory component Episodic Memory Information acquisition Spatialtemporal information Perceptual component STA neural module No Focusesofattention Yes Windowing Feature extraction Feature extraction Feature extraction Features Features Features Conceptual component CART CART CART Category Category Category Decision making No Confirm? Yes Action The Improved Dynamic Visual Model

Introduction • Motions of the Vehicles • Lane change • Speed change • Objective • Simple motion detection • Complex motion detection violet@ice.ntnu.edu.tw

Simple Motion Patterns violet@ice.ntnu.edu.tw

b5 b4 b1 b2 b3 Attention Map Partition violet@ice.ntnu.edu.tw

1 2 3 4 5 6 7 8 9 10 gi1 i i gi1 1 2 3 4 5 6 7 8 9 10 Feature Extraction (1) ---Skewness features violet@ice.ntnu.edu.tw

Feature Extraction (2) • The horizontal skewness features: gi1 : the skewness of intensity value mi2, mi3: the normalized second and third moments, respectively :the column means of intensity values, : the mean horizontal position of the intensity means violet@ice.ntnu.edu.tw

D t CART1 CART2 CARTi CARTn Classification Results violet@ice.ntnu.edu.tw

Temporal Fuzzy Integral (1) • Let nbe the number of CART neural networks • : the output strings of labels of CARTifrom time t-ri+1 tot • , k = 1, 2, …, ri • : the set of all labels, where l0 is null label • pj: the stored pattern corresponding to label lj • P : the set containing all stored patterns • ri: time period violet@ice.ntnu.edu.tw

Temporal Fuzzy Integral (2) • Fuzzy measure function where #p:the number of non-zero pixels of one stored pattern : the number of such pixels falling in the union of windows i or j . violet@ice.ntnu.edu.tw

Some Values of Fuzzy Measure Function violet@ice.ntnu.edu.tw

Temporal Fuzzy Integral (3) • Confidence function where j, k = 1, 2,…, ri, : a distance between pj and pk , : weight functions , : positive parameters violet@ice.ntnu.edu.tw

Temporal Fuzzy Integral (4) • Fuzzy integral: the integral value for : the fuzzy intersection characterized by a t-norm violet@ice.ntnu.edu.tw

Intermediate Decision of Individual CARTi where : a distance threshold violet@ice.ntnu.edu.tw

Collection of Classification Results • The final classification set where , : the corresponding integral value of : a threshold violet@ice.ntnu.edu.tw

b5 b4 b1 b2 b3 Experimental Results violet@ice.ntnu.edu.tw

b5 b4 b1 b2 b3 Experimental Results (1) violet@ice.ntnu.edu.tw

Complex Motion Sequence A B C violet@ice.ntnu.edu.tw

Experimental Results • Simple motion sequences • 12 sequences • accuracy rate: 97.9% • Complex motion sequences • 18 sequences • accuracy rate: 93.3% • Since our system only outputs a result for each input sequence, this ratio is enough for our system to recognize road signs correctly. violet@ice.ntnu.edu.tw

Experimental Results violet@ice.ntnu.edu.tw

Discussion • Improve attention map partition • Detect other dynamic obstacles violet@ice.ntnu.edu.tw

Conclusions • A neural-based dynamic visual model • Three major components: sensory, perceptual and conceptual component • Future Researches • Potential applications • Improvement of the DVM structure • DVM implementation violet@ice.ntnu.edu.tw

Physical stimuli Data compression Transducer Low-level feature extraction Sensory analyzer High-level feature extraction Perceptual analyzer Classification and recognition Conceptual analyzer Class of input stimuli Human Visual Process violet@ice.ntnu.edu.tw

Neural Modules • Spatial-temporal attention (STA) neural module • Configurable adaptive resonance theory (CART) neural module violet@ice.ntnu.edu.tw

STA Neural Network (1) ak ai Output layer (Attention layer) nk ni Inhibitory connection wij Excitatory connection xj nj Input layer violet@ice.ntnu.edu.tw

Gaussian function G Attention layer ni rk nk corresponding neurons wkj nj Input neuron The linking strengths between the input and the attention layers STA Neural Network (2) • The input to attention neuron nidue to input stimuli x: violet@ice.ntnu.edu.tw

Interaction + Lateral distance “Mexican-hat” function of lateral interaction STA Neural Network (3) • The input to attention neuron ni due to lateral interaction: violet@ice.ntnu.edu.tw

STA Neural Network (4) • The net input to attention neuron ni : : a threshold to limit the effects of noise where 1< d <0 violet@ice.ntnu.edu.tw

STA Neural Network (5) stimulus activation t 1 1 p pd The activation of an attention neuron in response to a stimulus. violet@ice.ntnu.edu.tw

Orienting subsystem Attentional subsystem Category representation field F2 y Signal generator Reset signal S Input representation field F1 ＋ q ＋＋ r ＋－＋－－＋＋ G p ＋ G ＋＋ G G － v ＋＋＋ u ＋＋ x －＋ G ＋ w ＋ Input vector i ART2 Neural Network (1) CART violet@ice.ntnu.edu.tw

ART2 Neural Network (2) • The activities on each of the six sublayers on F 1: where I is an input pattern where where the J th node on F 2 is the winner violet@ice.ntnu.edu.tw

ART2 Neural Network (3) • Initial weights: • Top-down weights: • Bottom-up weights: • Parameters: violet@ice.ntnu.edu.tw

A System to Detect Complex Motion of Nearby Vehicles on Freeways