Introduction
People with visual disabilities (blind or low vision) often encounter difficulties when crossing signalized intersections.[1] Generally, the tasks involving in crossing a street include (1) detecting, locating, and aligning the crossing location, (2) determining an appropriate time to cross, and (3) traveling in a straight path while crossing.[2] However, it is difficult for people with visual disabilities to execute those tasks precisely and perform accurate assessments without assistance. With the development of modern communicating technology and mobile equipment, the ideal and reliable solution in the future may be device-to-infrastructure communication (which mimics vehicle-to-infrastructure communication for self-driving vehicles).[3] Nonetheless, due to the lack of those infrastructures, vision-based traffic light/walking path detection, and recognition still play significant roles in the current traffic assistance system.
Several machine learning or convolution neural network (CNN) -based traffic light detection and recognition models using public (eg, LaRA, VIVA, WPI, Bosch) or self-constructed databases have been published.[3, 4, 5] The majority of those models consisted of two major components: (1) an object detector to find a region of interest on photos (i.e. traffic lights), and (2) a machine learning/deep learning/CNN-based classifier to recognize traffic lights signals (i.e. whether an intersection is able to cross). While achieving >90% of accuracy in recognizing traffic lights, most of those models were developed for self-driven vehicles. Pedestrians, especially people with visual disabilities, often need more assistance when crossing intersections. Roni Ash et al reported a model consisted of a self-constructed image/video dataset, an object detector, and a decision module.[6] Still, the model is not capable of detecting obstacles on roads.
In this project, we propose a CNN-based model to help pedestrian crossing intersections. The model consisted of traffic light detection and recognition, street width assessment, and obstacles evaluation. We also collected images and constructed a labeled database to train and test our model. By recognition pedestrian traffic lights, estimating an appropriate time to cross, and detecting dangerous objects on the walking path, users could know whether a crossing is safe to walk. We hope the model could benefit people with visual disabilities and could help develop applications that run on a mobile device.
Problem Statement
The primary outcome in our model is to assess whether an intersection is safe to cross or not. We also seek to identify the crossing path (zebra crossing) and give users instruction about the appropriate time needed to pass. We will utilize self-constructed datasets (real-world cross-road images in the view of pedestrians) and a fine-tuned CNN-based model. We expected to see our model successfully identifying traffic lights on images, recognizing traffic signals (i.e. the probability of the lights in red, yellow, green respectively), predicting whether a designated crossroad is safe to walk (0: stop, 1: safe), and calculating the appropriate time needed to pass by assessing road width (different categories: 2 lines, 4 lines, 6 lines, etc.).
Approach
Database and Data Preprocessing
To evaluate whether an intersection is safe to walk for pedestrians, we collected street images and evaluated them according to the following parameters: (1) traffic light signals, (2) zebra crossing, (3) obstacles in walking path (4) weathers, and (5) crossroad width (estimated by Google Map satellite image). We then labeled images as 0: stop/not able to pass or 1: safe to pass. If a complicated traffic condition was presented in the image or there was difficulty assessing whether an intersection is safe to pass, we use the method of majority decision to determine the label.
We utilize pandas, PIL, and matplotlib packages in data preprocessing. To regularize our images, we converted all our images into jpg format and rename them. Also, we crop our images to the size of 900x900. To enhance training efficacy, we pick out images labeled 0 and 1 in the ratio of 1:1. Then, we split our database randomly into training data/validation data/testing data in a 90:10:10 ratio.
CNN-based Model
To constructed a model with efficiency and accuracy, we selected four pre-trained models (VGG, Inceptionv3 (GoogLeNet), ResNet, EfficientNet), fine-tuned, and trained our data those models respectively to see which model show the most promising results. Hypothetically, our model could execute the following tasks:
- detecting region of interest on images, recognizing pedestrian traffic lights in red, yellow, green (in probabilities)
- evaluating the width of the crossroad, classifying them into different categories, and calculating the appropriate time to cross
- identifying the zebra crossing region and find whether obstacles are presented
- give users instruction (0: stop; 1: pass)
Preliminary Results
Database and Data Preprocessing
Totally, 1148 images were collected and carefully labeled. To make the ratio of 0 (stop) : 1 (pass) in 1:1, 1070 photos (stop: 535 images; pass: 535 images) were selected. After splitting our data randomly into training data/validation data/testing data in a 90:10:10 ratio, we have 956 images in the training set, 57 images in the validation set, and 57 images in the testing set. Below are figures and tables showing data distribution (counts vs. width of roads) and statistics.
Reference
- Abbas Riazi, Fatemeh Riazi, Rezvan Yoosfi, and Fatemeh Bahmeeic, Outdoor difficulties experienced by a group of visually impaired Iranian people, J Curr Ophthalmol. 2016 Jun; 28(2): 85–90.
- Janet M. Barlow, M.Ed., Billie Louise Bentzen, Ph.D., and Tamara Bond, M.A., Blind Pedestrians, and the Changing Technology and Geometry of Signalized Intersections: Safety, Orientation, and Independence, J Vis Impair Blind. 2005 Oct 1; 99(10): EJ720652.
- Zhenchao Ouyang; Jianwei Niu; Yu Liu; Mohsen Guizani, et al, Deep CNN-Based Real-Time Traffic Light Detector for Self-Driving Vehicles, IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020
- Any Gupta, Ayesha Choudhary, A Framework for Traffic Light Detection and Recognition using Deep Learning and Grassmann Manifolds, 2019 IEEE Intelligent Vehicles Symposium (IV) Paris, France. June 9-12, 2019
- Julian Müller, Klaus Dietmayer, Detecting Traffic Lights by Single Shot Detection, arXiv:1805.02523 [cs.CV]
- R. Ash, D. Ofri, J. Brokman, I. Friedman and Y. Moshe, "Real-time Pedestrian Traffic Light Detection," 2018 IEEE International Conference on the Science of Electrical Engineering in Israel (ICSEE), 2018, pp. 1-5, doi: 10.1109/ICSEE.2018.8646287.
Introduction
People with visual disabilities (blind or low vision) often encounter difficulties when crossing signalized intersections.[1] Generally, the tasks involving in crossing a street include (1) detecting, locating, and aligning the crossing location, (2) determining an appropriate time to cross, and (3) traveling in a straight path while crossing.[2] However, it is difficult for people with visual disabilities to execute those tasks precisely and perform accurate assessments without assistance. With the development of modern communicating technology and mobile equipment, the ideal and reliable solution in the future may be device-to-infrastructure communication (which mimics vehicle-to-infrastructure communication for self-driving vehicles).[3] Nonetheless, due to the lack of those infrastructures, vision-based traffic light/walking path detection, and recognition still play significant roles in the current traffic assistance system.
Several machine learning or convolution neural network (CNN) -based traffic light detection and recognition models using public (eg, LaRA, VIVA, WPI, Bosch) or self-constructed databases have been published.[3, 4, 5] The majority of those models consisted of two major components: (1) an object detector to find a region of interest on photos (i.e. traffic lights), and (2) a machine learning/deep learning/CNN-based classifier to recognize traffic lights signals (i.e. whether an intersection is able to cross). While achieving >90% of accuracy in recognizing traffic lights, most of those models were developed for self-driven vehicles. Pedestrians, especially people with visual disabilities, often need more assistance when crossing intersections. Roni Ash et al reported a model consisted of a self-constructed image/video dataset, an object detector, and a decision module.[6] Still, the model is not capable of detecting obstacles on roads.
In this project, we propose a CNN-based model to help pedestrian crossing intersections. The model consisted of traffic light detection and recognition, street width assessment, and obstacles evaluation. We also collected images and constructed a labeled database to train and test our model. By recognition pedestrian traffic lights, estimating an appropriate time to cross, and detecting dangerous objects on the walking path, users could know whether a crossing is safe to walk. We hope the model could benefit people with visual disabilities and could help develop applications that run on a mobile device.
Problem Statement
The primary outcome in our model is to assess whether an intersection is safe to cross or not. We also seek to identify the crossing path (zebra crossing) and give users instruction about the appropriate time needed to pass. We will utilize self-constructed datasets (real-world cross-road images in the view of pedestrians) and a fine-tuned CNN-based model. We expected to see our model successfully identifying traffic lights on images, recognizing traffic signals (i.e. the probability of the lights in red, yellow, green respectively), predicting whether a designated crossroad is safe to walk (0: stop, 1: safe), and calculating the appropriate time needed to pass by assessing road width (different categories: 2 lines, 4 lines, 6 lines, etc.).
Approach
Database and Data Preprocessing
To evaluate whether an intersection is safe to walk for pedestrians, we collected street images and evaluated them according to the following parameters: (1) traffic light signals, (2) zebra crossing, (3) obstacles in walking path (4) weathers, and (5) crossroad width (estimated by Google Map satellite image). We then labeled images as 0: stop/not able to pass or 1: safe to pass. If a complicated traffic condition was presented in the image or there was difficulty assessing whether an intersection is safe to pass, we use the method of majority decision to determine the label.
We utilize pandas, PIL, and matplotlib packages in data preprocessing. To regularize our images, we converted all our images into jpg format and rename them. Also, we crop our images to the size of 900x900. To enhance training efficacy, we pick out images labeled 0 and 1 in the ratio of 1:1. Then, we split our database randomly into training data/validation data/testing data in a 90:10:10 ratio.
CNN-based Model
To constructed a model with efficiency and accuracy, we selected four pre-trained models (VGG, Inceptionv3 (GoogLeNet), ResNet, EfficientNet), fine-tuned, and trained our data those models respectively to see which model show the most promising results. Hypothetically, our model could execute the following tasks:
Preliminary Results
Database and Data Preprocessing
Totally, 1148 images were collected and carefully labeled. To make the ratio of 0 (stop) : 1 (pass) in 1:1, 1070 photos (stop: 535 images; pass: 535 images) were selected. After splitting our data randomly into training data/validation data/testing data in a 90:10:10 ratio, we have 956 images in the training set, 57 images in the validation set, and 57 images in the testing set. Below are figures and tables showing data distribution (counts vs. width of roads) and statistics.
Reference