Suggestions

The overall project implementation result was barely satisfactory, while the objective of this project is achieved. According to training trial set 5, the final product achieved around 80% accuracy of classification using 20 testing images.

Overall conclusion of the project development

The implementation was not good in terms of the efficiency of the data set and performance development. First of all, the requirement of each training trial set was too high in average. This happened due to the lack of research towards the choice of data set and performance of image recognition. In result, too much trial sets were done for trial and refine. Second the performance of the final product was behind the original requirement, due to the same reason from the first problem. Third, only half month left for enhancing the overall recognition accuracy, which was too little to enhance the final recognition accuracy in a large extend. Overall, the most to me improved part is the research. Which should include more implementation details of other existing examples and practical techniques rather than focusing too much on the concepts, because one of the major objective of this project includes implementation.

Suggestion made towards image classification

Collect images from the location same as the final product This can reduce the number of images required for classification, because the image taken from a specific location would have several features that might not be easy for human to realize. For example, it was realized that plastic bottle has different water depth line when taken at different angle or placed at different orientation, and this can affect the performance in some extend. Although it is practicable for human to realize this feature, but some other features may be harder to be captured by human, but might be very useful for machine learning model, for example, lightning, contrast, background features, etc. It will be a good practice for enhancing performance and reducing data collection variety

Understand the target item image It is more important than it sounds. In the earlier stage of the project, the product outcome was to classify “plastic”, “metal” and “paper”. However, during the collection of data set, it was realized that the 3 classes was very broad and has much variety in terms of type and appearance. It would waste a lot of time for data collection and expected outcome if the target item was not understood and specific enough.

Start with the simplest possible expected outcome This also echo with the approach of iterative and incremental software development. Even after understanding the images of the target item, there could still be gaps between the reality which may be hard to realize. To easily trace for the problem of the dataset, it is better to start very simple. So that it is easier to trace the problem, because possible affecting factor are less. With reference to the implementation of this project, time was wasted to take around >10000 photos of data set and trained for >10 days on the computer. However the result could not show anything for helping to refine the data set Which was a total waste of time.

Build the training model gradually Syncing with the above suggestion. When each training trial has very little difference from the last one, it is easier to look for the problem of error and to trace what image factor maybe contributing more towards the accuracy of recognition.

Use seen data set for testing This is a tip for the above suggestions. Sometime, the result using the ideal testing data set could not show us any information. However using seen or even the training data for testing could help to show partial results and give some information for planning the next trail set. Example is the success of training trial set 3 of this project.

Frequently manage and analyse the result and data logging down any information, including the training data set, testing data set, results, training time and configuration can help to keep track the progress and make analysis accordingly. In trail set 5, it was realized that it would be a good idea to analyze the ratio of online images and self taken images against the classification accuracy when trying to log the implementation details recently. Also, it may give suggestion to late trails.

Be aware the similarity between classifying groups Since the objective of training a classification model is to classify the testing data, it is easy to focus on the difference between types for the training, but we easily overlooked the similarity. Which gives problem to the training set. For example, one of the trial set only included paper box form lemon for data training, but a bottle form lemon tea for testing. Since small training model had lower accuracy thus have higher chance to take more irrelevant information for recognition, in result, the training model regarded the lemon tea bottle as paper.

Do not over categorize One major reason for failure of trial set 2B was that, there are too many groups. Some even hard for human to judge, i.e. top view and normal view. It will also be misleading for the training model. So, it is important to make sure the images are easily classifiable by human. One guide to follow is, categorize by the nature of the target item.