Automating PCB Inspection with Arduino UNO Q Part 3: Data Acquisition, Model Training and Testing

Automating inspection of printed circuit boards with Arduino UNO Q and Edge Impulse.

In this series of articles, we take a look at how the unique capabilities of Arduino UNO Q (066-5593) may be used to seamlessly integrate AI and control of physical inputs and outputs, with user-friendly training of AI/ML models courtesy of the Edge Impulse software platform.

In Part 1, we outlined the use case and introduced the sophisticated prototyping features provided by UNO Q hardware and App Lab. In Part 2, we got up and running with Arduino App Lab and Edge Impulse, and prepared a selection of PCBAs. In this latest instalment, we will now proceed to data acquisition, model training and testing with Edge Impulse Studio.

Note that machine learning (ML) is regarded as a subset of artificial intelligence (AI) and this series of articles may use both terms interchangeably. 

The Task at Hand

It was decided that, for our example, we would count the number of SOT23 transistors in the frame and this simple check would be used to decide whether a board had passed or failed inspection. Of course, a more comprehensive system in a real-world deployment would be trained to recognise many more component types, but this will be sufficient for our purposes.

Model and Method

Training ML models from scratch can be an intensive process requiring very large datasets in order to achieve any sort of accuracy. In support of which there are standard datasets which may be used for both training and testing, with one example being ImageNet, which provides a database of more than 14 million hand-annotated images, spread across over 20,000 categories.

Fortunately, with much smaller datasets, we are able to utilise a technique called transfer learning, whereby we take a pre-trained model, use this as a starting point and then fine-tune it to our particular problem. Saving us not only the trouble of having to assemble an enormous dataset, but also requiring access to powerful compute resources on which to perform training.

The Edge Impulse platform leverages transfer learning for object detection and directly integrates support for a number of models, each optimised for slightly different tasks and/or deployment platforms, ranging from microcontrollers up to more powerful Linux-based systems. In addition to which it is also possible to upload your own custom models to the platform.

We settled upon FOMO (Faster Objects, More Objects) MobileNetV2 0.35, which is a novel model developed by Edge Impulse and uses the larger MobileNetV2 model as its backbone. Benefits include that it is particularly lightweight — using up to 30x less processing power and memory than MobileNet SSD — fast, and can count multiple objects and find their locations. For more details, including what the 0.1 and 0.35 suffixes indicate, see the FOMO documentation. 

Data Acquisition

In the previous article, we showed how software can be installed on the UNO Q to enable camera integration with Edge Impulse Studio, and we also showed how images can be captured directly using GStreamer, which may then be subsequently uploaded via the Edge Impulse CLI. With other methods available, including using a smartphone, the web interface and cloud storage.

Each data acquisition mechanism has its own benefits, such as for smaller datasets, the convenience of preview and capture from the web interface, and the Edge Impulse CLI being particularly useful when you might have a large number of files and perhaps machine-generated labelling and metadata associated with these. While being able to utilise Amazon or Google etc. cloud storage will be a major benefit to those working across teams and projects, particularly with very large datasets.

The object detection tutorial suggested capturing around 50x images for training and testing, so we captured 55x in total, which can be seen tiled in the image above. With different zoom and focus levels, along with rotated and cropped boards. The images were split across training and testing directories, with the recommended 80/20 balance.

{
    "version": 1,
    "files": [
        {
            "path": "training/PCBA01-A.jpg",
            "name": "PCBA01-A.jpg",
            "category": "training",
            "label": {
                "type": "label",
                "label": "SOT23"
            }
        },
        {
            "path": "training/PCBA01-B.jpg",
            "name": "PCBA01-B.jpg",
            "category": "training",
            "label": {
                "type": "label",
                "label": "SOT23"
            }
        },
        {
            "path": "training/PCBA01-C.jpg",
            "name": "PCBA01-C.jpg",
            "category": "training",
            "label": {
                "type": "label",
                "label": "SOT23"
            }
        },

We decided to try out using the Edge Impulse CLI to upload the images, and created an info.labels file for this purpose, a fragment of which can be seen above. Though we later realised that the “SOT23” label we had applied to each was redundant, when we subsequently went on to label the actual position of each of these parts via the Studio.

The images were then uploaded with:

$ edge-impulse-uploader --directory pcba-data --info-file pcba-data/info.labels

Next we went to the Studio and could see the images in Data acquisition.

Following which we could go to the Labelling queue and proceed to label the SOT23 parts visible in each captured image.

This raised the question of whether to label devices which were partially cropped, and note that the grey margins indicate image areas which will not be used due to the model input dimensions. Looking into this, it seemed that having some cropped objects could help make the model more robust, whereas if they are 50% or more cropped/occluded, this might introduce noise.

Creating an Impulse

An “impulse” is a pipeline that combines feature extraction and a machine learning model, which takes input data — such as our images — processing this and then using it to train the model.

We selected to create a new impulse from the side menu, with input images of 320x320 pixels, which are resized to fit the shortest axis. The image block (white) has the default options set. In the object detection block we can see that we only have one output feature, which is SOT23 as labelled.

Upon saving the new impulse and navigating to the Image menu item, we could then select RGB or grayscale for the colour depth. Following which we selected Generate Features. After a brief delay, this completed and we could explore features via the plot on the right.

Note that this plot would be much more interesting if we had two or more output features, since there would be clusters for each and we’d be hoping for these to be clearly defined/separated.

Training Attempt #1

We then proceeded to select Object detection, and from here we selected the mode, FOMO MobileNetV2 0.35, and left the training settings at their default.

Save & train was selected, and following which we had to wait a short while for the training process to complete. Unfortunately, our F1 score — a key evaluation metric — was only 68.3% and from the Confusion matrix, we could also see that the results were not great.

A further surprise came when we next proceeded to Model testing, and this resulted in a meagre 27.27% accuracy!

Which dial to turn?

We must admit to having been a little baffled and not quite sure where we went wrong. With numerous variables, could the poor performance be attributed to:

• Choice of model:
  • Perhaps FOMO MobileNetV2 0.35 wasn’t a good choice for this problem?
• Training settings:
  • It could be that we needed more training cycles?
  • Or perhaps a different learning rate?
  • Or to set some advanced options?
• The data:
  • Perhaps we should we should have labelled more or fewer cropped parts?
  • Had more or no out of focus images?
  • Maybe more or fewer rotated images?
  • Or perhaps it would help if we labelled other devices present in each frame?

Attempt #2

A second attempt was made, with a fresh start using a new set of images which were further zoomed out and this time labelling SOIC16 parts as well as SOT23.

Unfortunately, the results were even worse and at this point, we decided to take advice from the experts at Edge Impulse.

Analysis

In short, it would appear that our mistake was to mix focus and zoom levels with such a small dataset, and also where in deployment we would have consistent focus and zoom. Furthermore, our sample was too small for objects which are markedly different from those which the model was originally trained on. Remember, we are using transfer learning and it’s likely that the original dataset did not include any surface mount electronic components.

So rather than 50x or so images, it was recommended that we have at least 150-200x.

Success!

We started attempt #3 by creating a new Edge Impulse project and importing the images used in our first attempt, following which removing those which were out of focus or zoomed out. This meant that this work was not completely lost, as we had a small set of labelled images to start with. Following which we gathered many more images of the selection of boards with parts removed, rotating boards and adjusting the lighting level between shots.

149x images for attempt #3.

We repeated the steps to label images and, following which, generate features. One change which we made to the training settings was to enable data augmentation, which applies random transformations to images, in an effort to reduce overfitting and increase accuracy.

This resulted in a much improved F1 score of 97.8%.

With an accuracy in testing of a far more encouraging 96.67%!

Which just goes to show that, if quality of data input is a potential candidate for poor performance, it’s probably worth investigating this first.

It was interesting also to explore the test results and see where our model had failed to identify parts, as seen above. To our eye, the missed parts did not appear so different to those adjacent, which had been correctly identified. In any case, one additional piece of advice received was to label more cropped parts in the training data, which we previously hadn’t labelled. It also may be worth gathering an additional 50 or so images to see if this improves the accuracy.

Another neat testing feature available at our disposal is Live classification, which is available from the main menu and with edge-impulse-linux running on the UNO Q, this allows us to sample data and classify images live and direct via the browser. Which is useful for providing further validation and enabling us to quickly try out new board samples, crops and rotations etc.

Tuning Assistance

Finally, we should note that we haven’t as yet explored the EON Tuner, which helps you find the best performing impulse for your application, by automating testing with different models, plus parameters such as image dimensions and colour depth, and model learning rate and training cycles. We’ll save this topic for another day!

Next Steps

In Part 4 we’ll bring everything together and explore how we can use Arduino App Lab to deploy our custom model and integrate physical inputs and outputs for added convenience.

  — Andrew Back