Building a 3D Print Queue Automation System - Part 2
Building the key back-end functionality for automating 3D print queue management and file transfer.
Please see the first part of this series for some of the background to this problem and how I intended to approach it.
Queue Database and Interface
My first step was to convert our Google Sheets-based database into something more scalable. We’d been struggling with slow loading times and delays in every step of the process, so I looked at a range of dedicated database systems.
I settled on MariaDB, a widely used free and open-source fork of MySQL, which I had some prior experience with. The setup process was relatively easy on Windows with the provided tools, and it was straightforward to build up a database structure that matched our existing spreadsheet, including extra features like an automatically incrementing ID column and updating timestamps.
After that, I could start work on building an interface for the database - because it’s the language I know best I chose Python and used the mysql-connector library to get the basic database connection working.
MariaDB works using a query language called SQL, where you can structure a request to ask the database to return specific data as you require. Below is an example of a request I built to retrieve the next queued print from the database:
WHERE `print status` = 'Queued'
AND `printer type` = 'Test'
ORDER BY `added` ASC
Line by line, this code does the following:
- First I specify what data to return, in this case just the values from the ‘id’ column.
- I then specify which table from the database to take the data from - ‘prints’ contains all the details for each print job.
- After that I set conditions - the value of ‘print status’ in the given row must be ‘Queued’
- And the value of ‘printer type’ must be ‘Test’. The second condition is important for differentiating between machine code that is set up to run on different 3D printers, as bad instructions are unlikely to work as intended and could easily cause damage to the machine.
- Then I ask for the results of that to be ordered by the value of the ‘added’ column, which is a timestamp for when the print was added to the queue.
- Finally, I specify that only one result, in this case, the earliest queued print, is to be returned.
Other SQL queries can be built in the same way, and the language includes a number of powerful commands for handling data in a wide range of different ways.
I constructed a number of different queries and Python functions to use them, covering a number of eventualities including retrieving assorted print details and updating the database when starting a print, marking a print as complete and marking a print as failed.
After this I started on a method of retrieving the actual gcode files - because of its convenience and accessibility I decided that sticking with our existing Google Drive solution would work well.
Our existing upload program uses a Google API service account, which I then adapted to use to download the same files. Upon upload, Google Drive gives each file a unique address that is stored in the database, which can be used to later download the file. This address doesn’t change even if the file is moved, so we’re safe reorganising later.
While quite poorly documented in parts, using the Google API in Python was quite straightforward with a service account. With the file ID retrieved from the database, it was very simple to request the file from Google’s servers and download it, ready for transfer to a printer.
One more feature from the Google Sheets based solution was emailing users when their print was complete or had failed. Unfortunately, we didn’t have the access permissions required to set up automatic emailing using the Gmail API, so I had to look for another option. After discussing with the iForge’s IT team, we decided the best way to approach this is to use another mail server, which we’re yet to set up - when this happens I’ll update this article!
In order to test all of this, I built a simple command-line based file downloader to emulate how the supervisor program would use this module. It first asked what type of printer you wanted to retrieve a print for, then found the next queued print and downloaded the file, finally updating the print queue to say that the print had started.
Arguably the most important part at the centre of the whole project, this has to manage movement between the print queue and the printers, monitoring their status and updating the database as necessary.
In order to ensure that the system worked no matter what the state of the printers are at supervisor start time, I had to make every part of it rely on the data in the database and the retrieved octoprint state and not any internal state. For basic cases like when a print was currently running or paused this was easy, but when a running print finished or the printer was not running at the start time this became more complex.
To make sure all the necessary information was available to the software I had to make the print ID available from OctoPrint. The easiest way of doing this was changing the filename to the unique print ID and using this when checking what is running and what is meant to be done next.
My queue interface module handled all the database interaction for me, given a config file with valid credentials in the same folder. As I developed the supervisor I created a number of helper functions to make updating the database easier.
For interaction with the OctoPrint instances, I used the octorest Python module, which uses the standard REST web API to retrieve and send data to the instance. This proved quick to set up and easy to use, with most of the functionality of the API exposed as simple Python functions. While a few features didn’t quite work as I expected, like the ‘start print’ flag when uploading a print, these were possible to work around using other commands.
After building a number of helper functions I worked on the main part of the supervisor program - the queue management and file transfer flow.
Here’s how it works:
- Looping through every printer loaded into the system, it checks their current state. The only useful state for us here is “Operational”, when the printer is idle and not printing, waiting to print or in any other condition.
- It then checks whether there’s a print file in the working folder - Within that file is a marker set by OctoPrint depending on whether the print finished or not - if the printer was reset or the print was cancelled this will be marked as failed, and if the file is finished it gets marked as successful.
- If the print was marked as successful my system communicates with the queue database manager to mark it as complete and record how long it took, if the print is marked as failed then it reports that to the queue. The system then deletes the file so it doesn’t try to repeat the above steps.
- In order to allow the reps to mark prints as complete or failed properly, I add an artificial pause at the very end of each file sent to the printers. If the print is resumed then it gets marked as successful, but if the rep cancels the print or resets the printer at this point it gets marked as failed.
- In the case of premature failure on a print, by default the printer won’t pause to allow the build plate to be cleared, so the system uploads and starts a print that contains nothing but a pause with a message saying “Print ID#XXX has failed”.
- After that, the printer is in the “Operational” state and has no print in the active queue folder so the next print, if available, can be retrieved from the database, uploaded to the printer and started.
This cycle continues as long as the supervisor is running, there are printers online and there are prints waiting in the queue.
In the next part of this series of articles, I will go through developing the user interface and how I set up the project for deployment. Keep an eye out for part 3!