Easing access to high-throughput GPUs for Deep Learning researchers
Project Overview
There is a strong need in the university to run many Deep Learning training runs. This would be very costly to do either on the Cloud or by buying dedicated resources. However, many computers around the university have high-quality GPU cards, which we could make use of when their normal users aren’t using the computers. For example, during the night. I worked on developing a prototype HTCondor cluster within Newcastle University, along with creating tooling to make it easier to use the system for non-computing experts, to begin to solve this problem.
What were the key results of your research project?
- I successfully created a small HTCondor Cluster on campus, using unused university computers. These computers could communicate with each other, allowing me to submit jobs from the head node which then ran on the other nodes based on availability and resource requirements. This proved that creating a cluster in the university would be possible, as it would only require the addition of more machines in the same manner.
- I tested the performance of this cluster with and without Microsoft Defender. The final result was that Defender didn't have a significant impact on the performance, meaning that the University's security requirements could be satisfied.
- I built a user interface which could be used by non-computing experts and students to access the HTCondor cluster, allowing jobs to be submitted without using the command line.
GitHub repository: https://github.com/tomreece4/htcondor-interface (user interface only)
How do you feel you have benefited from completing this internship, and has it made you consider future career paths?
I've benefitted from this internship by learning a huge amount about different components of computer science and machine learning.
I've gained a lot of experience with HTCondor (which I had never used before), I've learnt about different security measures and their importance, and I’ve developed skills working physically on a large number of machines and installing operating systems from scratch, which I wouldn't have ever been able to do in my own time.