One of Arcitecta’s partners, Spectra Logic, invited Arcitecta to participate in the talk and created a discussion that included some of their customers such as Novartis Institutes for Biomedical Research and National Computational Infrastructure (ANU).
Arcitecta’s COO, Graham Beasley, made the journey from the USA to be the moderator for the panel. He described how the timing was perfect because not only had he planned on attending the event on Arcitecta’s behalf, but he also had the chance to overlap with Frank Radefeldt, Arcitecta’s SVP of Customer Success and Managing Director of Arcitecta’s new European office, and meet with customer Novartis who is based in Basel, Switzerland. Getting to moderate an exciting panel that Arcitecta CTO Jason Lohrey, Spectra Logic CTO Matt Starr, Novartis Data Engineering Architect Nick Whalen, and National Computational Infrastructure Deputy Director Allan Williams spoke on was the cherry on top.
The panel was known as a ‘Birds of a Feather’ talk, meaning it was a discussion for those with similar issues, which in this case is data management for HPC. Graham described the issue at large: “There is a data explosion going on and a lot of people have what is called machine-generated data. So, for example, there will be a weather instrument collecting weather data and the fidelity of that weather instrument is increasing at rapid rate. Or if it's a microscope looking at a cell like in Novartis, the resolution is increasing so rapidly that the resulting output files are overwhelming the storage infrastructure”. He continued that “the problem is that up until recently, people might be managing ones of petabytes. Yet now they're managing tens and even hundreds of petabytes”. This scale adds new challenges.
Participants discussed various solutions including how they moved data, how long they kept data, and what systems they used to manage data. Questions arose regarding multiple topics such as maintaining copies in case of lost or corrupted data and even how to keep track of large datasets that prove the effectiveness of AI models. Participants even stayed after the completion of the panel to continue one on one discussions about how metadata, the foundation of Mediaflux, could be used to solve some of their issues.
“Mediaflux was designed to give people that are managing data sophisticated tools to handle a lot more data with the same amount of hands,” said Graham. This could be a key to the data-explosion puzzle that attendees shared.
Going forward, Graham described numerous conferences and panels Arcitecta will be attending, including topics describing the capabilities of metadata, how Mediaflux can be used as a data orchestration platform, and even how university students can start careers in the HPC and data management space.
