In this article I discuss Aim — an open source AI experiment comparison tool that can deal with 100s of training runs easily.

Disclaimer: I am co-creator of Aim.

Image for post
Image for post
I see metrics, metrics everywhere! (image by author)

AI researchers take on increasingly ambitious problems. As a result demand for AI compute and data multiplies month over month.

With more compute resources and more data available AI engineers run not only longer but also a lot more experiments than they used to.

Usually, AI research starts with setting up the data pipeline (or several versions of them) followed by an initial group of experiments to test the pipeline, architecture and detect basic bugs. Once that’s established, the rounds of experiments begin!

Then folks play with different moving pieces (datasets, pipeline, hyperparameters, architecture etc) either by hand or using techniques like grid search for instance…

Can tools for AI be as good as they are for the rest of software engineering?

AI is eating software.

Back in 2015, AI grabbed the attention of builders, investors and companies. Deep learning is now mainstream because it’s a super efficient way to solve many types of problems, where we used to hardcode rules and features.

But AI tools are broken.

For the rest of software, we engineers have always built awesome tools for ourselves, like IDEs, version control, packaging, containers and monitoring. Launching a production-strength website, lib or app has never been easier.

But for AI the development cycle is very long — a preprocessing bug may be discovered only after training on 1024 TPUs, or never. …

Most of the ideas cannot be effectively shared straight away. It takes time and brain processing before the first contact with other brains is made. Even the simplest one sentence ideas get filtered (subconsciously). It doesn’t get easier when we think about multi-dimensional and multi-layered concepts. How to transfer complex ideas effectively?

Our thoughts and ideas do seem complex, but there are hidden regularities waiting to be discovered if we pay enough attention. Therefore, the stream of consciousness is less continuous and more inherently modular so that each module/component could be processed, considered and communicated. This approach is used extensively by engineers to break down and build complex systems.

Every company is a data company but not many think of data as a culture. The data collection in modern companies (and modern world) is not institutionalised enough. In the episode “How to Love Criticism” of the “WorkLife” podcast one of the heroes to a question why he likes transparent criticism, answers:

“It’s just data. It’s just data, objective data about what I’m like. I would rather know how bad the bad is and how good the good is so I can do something with it”.

This is a great demonstration of a culture that systematically collects data — in different ways. Criticism is one type of data and there are many others generated but not collected or treated as data. This short story is about a culture that justifies and enables relentless data collection in companies. …

Image for post
Image for post
Image taken from

Startups are these mysterious economy-building, team-infused social structures that are designed to grow and become, you know, like Google, Facebook and handful of others. Obviously not all the startups end up being on top of the mountain as so many things can go wrong along the way.

2.5 years ago I decided to join a very exciting early stage startup. A decision that cost me a delay of my masters graduation for about three years and, nevertheless, gave endless opportunities of learning and exploration. By that time I had read enough material about the startups, their culture, chemistry and the lifestyle. Because of that so many myths had formed in my imagination that had to be verified. …


Gev Sogomonian

Aim co-creator. Co-founder and CEO AimHub. Prior Altocloud (acqu. Genesys). Runner, Swimmer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store