**Christoph Schuhmann:** LAION stands for Large-Scale Artificial Intelligence Open Network. First and foremost, it’s simply a huge community of people who share the dream of open-source AI models, research, and datasets. That’s what connects us all. We have a [Discord server](https://discord.com/invite/xBPBXfcFHd) where anyone can come in and share a bit about the latest research in the field. You can also propose a new project and find people to work on it with you. And if you ask the mods, me, or other people, you might even get a channel for your project. That’s basically the core.

When we had such surprising success with our first dataset called [LAION-400M](https://laion.ai/blog/laion-400-open-dataset/), we set up a small non-profit association that doesn’t actually do anything. We have a bank account with a bit of money coming into it from a few companies that support us. That’s primarily Hugging Face, but also StabilityAI, although we’re mostly supported not by money but by cloud compute.

StabilityAI, for example, has a huge cluster with 4000 or now 5600 GPUs, and there we or our members who are approved by the core team can use preemptable GPUs, for example, what is not being used at the moment and is idle.

**devmio: So we can just come to you and contribute? Propose our ideas and ask for help with our projects or help with ongoing projects?**

**Christoph Schuhmann:** Exactly! You can now come to our Discord server and say that you want to contribute to a project or help us with PR or whatever. You are most welcome!

**devmio: Is LAION based in Germany? And you are the chairman and co-founder?**

**Christoph Schuhmann:** Exactly. I am a physics and computer science teacher, I have been regularly involved with machine learning, and I also have a background in reform-oriented education. I made a Kickstarter documentary seven or eight years ago about schools where you can learn without grades and curriculum. After that took off, I did tutorials on how to start such an independent school. So I knew how to set up a grassroots non-profit organization. I am not paid for my work at LAION.

## The Beginnings of LAION

**devmio: How did LAION come to life? How did you get to know the other members?**

**Christoph Schuhmann:** I actually started LAION after reading a lot about deep learning and machine learning and doing online courses in my spare time over the last five to six years. When the first version of DALL-E was published at the beginning of 2021, I was totally shocked by how good it was. At that time, however, many non-computer scientists didn’t find it that impressive.

I then asked on a few Discord servers about machine learning and what we would need to replicate something similar and make it open-source. There was a well-known open-source programmer at the time called Philip Wang (his alias on GitHub is lucidrains) who is a legend in the community because whenever a new paper comes out he has the associated codebase implemented within a few days. He also built an implementation of the first version of DALL-E in Pytorch called [DALLE-pytorch](https://github.com/lucidrains/DALLE-pytorch). This model was then trained by a few people using small data sets on Discord, and that was proof of concept.

But the data was missing, and I suggested going to [Common Crawl](https://commoncrawl.org/), a non-profit from Seattle that scraps HTML code from the internet every two to three months and makes it available. A snapshot, so to speak, of the HTML code of all possible websites, which is 250 terabytes zip file. I then suggested downloading a gigabyte as a test and wrote a script that extracts image tags together with alt tags and then uses the CLIP model to see how well they fit together.

Then two “machine learning nerds”, who were much better at it than I was at the time, implemented it efficiently but didn’t finish it. That was a shame, but they were developing the GPT open-source variant [GPT-J](https://huggingface.co/docs/transformers/model_doc/gptj) and therefore didn’t have the time.

Then in the spring of 2021, I sat down and just wrote down a huge spaghetti code in a Google Colab and then asked around on Discord who wanted to help me with it. Someone got in touch, who later turned out to be only 15 at the time. And he wrote a tracker, basically a server that manages lots of colabs, each of which gets a small job, extracts a gigabyte, and then uploads the results. At that time, the first version was still using Google Drive.

## The Road to the LAION-400M Dataset

It was a complete disaster because Google Drive wasn’t suitable for it, but it was the easiest thing we could do quickly. Then I looked for some people on a Discord server, made some more accounts, and then we ended up with 50 Google Colabs working all the time.

But it worked, and then, within a few weeks, we had filtered 3 million image-text pairs, which at the time was more than Google’s [Conceptual Captions](https://ai.google.com/research/ConceptualCaptions/), a very well-known dataset of 2019. That little success got us so much attention on the Discord server that people just started supporting us and writing things like, “I have 50 little virtual machines here from my work, you could use them, I don’t need them right now,” or “I have another 3090 lying around here with me, I can share it with you.”

After three months, we had 413 million filtered image-text pairs. That was our LAION-400M dataset. At the time, it was by far the largest image-text dataset freely available, over 30 times larger than [Google’s Conceptual Caption 12M](https://github.com/google-research-datasets/conceptual-12m), with about 12 million pairs.

We then did a [blog post about our dataset](https://laion.ai/blog/laion-400-open-dataset/), and after less than an hour, I already had an email from the Hugging Face people wanting to support us. I had then posted on the Discord server that if we had $5,000, we could probably create a billion image-text pairs. Shortly after, someone already agreed to pay that: “If it’s so little, I’ll pay it.” At some point, it turned out that the person had his own startup in text-to-image generation, and later he became the chief engineer of Midjourney.

As you can see, it was simply a huge community, just 100 people who only knew each other from chat groups with aliases. At some point, I made the suggestion to create an association, with a banking account, etc. That’s how LAION was founded.

## Even Bigger: LAION-5B and LAION-Aesthetics

We then also got some financial support from Hugging Face and started working on LAION-5B, which is a dataset containing five billion image-text pairs. By the end of 2021, we were done with just under 70 percent of it, and then we were approached by someone who wanted to create a start-up that was like OpenAI but really open-source. He offered to support us with GPUs from AWS. This was someone who introduced himself as a former investment banker or hedge fund manager, which I didn’t quite believe at first. In the end, it was just some guy from Discord. But then the access data for the first pods came, and it turned out that the guy was Emad Mostaque, the founder of StabilityAI.

**devmio: What is the relationship between LAION and Stability AI?**

**Christoph Schuhmann:** Contrary to what some AI-art critics claim, we are not a satellite organisation of Stability AI. On the contrary, Stability AI came to us after the LAION-5B dataset was almost finished and wanted to support us unconditionally. They then did the same with LAION-Aesthetics.

**devmio: Could you explain what LAION-Aesthetics is?**

**Christoph Schuhmann:** I trained a model that uses the CLIP embeddings of the LAION images to estimate how pretty the images are on a scale of one to ten. It’s a very small model, a multilayer perceptron running on a CPU. At some point, I ran the model over a couple of 100,000 images, sorted them, and thought that the ones with the high scores looked really good. The next step was to run it on 2.3 billion CLIP embeddings.

## From LAION-Aesthetics to Stable Diffusion

**devmio: How did LAION-Aesthetics help with the development of Stable Diffusion?**

**Christoph Schuhmann:** I had already heard about Robin Rombach, who was still a student in Heidelberg at the time and had helped develop latent diffusion models at the CompVis Group. Emad Mostaque, the founder of StabilityAI, told me in May 2022 that he would like to support Robin Rombach with compute time, and that’s how I got in touch with Robin.

I then sent him the LAION-Aesthetics dataset. The dataset can be thought of as a huge Excel spreadsheet containing links to images and the associated alt text. In addition, each image is given a score, such as whether something contains a watermark or smut. Robin and his team later trained the first prototype of Stable Diffusion on this. However, the model only got the name Stable Diffusion through Stability AI, to whom the model then migrated.

LAION also got access to the Stability AI cluster. But we were also lucky enough to be able to use JUWELS, one of the largest European supercomputers, because one of our founding members, Jenia Jitsev, is the lab director at the Jülich Supercomputer Center for Deep Learning. We then applied for compute time to train our own OpenCLIP models. And now we have the largest CLIP models available in open source.

## LAION’s OpenCLIP

**devmio: What exactly do CLIP models do? And what makes LAION’s OpenCLIP so special?**

**Christoph Schuhmann:** On the Stability AI cluster, a Ph.D. student from UC Washington has trained a model called CLIP-ViT-G. This model can tell you how well an image matches a text, and this model has managed to crack the 80 percent zero-shot mark. This means that we have now built a general-purpose AI model that is better than the best state-of-the-art models from five years ago that were built and trained specifically for this purpose.

These CLIP models are in turn used as text encoders, as “text building blocks” by Stable Diffusion and by many other models. CLIP models have an incredible number of applications. For example, they can be used for zero-shot image segmentation, zero-shot object detection with bounding boxes, zero-shot classification, or even for text-to-image generation.

We have trained and further developed these models. We now have a variant that not only trains these CLIP models but also generates captions through a text decoder. This model is called [CoCa](https://laion.ai/blog/coca/) and is quite close to the state of the art.

We have many such projects running at the same time, sometimes so many that I almost lose track of them. Currently, we cooperate with Mila, an institute of excellence from Montreal, and together we have access to the second largest supercomputer in the US, Summit. We have been given 6 million GPU hours there and are training all kinds of models.

**devmio: You have already talked a lot about Stable Diffusion, and Robin Rombach, the inventor, is a member of your team. Is Stable Diffusion managed by you, is that “your” model?**

**Christoph Schuhmann:** No, we don’t have anything to do with that for now. But we have made the development and training of Stable Diffusion easier with LAION-Aesthetics and LAION-5B.

## Open Source as a Superpower

**devmio: LAION is committed to making the latest developments in AI freely available. Why is open source so important in AI?**

**Christoph Schuhmann:** Let’s take the sentence: “AI should be open source so that it is available to the general public.” Now let’s take that sentence and replace “AI” with “superpowers”: “Superpowers should be open source and available to the public.” In this case, it becomes much more obvious what I’m actually getting at.

Imagine if there was such a thing as superpowers, and only OpenAI, Microsoft, Google, maybe the Chinese and American governments, and five other companies, have control over it and can decide what to do with it. Now, you could say that governments only ever want what’s best for their citizens. That’s debatable, of course, but let’s assume that’s the case. But does that also apply to Microsoft? Do they also have our best interests at heart, or does Microsoft simply want to sell its products?

If you have a very dark view of the world, you might say that there are a lot of bad people out there, and if everyone had superpowers now, there would certainly be 10, 20, or 30 percent of all people who would do really bad things. That’s why we have to control such things, for example through the state. But if you have a rather positive and optimistic view of the world, like me, for example, then you could say that most people are relatively nice. No angels, no do-gooders, but most people don’t want to actively do something bad, or destroy something, but simply live their lives. There are some people who are do-gooders and also people who have something bad in mind. But the latter are probably clearly in the minority.

If we assume that everyone has superpowers, then everyone would also have the opportunity to take action against destructive behaviour and limit its effects. In such a world, there would be a lot of positive things. Things like superpower art, superpower music, superpower computer games, and superpower productivity of companies that simply produce goods for the public. If you now ask yourself what kind of world you would like to live in and assume that you have a rather positive worldview, then you will probably decide that it would be good to make superpowers available to the general public as open source. And once you understand that, it’s very easy to understand that AI should also be open source.

AI is not the same as superpowers, of course, but in a world in which the internet plays an ever greater role, in which every child grows up with YouTube, in which AI is getting better and better, in which more and more autonomous systems are finding their way into our everyday lives, AI is incredibly important. Software and computerised things are sort of superpowers. And that’s going to get much more blatant, especially with ChatGPT. In three to four years, ChatGPT will be much better than it is today.

Now imagine if the whole world used technologies like ChatGPT and only OpenAI and Microsoft, Google and maybe two or three other big companies controlled those technologies. They can cut you off at any time, or tell you “Sorry, but I can’t do this task, it’s unethical in my opinion”, “I have to block you for an hour now”, or “Sorry, your request might be in competition with a Microsoft product, now I have to block you forever. Bye.”

**devmio: We had also spoken to other experts, for example, Pieter Buteneers and Christoph Henkelkmann, who had similar concerns. But the question remains whether everyone should really have unrestricted access to such technologies, right?**

**Christoph Schuhmann:** A lot of criticism, not directed at LAION but at Stable Diffusion, goes in this direction. There is criticism that there are open-source models like Stable Diffusion that can be used to create negative content, circumvent copyright and create fakes, etc. Of course, it’s wrong to violate copyright, and it’s also wrong to create negative content and fakes. But imagine if these technologies were only in the hands of Microsoft, Google, and a few more large research labs. They would develop really well in the background, and at some point, you would be able to generate everything perfectly with them. And then they leak out or there is a replica, and society is not prepared at all. Small and medium-sized university labs wouldn’t be prepared at all to look at the source code and discover the problems.

We have something similar with LAION-5B. There are also some questionable images in the dataset that we were unable to filter. As a result, there is also a disclaimer that it is a research dataset that should be thoroughly filtered and examined before being used in production. You have to handle this set carefully and responsibly. But this also means that you can find things in the set that you would like to remove from the internet.

For example, there is an organisation of artists, [Have I Been Trained](https://haveibeentrained.com/), that provides a tool that artists can use to determine if their artwork is included in LAION-5B. This organisation has simply taken our open-source code and used it for their own purposes to organise the disappointed artists.

And that’s a great thing because now all those artists who have images on the internet that they don’t want there can find them and have them removed. And not only artists! For example, if I have a picture of myself on the internet that I don’t want there, I can find out through LAION-5B where it is being used. We don’t have the images stored in LAION-5B, we just have a table with the links, it’s just an index. But through that, you can find out which URL is linked to the image and then contact the owners of the site and have the image removed. By doing this, LAION generates transparency and gives security researchers an early opportunity to work with these technologies and figure out how to make them more secure. And that’s important because this technology is coming one way or another.

In probably a lot less than five years, you’re going to be able to generate pretty much anything in terms of images that you can describe in words, photo-realistically, so that a human being with the naked eye can’t tell whether it’s a photo or not.

## AI in Law, Politics, and Society

**devmio: Because you also mentioned copyright: The legal situation in Germany regarding AI, copyright, and other issues is probably not entirely clear. Are there sufficient mechanisms? Do you think that the new EU regulations that are coming will be sufficient while not hindering creativity and research?**

**Christoph Schuhmann:** I am not a lawyer, but we have good lawyers advising us. There is a Data Mining Law, an EU-wide exception to copyright. It allows non-profit institutions, such as universities, but also associations like ours, whose focus is on research and who make their results publicly available, to download and analyse things that are openly available on the internet.

We are allowed to temporarily store the links, texts, whatever, and when we no longer need them for research, we have to delete them. This law explicitly allows data mining for research, and that is very good. I don’t think all the details of what’s going to happen in the future, especially with ChatGPT and other generative AIs for text and images, were anticipated in these laws. The people who made the law probably had more statistical analysis of the internet in mind and less training data for AIs.

I would like to see more clarity from legislators in the future. But I think that the current legal situation in Germany is very good, at least for non-profit organisations like LAION. I’m a bit worried that when the [EU AI Act](https://digital-strategy.ec.europa.eu/de/policies/european-approach-artificial-intelligence), which is being drafted, comes, something like general purpose AI, like ChatGPT, would be classified as high risk. If that were to be the case, it would mean that if you as an organisation operate or train a ChatGPT-like service, you would have to constantly account for everything meticulously and tick off a great many compliance rules, catalogues, and checklists.

Even if this is certainly well-intentioned, it would also extremely restrict research and development, especially of open source, associations, and of grassroots movements, so that only Big Tech Corporate would be able to comply with all the rules. Whether this will happen is unclear so far. I don’t want high-risk applications like facial recognition to go unregulated either. And I don’t want to be monitored all day.

But if any lawmakers are reading this: Politicians should keep in mind that it is very important to continue to enable open-source AI. It would be very good if we could continue to practice as we have been doing. Not only for LAION but for Europe. I am sure that quite a lot of companies and private people, maybe even state institutions can benefit from such models as CLIP or from the datasets that we are making.

And I believe that this can generate a lot of value for citizens and companies in the EU. So I would even go so far as to call for politicians and donors to maybe think about building something similar to a CERN for AI. With a billion euros, you could probably build a great open-source supercomputer that all companies and universities, in fact, anyone, could use to do AI research under two conditions: First, the whole thing has to be reviewed by some smart people, maybe experts and people from the open-source community. Second, all results, research papers, checkpoints of models, and datasets must be released under a fully open-source licence.

Because then a lot of companies that can’t afford a supercomputer at the moment could open source their research there and only keep the fine-tuning or anything that is really sensitive to the business model on the companies’ own computers. But all the other stuff happens openly. That would be great for a lot of companies, that would be great for a lot of medium and small universities, and that would also be great for groups like LAION.

_**Editor’s note**: After the interview, LAION started a petition for a CERN-like project. Read more on [LAION’s blog](https://laion.ai/blog/petition/)._

## AI for a Better World

**Christoph Schuhmann:** Another application for AI would be a project close to my heart: Imagine there is an open-source ChatGPT. You would then take, say, 100 teachers and have them answer questions from students about all sorts of subjects. For these questions, you could make really nice step-by-step explanations that really make sense. And then, you would collect data from the 100 teachers for the school material up to the tenth grade. That’s at least similar everywhere in the Western world, except, of course, history, politics, etc. But suppose you were to simply break down the subject matter from 100 countries, from 100 teachers, from the largest Western countries, and use that to fine-tune a ChatGPT model.

You need a model that has maybe 20 to 30 billion parameters, and you could use it to give access to first-class education to billions of children in the Third World who don’t have schools but have an old mobile phone and internet access. You don’t need high-tech future technology, you can do that with today’s technology. And these are big problems of the world that could be addressed with it.

Or another application: My mum is 83 years old, she can’t handle a computer and is often lonely. Imagine if she had a Siri that she could have a sensible conversation with. Not as a substitute for human relationships, but as a supplement. How many lonely old people do you think would be happier if they could just ask what’s going on in the world. Or “Remember when I told you that story, Siri? Back in my second marriage 30 years ago?” That would make a lot of people happier. And I think things like that can have a lot of effect with relatively little financial outlay.

**devmio: And what do you see next in AI development?**

**Christoph Schuhmann:** What I just talked about could happen in the next five years. Everything that happens after that,  I can’t really predict. It’s going to be insane.

**devmio: Thank you very much for taking the time to talk to us!**

The post AI as a Superpower: LAION and the Role of Open Source in Artificial Intelligence appeared first on ML Conference.

The application of AI in patent offices

This template is seen in the application of AI to patent office examinations, patent classifications, reporting, and other critical workflows. Within examinations, it is used to greatly accelerate and increase the accuracy of a necessary step in the patent approval process – prior art searches.

Prior art is any published evidence that an invention is already known, which can take numerous forms, from just a description of an idea or formula to a centuries-old piece of technology or an existing product. When a new patent application is filed, patent office searchers and examiners spend much of their time performing searches of documents and other assets around the work and evaluating the results to determine if the target application encroaches on existing prior art.

The results of these searches determine whether an invention meets the patent protection criteria for novelty and obviousness. The former is the notion that an invention must be new or novel and therefore not known in the public domain prior to the application filing date, while the latter is the notion that an invention must be non-obvious and not a logical extension of a pre-existing invention that any skilled member of that field could feasibly surmise. Across the millions of patents, a single instance of prior art can be used to reject a patent or to send it back to the applicant for revision.

The process of searching for prior art is complicated, iterative, and time-consuming. For each search, examiners must devise a search strategy, select which databases to search, create the search parameters, perform the search, evaluate the results, and then if needed, modify and rerun the search.

According to an analysis of search activity conducted by the European Patent Office, a comprehensive patent application search draws on around 1.3 billion technical records in 179 databases, leading to about 600 million documents appearing in search results monthly. Another study by the Japan Patent Office estimated that its staff spent around 40% of their time conducting and reviewing prior art searches through traditional and rather labor-intensive tools.

The rapid growth in patent applications and the complexity of inventions coupled with the staggering volume of materials to search means that patent offices are always considering new ways to accelerate the application process to avoid long pendencies and in a few cases, backlogs. Indeed, according to WIPO (World Intellectual Property Office) in 2019, there were 5.7 million patent applications pending worldwide. To keep up with this flood of applications, patent offices hire more examiners and adopt technologies to improve productivity.

The integration of AI solutions in Brazil’s patent office

In 2020, one such office experiencing a sizeable patent backlog was INPI Brazil. With around 150,000 applications pending and an average wait time of more than 10 years, their backlog was significantly impacting innovation in Latin America’s largest economy and thereby limiting investments.

A sizeable chunk of their backlog, around 15%, consisted of chemistry patents. Chemistry patents require searches of both text and chemical structures within patent and non-patent publications and include full text and structure queries, which make finding similarities and relevance between the application patent and existing art a far more demanding review process than other patent applications.

INPI partnered with CAS, who offered an AI solution that could analyze the complexities of chemistry prior art to solve this problem, streamline their workflow processes, and tackle their backlog. In collaboration with INPI, a unique AI approach was created, which accelerated the laborious task of discovering prior art by focusing the solution’s search algorithms on multiple facets of patents to determine similarity between the target patent application and existing patent and non-patent publications, and refine results. An additional algorithm then created a relevant ranked data set for examiners to review. The results of this solution were impressive, with up to a 50% reduction in examination times, reduced search times for over 75% of applications processed, and contributing to an overall reduction of 80% in the office’s patent backlog. However, CAS arrived at the tailored solution with constant refinement and consideration of three factors.

Three considerations when implementing AI:

1. Quality data and human-curated data sets

While AI solutions can speed up prior art searches exponentially, AI alone is not a silver bullet and cannot replace patent examiners. However, AI can become a powerful tool patent examiners can use to enhance performance of their workflows. The secret lies in possessing curated and highly structured content that can train an algorithm correctly and then utilizing experts to maximize its application. In this regard, CAS see AI as just the latest technology that they layer upon their continuously updated data sets to improve search and retrieval of information and supplement this data and technology with extensive subject matter expertise and services.

Two waves in publishing have made the careful curation of content even more necessary, namely digitization and globalization. Digitization is the process of converting physical materials, such as books, illustrations, objects, and analog recordings and photos, into digital form. While globalization is the translation of these sorts of materials into other languages, as patents are territorial and must be filed in each country where protection is sought. These waves pose significant roadblocks to optimizing AI-powered prior art searches. Digitization often leads to transcription errors, mislabelled units, and overly complex patent language, while globalization leads to patents in dozens of languages. Each of these make human curation a necessity for quality data that can be easily searched and retrieved.

Thankfully CAS has a vast catalog of expertly human-curated data. In fact, CAS has been crowdsourcing data for over a century, by gathering abstracts from public and private domains since 1907. This vast catalog has been normalized, prepared, and connected in a structured format which improves the training of AI algorithms and increases the performance of prior art searches. By augmenting AI technology with human expertise for INPI, CAS scientists fed clean and structured data to the AI solution improving the predictive accuracy.

2. Domain expertise

Another consideration is to leverage the know-how of domain experts to refine the AI solution throughout a project. The INPI project required CAS to provide a wide array of expertise from distributed algorithms and machine learning to data science, cheminformatics, patent searching, and high-performance computing.

The CAS IP search team was therefore able to support the examiners’ searches by validating algorithm results during development and performing highly complex searches to augment the office’s capabilities. With individual prior art searches often variable in scope, different search professionals are likely to design different strategies for a given search. Having a team of search experts available to analyze algorithm results enabled them to yield insights into how those algorithms can be fine-tuned to improve relevancy.

3. Choosing the right algorithms

As has been established, completing a comprehensive prior art search is a painstaking process that requires the consideration of multiple facets of possible similarity. Therefore, choosing only one algorithm focusing on a single type of analysis, such as semantics, will prove insufficient to the task. For the INPI project, CAS chose to integrate four types of algorithms for text-based and substance-based analysis, including deep learning and term frequency-inverse document frequency. Using multiple algorithms allowed the AI to find semantic, syntactic, and substance similarities all in one multifaceted solution.

Traditional knowledge graphs were also added to analyze the connectedness between the vast amounts of data. The INPI Brazil project deployed one for chemistry and one for non-chemistry to determine ontological similarity and connectedness between documents using keywords, scientific topics, roles, and nomenclature.

The first-level algorithms evaluated semantics, such as title, abstract, and claims between patent and non-patent publications, and used a syntactic-driven algorithm that compared the prevalence of special terms in the target document to their uniqueness across all other documents to return an accurate set of similarity results.

Then, at the second level, an algorithm for a patented ensemble learning process combined the results to produce an optimal predictive model, which was then used to generate relevance-ranked results based on search context and each algorithm’s strengths and limitations. The ensemble learning algorithm then analyzed the ranked results arriving at a single prioritized list of patent and non-patent publications that were most likely to conflict with the target patent for the examiners to review.

Worldwide applicability of tailored AI

When implemented correctly, as in the INPI project, AI can transform patent office workflows and remove tedious tasks to free up researchers’ and examiners’ time for value-add work. There is no one size fits all solutions for these complex workflows and undertakings. The key is having close collaboration between the office and solutions experts to ensure the approach is perfectly aligned with the office’s strategic objectives.

Global patent offices face fundamental challenges that put their operational sustainability at risk. By combining AI, human-curated data, and workflow transformation, CAS has established an extremely effective approach for improving patent office timeliness, patent quality, and efficiency to help accelerate innovation around the world.

The post Three Key Considerations When Implementing AI appeared first on ML Conference.

Fortunately, it’s possible to accelerate and improve the processes by involving AI in vaccine development and rollout.

This article describes use cases and tools that AI healthcare companies [2] and research teams built to facilitate vaccine design, speed up trials, predict mutations, prioritize patients, and address vaccine hesitancy.

How AI contributes to vaccine development

Artificial intelligence can analyze massive datasets representing virus structure to pinpoint viable vaccine targets, predict virus mutation, assist in clinical trials, and help researchers organize and access a large volume of scientific publications.

AI identifies vaccine targets

Vaccine development is a data-intensive process as one needs to understand the virus itself and how the immune system will react to it. Machine learning algorithms [3] can analyze large datasets to identify which targets (or epitopes) of a virus are most likely to provoke an immune response. After obtaining a list of targets, scientists design matching vaccines.

While determining vaccine targets, one needs to be very careful to not enlist any entities similar to the host proteins that inhibit human bodies to avoid cross-reactions and undesirable side effects.

Protein-based AI vaccines

Machine learning algorithms can identify antigens from protein sequences and determine the most viable vaccine target. There are several research initiatives [4] that use AI models to fight COVID-19. One method employs AI to develop a vaccine that would contain both T-cell and B-cell epitopes (the part of an antigen that the immune system can recognize). This study discovered 17 potential vaccine peptides working with both immune cells.

In another example, a team of researchers from Baylor College of Medicine and Amity University in India built an AI-driven platform that facilitates vaccine target discovery [5]. Researchers used this software to develop a vaccine against Chagas disease. They identified eight main target proteins and top epitopes for each target, producing a multi-epitope vaccine. Since the emergence of the Delta coronavirus variant, the scientists have been collaborating with several pharmaceutical companies to design a new vaccine.

DNA and RNA-based AI vaccines

Such vaccines are supposed to mimic a partial genetic sequence of a virus. They encapsulate a part of the virus’s genetic code representing the targeted epitope in the form of RNA or DNA. When this code enters a human cell, it produces the epitope in question triggering an immune response. Given the ability of viruses to mutate, the vaccine needs to be based on a relatively stable genetic component to have a long-lasting effect. That’s where artificial intelligence becomes useful. AI algorithms can analyze enormous datasets containing genetically sequenced viruses to identify the more stable parts.

AI facilitates preclinical testing and clinical trials

Preclinical testing

The goal of this phase is to evaluate a vaccine’s safety and efficacy prior to testing it on people in clinical trials. Preclinical testing is typically conducted on a suitable animal model, but since the past decade, regulatory agencies have been calling for the use of alternative methods when possible. AI can be one of these methods as ML algorithms can predict compound toxicity.

Even if AI can’t fully replace preclinical testing, the technology can facilitate it by helping to set the proper dosage, anticipating some immune responses, and even selecting the best-suited animal model.

Clinical trials

Again, AI can’t make clinical trials virtual, but it can largely facilitate them. First, artificial intelligence can analyze the data obtained from preclinical testing and anticipate human immunogenic reactions.

Second, AI algorithms can help researchers find the best location for clinical trials. For example, the MIT School of Engineering built an ML-powered COVID-19 epidemiological model [6] that generates real-time insights about the pandemic and captures people’s behavior and health status (exposed, recovered, etc.). It can also predict how different governments would react to the challenge and their policy choices. All of this enabled the model to predict when and where COVID will spike, pinpointing ideal locations for clinical trials.

This AI tool could make intelligent predictions on 120 countries in addition to all 50 US states.

Third, AI accelerates vaccine rollout as it helps select the right people for trials through electronic health records mining. Provided that 86% of clinical trials can’t recruit [7] candidates within the expected time frame, any help is welcome.

AI outsmarts virus mutation

There is a general concern about viruses being able to change and adapt to medication. In light of the pandemic, SARS-CoV-2 is mutating and people are afraid that the publicly available vaccines will not provide long-lasting protection.

The scientific community is trying to stay ahead as researchers at USC Viterbi experiment with AI to predict and counter mutations [8]. This team has already used an AI-powered tool to determine potential vaccine targets using one B-cell and one T-cell epitope. Employing a wider dataset for AI-driven vaccine design will enable them to fight mutations more effectively. Especially that scientists claim their model can make accurate predictions using a set of 700,000 proteins.

Paul Bogdan, Associate Professor of Electrical and Computer Engineering at USC Viterbi, pointed out that their AI-based method could also vastly accelerate vaccine design, “This AI framework, applied to the specifics of this virus, can provide vaccine candidates within seconds and move them to clinical trials quickly to achieve preventive medical therapies without compromising safety.”

AI organizes data and makes it available for researchers

Researchers keep adding new reports to the already enormous stack of literature on the novel coronavirus and other viruses for that matter. It is becoming increasingly challenging to sift through all these publications. And again, scientists turn to AI to extract valuable insights from these papers.

For example, the Allen Institute built a resource called CORD-19, [9] which offers scientific articles on COVID-19 in a machine-readable format. Other researchers can develop AI algorithms to access this platform and answer queries.

How AI supports vaccine rollout

AI technology’s potential spans beyond vaccine development to its distribution, tracking, administration, and offering counseling. Artificial intelligence prioritizes people for vaccination

Many hospitals prioritize patients solely based on their age and rush to vaccinate everyone in the 65+ age category without further discrimination. For more awareness in vaccine distribution, AI-powered algorithms can help medical facilities identify the most fragile population segments.

Sanford Health, a Dakota-based healthcare organization, deployed AI to identify people at risk of having poor outcomes from COVID-19 [10]. They ran an algorithm on their patients of the age of 65 and older to produce a prioritized list based on various health-related factors, such as obesity, kidney disease, heart disease, and diabetes among others.

Artificial intelligence monitors vaccine distribution and tracking

Using artificial intelligence in vaccine distribution, handling, and storage can have many benefits. Cheryl Rodenfels, Healthcare Strategist at Nutanix, mentions some of them: [11] “Relying on the technology [AI] to manage distribution data eliminates human error and ensures that healthcare organizations are accurately tracking the vast amounts of data associated with the vaccine rollout.”

However, deploying AI at this level is difficult, as every manufacturer has its own procedures for vaccine storage and handling. There are no unified standards on, for example, how many vaccines a medical facility must store.

Eases vaccine hesitancy

The spread of misinformation and vaccine hesitancy presents another problem that AI can help address. AI-powered chatbots that combine knowledge of psychology, public health, and infectious diseases can offer counseling and answer some of the sensitive questions. A recent study conducted in France [12] shows that bots can make people feel more positive towards vaccines.

Johns Hopkins Bloomberg School of Public Health teamed with IBM to develop a chatbot named Vira (Vaccine Information Resource Assistant). [12] They trained the bot through conversations with healthcare workers. Now Vira is used by regular people, and it continues to improve and learn.

Obstacles on the way to AI deployment

No doubt that AI can analyze large volumes of data much faster than humans. According to Dr. Kamal Rawal, Associate Professor at Amity University, who participated in building an AI-driven platform [13] for vaccine development, “The key innovation is using artificial intelligence to combine several hundred parameters to mine several thousand proteins and genes to reach to the right targets and design vaccine using these proteins.”

One interesting characteristic of AI is that it doesn’t make assumptions about what is right and wrong, so it can test the options that researchers tend to discard based on biased beliefs. However, there are things to consider when deploying AI in vaccine development and administration:

• Black-box models [14] are powerful, but their results can’t be justified, and bias can sneak in unnoticed. It is advisable to use explainable AI to understand how algorithms arrive at their conclusions. However, this will compromise their predictive power, so there is a tradeoff to make.
• The performance of machine learning algorithms depends on the training dataset, and immunology models are being trained on significantly smaller datasets [15] than the ones available for other disciplines, such as voice recognition.
• AI ethics is still a complex topic to approach. Using AI in vaccine development might grant it access to patient records, and the issue of privacy comes in. Another ethical concern arises when using AI in vaccine prioritization. Research shows [16] that race and ethnicity contribute to higher hospitalization risks in the case of COVID, but is it ethical to use such data?

Salesforce launched a Vaccine Cloud tool which is expected to help healthcare organizations manage vaccine administration. The company faced the same ethical concern. Here is what a Salesforce spokesperson told Healthcare IT News: “Our Principles for the Ethical Use of COVID-19 Vaccine Technology Solutions explicitly state that AI should not be used to predict personal characteristics or beliefs that would affect a person’s or group’s prioritization for access to vaccines, and we work closely with our partners and teams on this guidance.”

On a final note

With its analytical power, AI still can’t foresee everything. As Oren Etzioni, CEO at the Allen Institute for Artificial Intelligence, said, [17] “The human body is so complex that our models cannot necessarily predict with reliability what this molecule or this vaccine will do for the body.” So, using AI can’t replace clinical trials and can’t make vaccine development entirely virtual and fully automated.

Still, artificial intelligence can analyze large volumes of data and detect patterns that escape the human eye. With all the applications mentioned above, AI can vastly accelerate vaccine development and control their rollouts.

Links & Literature

[1] https://www.frontiersin.org/articles/10.3389/fimmu.2020.517290/full

[2] https://itrexgroup.com/services/ai-for-healthcare/

[3] https://jaxenter.com/basic-introduction-machine-learning-145140.html

[4] https://www.frontiersin.org/articles/10.3389/frai.2020.00065/full#B117

[5] https://www.bcm.edu/news/researchers-develop-ai-platform-to-boost-vaccine-development

[6] https://news.mit.edu/2021/behind-covid-19-vaccine-development-0518

[7] https://bioprocessintl.com/manufacturing/information-technology/in-silico-vaccine-design-the-role-of-artificial-intelligence-and-digital-health-part-1/

[8] https://news.usc.edu/181226/artificial-intelligence-ai-coronavirus-vaccines-mutations-usc-research/

[9] https://www.semanticscholar.org/cord19

[10] https://www.mprnews.org/story/2021/02/10/one-minn-health-care-provider-using-ai-to-pair-patients-with-covid19-shots

[11] https://www.techrepublic.com/article/how-ai-is-being-used-for-covid-19-vaccine-creation-and-distribution/

[12] https://psyarxiv.com/eb2gt/

[13] https://www.gavi.org/vaccineswork/are-chatbots-better-humans-fighting-vaccine-hesitancy

[14] https://www.bcm.edu/news/researchers-develop-ai-platform-to-boost-vaccine-development

[15] https://jaxenter.com/data-ai-models-172220.html

[16] https://www.brookings.edu/techstream/can-artificial-intelligence-help-us-design-vaccines/

[17] https://www.healthcareitnews.com/news/ai-has-advantages-covid-19-vaccine-rollout-potential-dangers-too

[18] https://spectrum.ieee.org/what-ai-can-and-cant-do-in-the-race-for-a-coronavirus-vaccine

The post AI in Vaccine Development and Rollout appeared first on ML Conference.

Christoph Henklemann holds a degree in Computer Science from the University of Bonn. He currently works at DIVISIO, an AI company from Cologne, where he is CTO and co-founder. At DIVISIO, he combines practical knowledge from two decades of server and mobile development with proven AI and ML technology. In his free time he grows cacti, practices the piano, and plays video games. Watch what Christoph Henkelmann has to say about the unique conference experience at ML Conference and why it is a must-attend conference for ML enthusiasts. Stay ahead of the trends and attend the ML Conference across the globe and online.

The post Speaker Testimonial: Christoph Henkelmann appeared first on ML Conference.

So you don’t have to deal with the wording, we have the ultimate text template for your boss.

A template for the e-mail to your boss

The post 4 arguments to convince your boss appeared first on ML Conference.

Ward Van Laer: The black box problem is a perception created by, for most people, the unintelligible jumble of machine learning models. But the decision the models make are always based on the data we feed the model. Will we be able to design completely transparent models without having to compromise the complexity of the problems to be solved?  In my opinion, the real question is what kind of explainability do we really need to demystify the black box perception.

JAXenter: In your talk at the ML Conference you show how to develop transparent machine learning models. How does that work?

Ward Van Laer: I will demonstrate that explainability can be interpreted in multiple ways. Depending on the perspective from which we look at an AI system, explainable AI can mean different things.

We can look at explainability in a technical way, which means we are looking through the eyes of machine learning engineers, for example. In this case, transparent AI can help to spot dataset biases. More importantly, this technical explainability is not interesting or understandable for an end-user. From this perspective, UX will play a crucial role in demystifying AI applications.

JAXenter: Why do you think transparency in ML is important?

Ward Van Laer: I believe we need more than just technical transparency, or as it is referred to at the moment, “explainable AI”. We need to pinpoint the needed properties that lay at the ground of a trustworthy AI, instead of focussing on full transparency.

JAXenter: Can you give an example of how a good UX changes the acceptance of AI solutions?

Ward Van Laer: In one of our projects in the health care industry we visualize links between classification results and the dataset, which helps physicians understand why certain decisions are made.

To have more insight in the possibilities I can certainly encourage everyone to attend my talk at MLConference 2019

JAXenter: What is the core message of your session that every participant should take home?

Ward Van Laer: Creating a well-working machine learning model is only half of the work. Developing a thought-through User Experience is the key to successful AI.

Please complete the following sentences:

The fascinating thing about Machine Learning for me is…

… that it will be able to help us solve many complex problems (e.g. health care).

Without Machine Learning, humanity could never…

… improve itself.

The biggest current challenge in machine learning is…

… making sure that an AI system is successful in the eyes of user.

I advise everyone to get started with machine learning …

… to better understand what the real possibilities are.

Once the machines have taken power…

… hmm let’s hope we can explain how it happened!

JAXenter: Thank you very much!

The post How UX can demystify AI: “We need more than just technical transparency” appeared first on ML Conference.

Whenever we say, “That’s not my problem,” or, “My company won’t let me do that,” we are handing over our ethical responsibility to someone else – for better or for worse. Do innocent decisions evolve so that they promote racism or gender discrimination through inadvertent cognitive bias or unwitting apathy? Far too often they do.

We, as technologists, hold incredible power to shape the things to come. That’s why he shares his thoughts with you so you can use this power to truly build a better world for those who come after us!

The post The Ethics of AI – dealing with difficult choices in a non-binary world appeared first on ML Conference.

Experience Dr. Pieter Buteneers live at ML Conference 2019 in Munich with his workshop: Machine Learning 101 ++ using Python

The post Keynote http://commodity.ai appeared first on ML Conference.

Ready-made AI has a limited value

The first step to answering this question begins with the availability of data. Admittedly, there are AI services like Microsoft’s Text Analytics and Computer Vision or Google’s Cloud Vision API which are completely finished. For example, to recognize the language of a text with Text Analytics, you need neither training data nor an understanding of machine learning. If you can send a text to a web API, you’re ready to go. For some applications, this may be enough as an introduction (e.g. assigning a support case to a team member who speaks the right language). In most cases, however, this isn’t enough. AI and machine learning only have real added value if they are adapted to a specific application.

Customizable AI services

If there is no ready-made AI service, that doesn’t mean that you have to do everything from scratch with libraries like TensorFlow or Microsoft Cognitive Toolkit (CNTK). There‘s a middle way: Customizable AI and ML models that you can train with your own data. Here are two examples from Microsoft’s product portfolio:

Data is worth its weight in gold

These two examples show the fundamentally new approach of the “programming” of (semi)finished AI services, in comparison to the classical development of program libraries. Our role as developers is no longer to write an algorithm. We have to take care of the training data. This task is anything but trivial because the quality of the resulting deep learning model depends on the quality of the training data. If too little data is available or if the existing training data sets are incorrect, like incorrect keywords, of poor quality (for example poor image quality, photos are too similar, etc.) or not representative, meaning sample sets which no real user would ever use, then the result is useless. Additionally, just training data is not enough. More datasets are required so that the models can be tested.

Data is the new gold in the world of AI and ML. Prefabricated AI services in the cloud don’t change this – on the contrary. As a team that wants to enter this world, the first thing you have to ask yourself is how to get the data you need. It’s also this hurdle, which makes the market entry for start-ups so difficult. Established companies either have existing databases or can fall back on an existing community which can be motivated to test AI-based software components such as bots, give feedback and, thus, indirectly provide the necessary training data.

Iterative model development

The iterative model development is an important aspect in this sense. Customizable AI services such as the ones mentioned above contain ready-made components which can be used to check real operational data (for example, sentences that users have said about a bot or images that have been uploaded for tagging). It‘s easy to add the real data with correct metadata to the training set and thus improve the AI model step by step (Fig. 2), if classification errors are discovered.

In order for the iterative approach to work in practice, mechanisms must be in place to make the versioning, testing, and deployment of models simple and robust. Usually, AI services are available serverless. As a development team, you don’t have to worry in any way about the operation or scaling of the server. You can deploy the model with one click, differentiate between test and production environment, have a built-in version management, export the models to a source code management to archive it and much more (Fig. 3). The required time for administration and DevOps processes is reduced to a minimum by such functions.

APIs for meta programming

Another feature of AI services is important for SaaS providers: not only are all functions interactively available via a WebUI, but the exact same functions can also be automated via web APIs. When you develop your own multi-tenant SaaS solution, you often can’t lump all your end customers together. Every customer has slightly different requirements. The data models differ, workflows are customer-specific, master data is naturally different for each customer and much more. For example, if you want to offer each SaaS end customer an individual bot, the model must differ from customer to customer in order to achieve a high-quality result. The training data is different and in many cases, the models also differ structurally.

Through the APIs of AI services, it is possible for SaaS providers to practice meta-programming. This means that you write a program that is not used by the end customer, but that creates another program: in this case, an AI model using an AI service.

Challenges

It all sounds so very tempting, doesn’t it? AI and ML can be used in any project without any problems, even if there is no relevant prior knowledge and only a limited budget. This statement is basically correct, but there are some challenges to overcome in detail. The first one has already been briefly mentioned above: You need a lot of training good quality data. At the age of GDPR, this is not only a technical but also a legal hurdle. The second challenge is the risk of expecting more from the selected AI service than it can provide. As already mentioned, modern AI services offer the possibility to adapt the prefabricated models. But you can’t control all aspects. After all, it‘s precisely the strength of these services which reduces their complexity. Compared to classic SaaS and PaaS services of the cloud, however, evaluating AI services is a lot more difficult.

Rainer Stropek has been an entrepreneur in the IT industry for more than twenty years. During this time, he founded and managed several IT service companies and is currently developing the award-winning Software time cockpit in his company software architects together with his team. Rainer holds degrees from the Technical College of MIS, Leonding (AT) and the University of Derby (UK). He is the author of several books and articles in magazines around Microsoft .NET and C#. His technical focus is on C# and the .NET framework, XAML, the Microsoft-Azure platform, SQL Server and Web development. Rainer regularly appears as a speaker and trainer at renowned conferences in Europe and the USA. In 2010, Rainer was named by Microsoft as one of the first MVPs for the Windows-Azure platform. Rainer has also been a Microsoft Regional Director since 2015.

The post AI as a smart services for everyone appeared first on ML Conference.

Find out how to solve the cold start problem!

JAXenter: Databases need maintenance, we know that. But over the years impenetrable data thickets have grown in many companies. In your session you talk about unraveling the chaos, but where do you start?  

Markus Nutz: Fortunately, Freeyou hasn’t been around for that long, so we’ve been able to keep track of everything so far. The answer is probably pretty boring: documentation. Documentation includes all involved parties, which means that the requirements of the product owners, data scientists and data architects all have equal status.  We are aware that the data is our basis for differentiating ourselves from other insurers.

Thomas Pawlitzki: I have nothing more to add to this. Our own database is still controllable. The development team talks a lot about features and changes so that the individual team members are aware of database changes. You don’t have to explain anything to data gurus like Markus.

In the last few weeks, I have also looked at various frameworks that we can use in the development of our API. Some of them already offer features for data migration. For example, there you can store schema changes in relational databases as code and apply them, but also perform a rollback. Perhaps we will soon use such solutions to test the whole thing in its early stages.

JAXenter: How can we solve the “cold start problem”?

Markus Nutz: In general, keep your eyes open to see where and what kind of data is available. Statistics about traffic accidents, for example, are often available in small inquiries in the state parliament. This was quite surprising to me. Pictures for a first image classifier are available online. Customer inquiries arise all by themselves!

Thomas Pawlitzki: You should also consider when it makes sense to create your own model or which “ready-made” model to take. For example, we also use an API for image recognition. These APIs are very easy to integrate and do a really good job with general problems. We’d rather put our energy into providing solutions to problems which general APIs can’t solve. We still have very little data here. Fortunately Markus knows enough tricks to polish small data sets and still come up with usable models.

Markus Nutz: Data augmentation, e.g. changing images, inserting spelling mistakes into words, translating mails into English and back again, window slicing at Time Series Data – these are all strategies that make the existing “few” data as efficient as possible to use! When it comes to models, regarding images and text transfer learning or course, we are particularly interested in Tensorflow Hub. Its a library from Google for reusable Machine Learning modules.

In general, we also pay attention to using suitable models for our existing data, which don’t require the largest amounts of data to function well. Logistic regression or random forests are simply super!

JAXenter: In connection with the construction of a data pipeline you speak of “zero data”. Please give us a concrete example.

Markus Nutz: Oh, that was misleadingly described then. We have chosen “Zero Data” because data – now it’s getting trite – exists everywhere around us and also available to us. We can evaluate initial ideas with data sets from Kaggle or the relatively new Google Dataset Search, and reference official statistics using Openstreetmap data. The simply unbelievably detailed data allows us, for example, to estimate the risk of vandalism for bicycles and cars based on a location or find a good route from bicycle dealer to bicycle dealer for our distribution. It’s a free lunch, so to say.

Thomas Pawlitzki: Yes, that was really surprising and enjoyable when we encountered a problem with theft in a workshop on our bike insurance. We had briefly considered how we could access a good database and whether we could address the various police stations or not. However, a 5-minute search on the net has shown that (at least for the location we examined) a daily newspaper offers up-to-date data. We were surprised and of course very happy about that.

JAXenter: How do you maintain your data pipeline?

Markus Nutz: Phew! I’d like to have a good answer to that, but we don’t have a good recipe yet. I’d say: testing. What helps in any case is that we, as an organization, have a common understanding. Data is what enables us to offer a better product that can distinguish us from the market. That’s why we’re all very motivated to make this happen!

Thomas Pawlitzki:  Yes, sometimes we are a bit “casual” and there’s still room for improvement. Nevertheless, the whole thing works surprisingly well, probably due to the great commitment of all the team members.

Thank you very much!

Markus Nutz and Thomas Pawlitzki will be delivering a talk at ML Conference in Berlin on Wednesday, December 5 about their experience with the cold start problem and building a data pipeline. Starting from “zero data”, how do they arrive to a data pipeline with open, found and collected data? Their data pipeline enables building data products that help customers in their daily life.

• Too many ideas, too little data – How to overcome the cold start problem when building data driven products and services

Carina Schipper has been an editor at Java Magazine, Business Technology and JAXenter since 2017. She studied German and European Ethnology at the Julius-Maximilians-University Würzburg.

The post Too many ideas, too little data – Overcome the cold start problem appeared first on ML Conference.