Which background removal tools to use and why?
By Mirna Veršić on March 8, 2023
Table of contents
Even though separating foreground from background is a fairly simple task for any human, it can be quite troublesome to find adequate tools to do that for you. With the advent of artificial intelligence, background removal, once possible only with manual labor, started being something that can be automated. Now we have a lot of free tools, commercial API services and open-source algorithms. All of them are using some kind of AI, mostly machine learning algorithms for computer vision.
We are an AI company that works on different computer vision tasks, and often times background removal is a part of the processing pipeline. Working on one of my latest projects, I needed a good background removal tool. I was looking into both commercial API solutions and open source projects. After searching the web for available options, I came up with a list of 9 different solutions. I will describe each of these solutions in more detail in the following chapters.
Measuring performance of the background removal tools
Before I was ready to test the listed options, I had to create a test dataset and figure out a way to measure performance. The chosen dataset consisted of a small sample of different images, mostly containing a single fashion object. Some images included models posing in the outfit, and some also included situations with multiple smaller objects in the same image.
The best way to measure performance for different tools, in this case, was to create a dataset containing, what I considered, easier and more difficult examples. Of course, every tool that didn’t perform well on the easier examples was immediately discarded, but what made the difference between well-performing tools were the obtained results on more difficult examples.
The images below depict a sample of the dataset I used, which contains one easy, one medium and one difficult example, with the far left image being the easiest example.
We scored the solutions based on the number of correctly processed images - the image is correctly processed if the number of pixels classified as foreground/background is above a certain threshold.
After choosing the appropriate dataset and a method for comparing different tools, I was ready to start trying out listed solutions. I first started looking into available open source tools, so let’s take a look at those first.
Free and open source tools
There’s a handful of available open source background removal projects out there, but only the following 4 got on my list based on the type of the problem I was solving. Those were:
rembg
rembg is an open source background removal project that offers 3 different models for removing the background. The available models are U2net, U2netp and U2net_human_seg. All these models are based on the U2net architecture, with the difference that U2netp has less feature maps than U2net, and is therefore a lighter model, and U2net_human_seg is fine-tuned for tasks of human segmentation.
There was no available information about the dataset used for training these models, so it was uncertain if this project would fit my needs for fashion objects.
The next logical step was to test rembg using the created dataset. After testing the performance of all 3 available models, the following results were obtained.
| Model | rembg Accuracy |
|---|---|
| U2net | 29.2% |
| U2netp | 37.5% |
| U2net_human_seg | 25.0% |
Images below depict the obtained results for the rembg project.
As seen from the results, none of these models worked particularly well. U2netp seemed to produce the best results, but even its accuracy was beneath 40%. U2netp performed well on easy and medium difficult examples with clear backgrounds, but failed on all the test images with more demanding backgrounds.
Background Remover
After testing the performance of the rembg tool, it was time to move to the second tool on the open source list, BackgroundRemover. Similarly to the rembg project, BackgroundRemover also supported 3 different models: U2net, U2netp and U2net_human_seg. There was also no information about the dataset used for training the models, so the only way to know more about them was to test them.
I tested BackgroundRemover the same way as rembg, with all 3 available models tested and scored separately on the same dataset.
| Model | Background Remover Accuracy |
|---|---|
| U2net | 29.2% |
| U2netp | 37.5% |
| U2net_human_seg | 25.0% |
BackgroundRemover had slightly better results than rembg, but it still had barely over 40% accuracy for the U2netp model. It shared similar problems as rembg. It performed well on easy and medium difficult examples with clear backgrounds, but as soon as I tested it on more demanding examples, it started to fail more and more.
Clothes Segmentation
Next in line was the Clothes Segmentation project. This project looked promising, since it was specifically trained for tasks of clothes segmentation, and the sample results on the project’s main page looked amazing. The following images depict the sample images provided on the main page of the Clothes Segmentation project.
Unfortunately, after researching more about this project, I found it had two big flaws. The first flaw was that the segmentation of clothes completely fails if the person wears a clothing style that is extremely different from the trained dataset.
The second flaw was that the model doesn’t work if the image contains multiple people/objects. This made it impossible to use this tool for my dataset, since my dataset contained images with multiple fashion objects. That is why I didn’t move to the testing phase with this project.
DIS
DIS stands for Dichotomous Image Segmentation. DIS is an open source image segmentation project trained using the DIS5k dataset. This dataset contains 225 categories in 22 groups. One of the groups covers accessories, so there was reason to believe DIS might be a good fit for segmentation of fashion objects.
One of the major advantages of DIS is the ability to control which parts of the image are considered background by using an obtained segmentation mask as alpha channel, and filtering the original image by setting a threshold.
There are two pre-trained DIS models available. One was trained for academic comparisons, and the other was trained for general use. The obtained results are shown in the table below.
| Model | DIS Accuracy |
|---|---|
| General purpose | 41.7% |
| Academic | 16.7% |
Images below depict the obtained results for the DIS project. The first row contains results obtained when using the model for academic comparisons, and the second row contains results obtained when using the general purpose model.
Even though DIS seemed promising, the obtained results were similar to the rembg and BackgroundRemover projects. DIS probably failed on the test dataset because it was trained for many categories on a not that large dataset.
According to the announcements on DIS’s main page, dataset DIS V2.0 is coming out soon. This dataset should be much larger than DIS V1.0, and DIS might be better suited for my needs. But for now, we are moving to commercial background removal tools, as we have depleted all open source tools without getting desired results.
Commercial APIs
Since open source projects didn’t produce desired results, it was time to look into commercial APIs, in the hope of finding a tool that will be satisfactory.
After extended search for available commercial background removal APIs, I ended up with a list of 5 different tools. The list was as follows:
Evaluation
Considering I was now looking into paid background removal tools, the price of a tool became another important question alongside the tool’s performance. Before moving onto testing of listed tools, I first looked at prices.
Since my test dataset was relatively small, but the dataset I wanted to remove the background from was relatively large, I compared both the prices for small datasets and for large datasets. Small datasets are, in this case, datasets smaller than 500 images, and large datasets are larger than 5000 images. The price for each of the tools is listed in the table below.
| Tool | Small dataset | Large dataset |
|---|---|---|
| Removebg | 89 EUR | 589 EUR |
| Slazzer | 65 EUR | 412 EUR |
| Photoroom | 70 EUR | 500 EUR |
| Removalai | 64 EUR | 360 EUR |
| ClipDrop | 29 EUR | 149 EUR |
None of these tools shared any information about the model or the dataset used for training the model, so the only way to see which tool is the best was to test them all and measure their performance. I tested all the listed APIs using the same test dataset. The results are listed in the table below.
| Service | Accuracy |
|---|---|
| Removebg | 41.7% |
| Slazzer | 54.2% |
| Photoroom | 70.8% |
| Removalai | 37.5% |
| ClipDrop | 41.7% |
Images below depict the obtained results for commercial APIs. Each row of pictures represents a different tool. The tools are listed as follows: Removebg, Slazzer, Photoroom, Removalai, ClipDrop.
Finding the best tool
Removebg, Removalai and ClipDrop all had similar results as DIS and rembg when I used U2netp. A nice touch to removebg is its ability to return an error message if it is unable to find the background for the image. This information could be used to select difficult images and maybe use a different tool on them.
I discarded all three mentioned APIs, because the same results were obtained by using any of the available open source models. Furthermore, in one of the examples, ClipDrop changed the color of a small part of the image when removing the background. A background removal tool should never affect the colors in the image, and that was another reason for discarding the ClipDrop tool.
The only tool listed here that stood out was Photoroom. Photoroom performed exceptionally well on most of the images from the dataset, and it scored an impressive 70% accuracy.
It failed only on the hardest examples in the dataset, for which it was not expected that a computer would completely remove the background correctly. Photoroom’s impressive performance on my test dataset was no surprise, considering one of the people creating it was Yann LeCun, one of the most famous computer scientists in the world and a father of convolutional neural networks.
Conclusion
In the search for the best background removal tools, I tried both free, open source projects and commercial API services. Open source projects often share the information about the model and dataset used for training that model. With that information, it is possible to fine-tune a given model to fit your dataset and create a better tool for your specific task.
On the other hand, working with open source projects produces lower results, mostly because of the size of the dataset used for training those models. Many of the projects covered in this post either didn’t share information about the dataset or used what seemed a relatively small dataset.
When working with the commercial tools, we usually have no information about the dataset, or the model used. The only way to know how the tool works is to test it. That is why most commercial bacground removal tools offer free demos to try them out.
In this blog post, we tried out 8 different background removal tools. Three of them were open source projects, and five of them were commercial solutions. Photoroom, a commercial service, achieved the highest score. Majority of other solutions had significantly lower performance.
| Background removal tool | Accuracy |
|---|---|
| Rembg U2Netp | 37.5% |
| BackgroundRemover U2Netp | 41.7% |
| DIS: General purpose | 41.7% |
| Removebg | 41.7% |
| Slazzer | 54.2% |
| Photoroom | 70.8% |
| Removalai | 37.5% |
| ClipDrop | 41.7% |
If money is not the problem, the best solution out of the tested background removal tools is Photoroom. Its performance is much better than the other options for our use case. For your use case, it’s best to evaluate them on your own.
Out of the available open source projects, I would highly recommend either using DIS for general-purpose datasets or, if the dataset is fashion-oriented and doesn’t contain multiple people in the same image, I would recommend trying out Clothes Segmentation.
Let us know if there are other better models or services for background removal available. You can follow us on LinkedIn for new updates!
Recent Blog Posts
We build AI for your needs
Partner with us to develop an AI solution specifically tailored to your business.
Contact usMembers of
