If you are looking to outsource image annotation, there are many points to consider before contacting image annotation companies. With an influx of AI training data companies, choosing the best one for your project can be a difficult task.
As a provider of image tagging services, Lionbridge has 20 years of experience and knowledge that can help save you from unnecessary headaches when outsourcing training data. From standards of quality to platform considerations, below are five basic tips on how to outsource image annotation.
5 Tips on How To Outsource Image Annotation
1. Examine Workbench Demos or Data Samples
Evaluating the company background is one of the most rudimentary, yet important things to do. Of course, you should research the company’s proven experience in the market. However, since AI training data companies often provide multiple services, you should look for proof of their image annotation capabilities.
Sample of Lionbridge’s Image Categorization Workbench
Browse the company’s site and look at the quality of the photos, site design, and instructional videos. Most of the leading image annotation companies will provide graphics, gifs, or videos to demonstrate their services. You should look for graphics or videos that show the company’s workbench, image annotation tools, and types of image annotation they provide. Some companies may even have samples of annotated data available for you to evaluate.
2. Decide How You Want the Data Verified
Based on the needs of your project, it is important to accurately identify how the service will be handled. One element you should decide beforehand is how you want the image annotations to be verified.
A good example of this is image classification. With image classification, you task annotators with selecting the appropriate label for each image. However, with human-powered classification there is a chance of human bias affecting the data. To combat this, there are multiple levels of verification you can implement:
Single pass: one annotator labels the image.
Double-blind verification: two annotators label the image without seeing each other’s work. If the labels don’t match, a supervisor observes the case and decides the correct label.
Multi-blind verification: 3 or more annotators label the image without seeing each other’s work. If the labels don’t match there are two options: consensus or adjudication. With consensus, majority rules and the most common label among the annotators is chosen. With adjudication, a supervisor is brought in and decides the correct label.
While different companies may use different terms, these are the standard systems used in data entry and annotation. Stricter verification will take more time and may come with additional costs. Be sure to decide which verification method is best for your project and budget.
3. Define Your Standards of Quality
Aside from verification methods, you must also clearly dictate your standard of quality. Many companies will say they provide “accurate” training data. However, what does “accurate” mean to you?
For example: with bounding boxes, cuboids, and polygons, how precisely do you need these annotations to be drawn? Some projects may require annotators to zoom in to the very edge of the object and adjust the the annotation down to the pixel. Other projects may allow for a margin of error. The same goes for accuracy of the image data as a whole. Some clients require 100% accuracy in the data. Meanwhile, others may allow for 90% accuracy.
The important point is that you dictate these terms and establish guidelines before consulting with the outsourcing company. Give the company examples of what you consider correct image annotations, and incorrect image annotations. Explain the exact format or file types you require, how you want the data batched, and quality control systems you want implemented. The more information you give the company, the better the final product will be.
Request a Trial Project
The best way to make sure the company understands your guidelines is to request a trial of their services. Some companies will offer free trials, and others may charge a small fee.
Through the trial project, the company should demonstrate their workflow, the speed at which they can annotate your images, and the quality they can provide. Even if there is a fee involved, it is worth paying to judge the company’s capabilities before committing to a full project.
4. Determine Who Will be Annotating Your Images
Every AI training data company has their own workflows, business model, and staffing system. If you are annotating images used in a specialized field, it is important to determine exactly who will be working on your project.
Does the company provide onsite staff, remote staff, or both? How are the workers trained and do they hold certain qualification standards? You should ask these questions to every outsourcing company you speak to.
While some annotation tasks may not require specialist knowledge, the same cannot be said for annotating lesions or tumors in CT scans. If you require experts in a certain field to annotate your images, make sure the company has the resources to source specialized staff.
5. Decide On an Image Annotation Platform
One important element to consider is whether the company has their own image annotation platform. At Lionbridge, we don’t charge any platform fees. However, some companies may charge additional fees to use their platform.
Lionbridge Image Annotation Platform
Annotating on the company’s proprietary platform has many benefits. Firstly, the company owns the platform and can make changes to it if your project requires a specific feature. If the company crowdsources workers, their annotators should already know how to use their platform. Therefore, they should not require additional platform training.
On the other hand, you may have your own custom platform or a third party’s platform that you want to use. If you need your images annotated on an external platform, companies may charge extra fees to train their staff on that platform. Be sure to inquire about platform fees when getting a quote.
When outsourcing image annotation, you may run into companies that only provide the image annotation software. You may also find companies that only provide the staff. It will be much more efficient for you in the long run if you find a company that provides both. At Lionbridge, we provide the software, the annotators, and the added value of project management. You give us the data and your guidelines, we’ll handle the rest.
If you’re looking for more advice on sourcing annotated image data, please see our related resources below. Need a quote on image annotation services right now? Get in touch with our sales team.