Cloud Vision API

Core capabilities: what the API detects

Cloud Vision recognizes six main categories of information in images:

Label detection assigns high-confidence tags—'dog', 'beach', 'sunset'—to the whole image or regions within it. It understands both concrete objects and abstract concepts (emotion, activity). Confidence scores range from 0 to 1, so you can filter weak matches.

Face detection locates human faces and returns bounding boxes, facial landmarks (eyes, nose, mouth), and emotions (joy, anger, sorrow, surprise) as sentiment scores. Unlike some competitors, Google does not store face embeddings or profiles in the service layer by default.

Text recognition (OCR) extracts printed and handwritten text, returns full documents or individual words, and works across 50+ languages. Accuracy on printed text rivals specialized OCR engines; handwriting is lower but usable.

Safe search classifies image content as appropriate or not on four axes: adult, spoof, medical, and violence. Each axis gets a likelihood score (very likely to very unlikely), letting you flag or block content at thresholds you set.

Logo detection identifies brand logos and company symbols, useful for content rights enforcement and product cataloging.

Web detection finds the most visually similar images elsewhere on the public web, useful for reverse-image search and detecting reused content.

AutoML Vision: train custom models on your data

The pre-trained detectors work well out of the box for common objects, but they miss domain-specific patterns. AutoML Vision lets you upload labeled images—100+ per class is typical—and train a custom classifier in hours. Google handles the feature engineering, architecture search, and hyperparameter tuning; you get back a model API identical to the pre-trained one.

Use cases: detecting product defects under specific lighting, classifying medical X-ray scans, recognizing rare plant species, sorting agricultural produce. AutoML training costs $1–10 per image depending on model size and complexity. Inference costs are the same as pre-trained ($0.15–0.60 per image for basic models, cheaper in batch). The training data stays in your GCP project; Google does not use it to improve public models.

Deployment options: call the API (same as pre-trained), download a model for on-premise or edge use (TensorFlow Lite for mobile), or export to Google Cloud Storage for batch processing of terabytes of images.

How it works: REST API design

The API is stateless and request-response. Send a POST to vision.googleapis.com/v1/images:annotate with an image (base64-encoded, URL, or GCS path) and a list of features to detect (label, face, text, logo, etc.). The service runs each feature in parallel and returns JSON in under a second for most images.

POST /v1/images:annotate
{
  "requests": [
    {
      "image": {
        "source": { "imageUri": "gs://..." }
      },
      "features": [
        { "type": "LABEL_DETECTION", "maxResults": 10 },
        { "type": "FACE_DETECTION" },
        { "type": "TEXT_DETECTION" }
      ]
    }
  ]
}

Response: labels with confidence, face bounds + landmarks + emotions, OCR text + confidence per word

Batch processing via batchAnnotateImages lets you send 100s of images in one call, useful for processing image archives. For very large pipelines (millions of images), pair with Cloud Dataflow for distributed parallel work.

Privacy and data handling

Cloud Vision does not store images, labels, or face data by default. Each request is processed and discarded; nothing goes into a learning feedback loop or model improvement dataset. This is the key difference from competitors: AWS Rekognition's terms historically permitted face storage and reuse; Microsoft's Computer Vision has similar capabilities but with different data retention policies.

If you opt into the optional feedback mechanism—sending corrections back to improve models—Google keeps that data in your project, not commingled with others. Most users skip this and use the service read-only. For regulated domains (healthcare, biometrics, law enforcement), this default privacy is legally and ethically important.

Encryption in transit (TLS) is always on. Encryption at rest depends on your GCP project settings (default is Google-managed keys; you can use Cloud KMS for customer-managed keys). If processing sensitive data, enable project-level audit logging to track all API calls.

Comparison with AWS Rekognition and Azure Computer Vision

All three services offer similar core features—label detection, face detection, OCR. Key differences:

Face handling: Rekognition stores face vectors by default in a separate collection for face search and comparison; you must opt out. Cloud Vision does not store faces at all. Azure Computer Vision returns face rectangles and attributes but not embeddings, unless you explicitly use the Face API (a separate service).

Pricing: Cloud Vision charges per feature per image ($0.15–0.60). Rekognition charges per image if you use face detection ($0.001 per image for labels, $0.02 for faces). Azure charges per transaction ($1–2 per 1000 images depending on feature). At scale (millions of images), Rekognition edges cheaper; for small to mid-scale, differences flatten.

Accuracy: All three use deep learning and achieve similar top-1 accuracy (95%+) on common object labels. Google's OCR is marginally better on printed text; Rekognition excels at celebrity recognition. For custom models (AutoML), Google and AWS (SageMaker) offer similar ease; Azure requires more manual setup.

Integration: Cloud Vision integrates tightly with BigQuery, Cloud Storage, and Pub/Sub for serverless pipelines. Rekognition integrates with S3, Lambda, and SNS. Choose based on your existing cloud vendor lock-in.

Real-world use cases

Content moderation at scale: Detect adult, violent, or medical imagery in user-generated uploads. Safe Search classifies on four axes; batch 10K images per hour for cost-effective flagging. Combine with text detection to flag offensive text overlays.

Photo organization and search: Automatically tag uploaded photos with labels (dog, beach, sunset) for searchable galleries. Google Photos uses the same tech internally. Label detection works offline in Vertex AI, or call the API from your app.

Product and inventory management: Train AutoML models to classify product photos by category, detect missing barcodes, or identify counterfeits. Integrate with your e-commerce platform for automatic catalog enrichment.

Accessibility: Extract text and describe images for blind users. OCR extracts captions; label detection gives alt-text; face detection can warn when cameras are live.

Document processing: Extract text from scanned invoices, receipts, or forms (works well on printed documents). For structured extraction (field parsing), pair with Document AI, which is specialized for that.

Pricing and quotas

Cloud Vision charges per API call, per feature. As of 2026:

Pre-trained models: $0.15 per 1K requests for LABEL_DETECTION, $0.60 per 1K for FACE_DETECTION, $0.15 for TEXT_DETECTION (OCR). Batch requests (up to 100 images per API call) reduce cost slightly. A million label detections costs ~$150.

AutoML training: $1–10 per image to train a model, depending on dataset size and model type. Inference costs the same as pre-trained (roughly $0.15–0.30 per prediction for basic models).

Quotas: Free tier: 1K free requests/month for LABEL, FACE, TEXT; 25K free for SAFE_SEARCH. Paid tier: up to 10K requests/second per feature; higher on request.

Cost-optimization: use batch processing for off-line archives (2–3× cheaper per image); cache results locally to avoid re-processing the same images; use LABEL (cheapest) as a first filter before FACE or TEXT (pricier).

Integration patterns and code

Python client via google-cloud-vision: install, authenticate with a service account key, and call client.annotate_image(request). Responses are dictionaries with labels, face_annotations, text_annotations arrays.

Serverless pipeline: Upload images to Cloud Storage, trigger a Cloud Function or Cloud Run container on upload, call Vision API, write results to BigQuery. Scale automatically; pay only for processing time.

Batch processing: Use Dataflow or a Compute Engine VM to process millions of images in parallel. Split the dataset, distribute across workers, call the batch API, and write results to GCS or BigQuery.

Real-time streaming: Receive images from a camera or video stream, call the API for each frame, and trigger alerts on detected conditions (face detected, safe search violation, specific object found). Use Pub/Sub to decouple producers from consumers.

Best practices and limitations

Image size and quality: Images must be <10 MB and ≥32×32 pixels. Larger images take longer but are not cheaper; compress for speed if bandwidth is a constraint. Poor lighting, motion blur, and occlusion reduce accuracy; natural, well-lit photos work best.

Batch vs. streaming: Batch is 50% cheaper per image and best for off-line archives. Streaming is faster for small jobs (<100 images). For millions, batch via Dataflow wins on cost.

Combining features: Request only the features you need. Requesting all six costs 3× more than label detection alone. If you only need OCR, specify TEXT_DETECTION only.

Limitations: Face detection does not return embeddings or perform face comparison/verification (use a separate embedding model if needed). OCR confidence on handwriting is lower than printed text. Label detection is not designed for dense scene understanding or pixel-perfect segmentation (use Vertex AI Image Segmentation for that). FACE_DETECTION may miss blurry or obscured faces.

When to use alternatives or augment

If you need face verification (comparing two faces): Cloud Vision returns face landmarks but not embeddings. Use a separate face-embedding model (e.g., FaceNet, ArcFace via Vertex AI) or AWS Rekognition's CompareFaces API.

If you need semantic segmentation (pixel-level masks): Cloud Vision returns bounding boxes, not masks. Use Vertex AI Segmentation or a specialized computer vision model (e.g., Mask R-CNN via TensorFlow).

If you need structured form extraction: Vision's OCR works for general text. For forms (invoices, ID cards, tax documents), use Google Cloud Document AI, which combines Vision with NLP to extract structured fields (invoice amount, date, vendor) automatically.

If you need real-time video processing at edge: Download AutoML models as TensorFlow Lite and deploy on phones, edge devices, or robots. For cloud-based video analysis (frame-by-frame), use Video Intelligence API instead.

Video Intelligence vs. Cloud Vision

Cloud Vision processes static images. For video files or live streams, Video Intelligence API is more efficient: it extracts keyframes, processes them, and correlates results across frames. It detects objects over time (tracking), speech, and scene changes. It costs more per video minute (~$0.10–0.40 depending on features) but saves you from processing every frame independently. Use Video Intelligence for security footage, sports analysis, or long video archives; use Cloud Vision if you have a frame grab or screenshot.

Custom models with AutoML vs. Vision API pre-trained

Start with the pre-trained API; 80% of use cases are solved by it. Custom models shine when the problem is domain-specific (rare plant species, specific manufacturing defects, internal product catalogs) or when accuracy matters more than cost (medical imaging, legal document review). Pre-trained trains instantly and costs nothing to ship. AutoML takes hours to train and costs money per image, but often returns higher accuracy on niche problems.

Hybrid: use Vision API as a first filter (is this an invoice?) and route high-uncertainty images to a custom AutoML model fine-tuned for your domain. Saves on API calls and improves accuracy where it matters.

Getting started: step-by-step

1. Enable the Vision API in your GCP project and create a service account key. 2. Install pip install google-cloud-vision. 3. Set GOOGLE_APPLICATION_CREDENTIALS to your key file path. 4. Make your first call: from google.cloud import vision; client = vision.ImageAnnotatorClient(); image = vision.Image(content=b'...'); 5. Add requests for the features you need and inspect the response. 6. For production, use a managed compute service (Cloud Run, Dataflow) and connect to BigQuery for results storage. 7. Monitor costs with Cloud Billing.