Job Data Scraping: How It Works and How to Build One

Job data scraping is the automated collection of job-listing data into structured records for labor-market analysis, recruitment operations, or monitoring your own postings. As of October 2026, the fastest reliable path is to skip scraping where you can: use your ATS export, a public ATS job board API such as Greenhouse or Lever, or a licensed feed, and scrape permitted career pages only when no feed exists.

A reliable job scraper does more than copy titles and descriptions. It chooses an authorized source, preserves employer and location context, detects expired roles, and prevents duplicate listings from distorting the dataset. If you do collect public career pages, keep the source list and fields narrow, review site terms and robots.txt, avoid applicant or profile data, and use conservative request limits. This guide covers that end-to-end workflow.

What teams use job data for

• Labor-market research, such as tracking how demand for a skill or title changes by region.
• Recruitment and staffing, such as spotting which employers are hiring.
• Competitive hiring signals, such as which roles a rival opens and closes.
• Monitoring your own postings across your careers page and the boards that syndicate them.

The use decides the source. Research across many employers usually needs a licensed dataset, while monitoring your own jobs needs only your ATS.

How job scraping works

A job scraper repeats the same loop for every source:

1. Discover listing URLs or IDs from an API, sitemap, or careers index.
2. Fetch each listing (or one API response that holds many).
3. Extract title, employer, location, employment type, pay, dates, and the source ID.
4. Normalize titles, locations, and pay into consistent fields.
5. Deduplicate the same role seen on several sites or URLs.
6. Track changes so updated and closed roles are marked, not left active.

Most of the work is in steps 4 to 6. Getting a title off a page is easy. Keeping a dataset of open roles accurate over weeks is the hard part.

Build, buy a tool, or buy the data?

Job boards such as Indeed and LinkedIn prohibit scraping in their terms without permission. The Indeed job scraping guide quotes the exact clauses. Employer career pages and public ATS job board APIs are usually the better place to build. The data scraping tools comparison covers no-code and code options, and the LinkedIn scraping guide covers that platform separately.

Start with the right source

The source determines the reliability and maintenance cost of the whole pipeline. Do not open a browser and search for selectors until you know which access method is permitted and which one owns the most authoritative version of the listing.

Use this order of preference:

1. Your own ATS export or webhook. This is usually best for monitoring listings your organization publishes.
2. An official API or partner feed. It provides a documented schema, identifiers, pagination, and update behavior.
3. A licensed job dataset. Confirm its provenance, permitted uses, refresh interval, and deletion process.
4. Permitted public career pages. Collect only the approved pages and fields, with source-specific request budgets.
5. Do not collect. If the source prohibits the method or the project lacks a valid basis for processing the data, a different tool or proxy does not fix the problem.

This sequence avoids rebuilding data access that a source already supports. It also makes corrections and closures easier to propagate because stable source identifiers survive page-layout changes.

Public visibility is not the same as unrestricted reuse. Review the source's terms, access controls, privacy obligations, database and content rights, and the laws that apply to your project. The Robots Exclusion Protocol standardizes how crawlers interpret robots.txt, but robots rules are only one technical signal; they do not replace permission or legal review.

Define the job record before collecting pages

A job listing is a changing record, not a permanent document. Define a small schema that supports the decision you need, and reject unexpected fields instead of storing full pages by default.

A practical record might look like this:

{
  "source": "careers.example",
  "source_job_id": "ENG-142",
  "canonical_url": "https://careers.example/jobs/ENG-142",
  "title": "Platform Engineer",
  "organization": "Example Company",
  "employment_type": "FULL_TIME",
  "location": {
    "country": "US",
    "region": "NY",
    "locality": "New York"
  },
  "remote_policy": "hybrid",
  "date_posted": "2026-07-08",
  "valid_through": "2026-08-08T23:59:59Z",
  "observed_at": "2026-07-10T12:00:00Z"
}

Keep compensation as structured amount, currency, and time unit fields rather than one text string. Store remote eligibility separately from the office address: a role can have an office location while accepting applicants only from specific countries or regions.

The vocabulary in Schema.org's JobPosting type is a useful starting point for field names. Google's job posting structured-data documentation also distinguishes physical locations, remote roles, applicant-location requirements, publication dates, expiry dates, employment types, and compensation. Not every page implements those fields correctly, so validate them against visible listing content and source rules.

Avoid collecting applicant details, employee profiles, email addresses, or inferred sensitive attributes just because they appear near a listing. They are not needed for a job-market record and introduce additional privacy and security risk.

Public ATS job board APIs

Many employers host their careers page on an applicant tracking system (ATS) that publishes open roles through a documented, public API. These are often the cleanest source for employer job data, because the employer chose to publish the data there.

• Greenhouse. The Job Board API says job board data is public and GET requests need no authentication. List jobs with GET https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs. Add content=true for descriptions, departments, and offices.
• Lever. The Postings API lists published roles at GET https://api.lever.co/v0/postings/{site} (EU accounts use api.eu.lever.co). GET requests need no API key.

import requests


def greenhouse_jobs(board_token: str) -> list[dict]:
    response = requests.get(
        f"https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs",
        timeout=30,
    )
    response.raise_for_status()
    return [
        {
            "source": "greenhouse",
            "source_job_id": str(job["id"]),
            "title": job["title"],
            "location": (job.get("location") or {}).get("name"),
            "canonical_url": job["absolute_url"],
            "updated_at": job.get("updated_at"),
        }
        for job in response.json().get("jobs", [])
    ]


def lever_jobs(site: str) -> list[dict]:
    response = requests.get(
        f"https://api.lever.co/v0/postings/{site}",
        params={"mode": "json"},
        timeout=30,
    )
    response.raise_for_status()
    return [
        {
            "source": "lever",
            "source_job_id": posting["id"],
            "title": posting["text"],
            "location": (posting.get("categories") or {}).get("location"),
            "canonical_url": posting["hostedUrl"],
        }
        for posting in response.json()
    ]

The board token or site name is usually visible in the employer's careers page URL (for example, job-boards.greenhouse.io/{board_token} or jobs.lever.co/{site}). A role that disappears from the API response has usually been closed or unpublished, which makes lifecycle tracking much simpler than with HTML pages. Still confirm the absence on a second run before you mark it closed.

Discover listings without an unbounded crawl

Use a source registry rather than starting from arbitrary search results. Each source entry should identify the approved hostname, discovery method, URL patterns, parser version, request budget, expected fields, and an owner who can pause collection.

Useful discovery inputs include:

• A documented API or ATS feed.
• A sitemap limited to job-detail URLs.
• A company careers index with explicit pagination.
• A controlled list of employer career pages.
• Known listing URLs supplied by your own system.

Normalize a URL before enqueueing it. Remove tracking parameters, resolve relative URLs, require HTTPS, reject redirects outside the approved host, and deduplicate canonical forms. Never let page links turn a scoped job collector into a general site crawler.

Job aggregators often repeat the same underlying role, while an employer may publish one position under several city URLs. Treat discovery URLs as observations, not unique jobs.

Extract structured data before using selectors

For a permitted career page, inspect machine-readable data before building layout-dependent selectors. Many job pages include JSON-LD with a JobPosting object. An ATS endpoint or embedded state may also expose the same fields in a structured response.

Use this extraction order:

1. Supported API or feed response.
2. Valid JobPosting JSON-LD that matches the visible listing.
3. Stable semantic HTML fields.
4. Source-specific selectors as a last resort.

Google's job posting documentation lists title, description, datePosted, hiringOrganization, and jobLocation as required properties. It requires validThrough for postings that expire. This script pulls JobPosting objects out of one permitted page:

import json

import requests
from bs4 import BeautifulSoup


def jobposting_jsonld(url: str) -> list[dict]:
    response = requests.get(
        url,
        headers={"User-Agent": "job-data-bot/1.0 (+https://example.com/contact)"},
        timeout=30,
    )
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    postings = []
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        nodes = data.get("@graph", [data]) if isinstance(data, dict) else data
        for node in nodes:
            if not isinstance(node, dict):
                continue
            types = node.get("@type")
            if "JobPosting" in (types if isinstance(types, list) else [types]):
                postings.append(node)
    return postings

Do not trust structured data blindly. Confirm that the object belongs to the current page, the employer matches the source, the URL is canonical, and the listing is still open. Pages sometimes retain stale markup after visible content changes.

Use a normal HTTP client when the permitted data is present in the initial response. A browser is justified only when the required fields genuinely depend on JavaScript or the authorized workflow requires browser state. Browser collection consumes more bandwidth and adds failure modes around cookies, consent screens, rendering, and session expiry. For an implementation pattern, see the Playwright proxy guide.

Normalize job titles, locations, and employment types

Raw strings should remain available for audit, but analytics need normalized fields. Keep the transformation deterministic and versioned so a taxonomy update does not silently rewrite history.

Do not strip meaningful qualifiers such as contract duration, shift, clearance requirement, or applicant location. A remote role restricted to Canada is not equivalent to a globally remote role.

Normalize conservatively. If a field is ambiguous, store unknown and retain the source value. Guessing that "Springfield" belongs to a particular state or that "competitive" indicates a salary creates plausible but false data.

Deduplicate listings with stable evidence

Use the source's job ID as the first identity key. If no stable ID exists, build a fingerprint from normalized employer, title, location or remote scope, and a source-controlled requisition reference. Do not deduplicate on title alone.

Separate three situations:

• Same source, same ID: update the existing record and append an observation.
• Same employer role on multiple location URLs: attach the pages to one role only when the content provides evidence that they share a requisition.
• Aggregator copy and employer original: keep source lineage, prefer the employer record as canonical, and record the aggregator URL as a secondary observation if your license permits it.

Descriptions are weak deduplication keys because boilerplate can dominate them and minor edits produce different hashes. Use description similarity only as supporting evidence, with a review threshold for uncertain matches.

Track the full listing lifecycle

Job data becomes misleading when closed roles remain active. Store every fetch as an observation and derive the current state from explicit evidence.

Classify these outcomes separately:

• Active listing with unchanged fields.
• Active listing with a material update.
• Explicitly expired through validThrough or a closed status.
• Removed with HTTP 404 or 410.
• Temporarily unavailable because of a timeout or server error.
• Unexpected page, login screen, consent page, or challenge.

One failed request is not proof that a job closed. Retry transient failures with capped exponential backoff and jitter. Require a second observation or an explicit closure signal before changing a role from active to closed.

Keep date_posted, valid_through, first_seen_at, last_seen_at, and closed_at distinct. The source publication date and your collector's first observation answer different questions.

Pace scraping per source

Scraping job sites responsibly means giving each approved source its own queue, concurrency cap, cache policy, and stop conditions. A global requests-per-second limit is not enough because one small employer site and one documented API have different capacities and rules.

Begin with a low request rate and increase only when the source permits it and monitoring shows capacity. Add jitter to schedules, use conditional requests such as ETag or Last-Modified where supported, and avoid fetching images, fonts, and other assets that do not contribute to extraction.

Classify responses before retrying:

The HTTP 429 guide explains backoff and request pacing in more detail.

When proxies help job scraping

Proxies are useful when an authorized job scraping project needs controlled regional egress, stable outbound IPs, or isolation between independent source workers. They do not grant permission, repair a parser, or make a prohibited request acceptable.

Match routing to the collection task:

For regional checks, residential proxies can provide country, state, or city routing. For a session-heavy ATS or company careers workflow, keep the proxy, cookies, headers, and browser context stable from the first page through pagination. Rotate only between independent jobs when the source rules allow it.

Do not build an unlimited retry loop that swaps an IP after every denial. That hides the operational signal and can turn a recoverable configuration problem into abusive traffic.

Validate job data before publishing analytics

A successful HTTP response is not the same as a valid job observation. Put a validation boundary between extraction and storage.

Check that:

• The page is a job detail page for the expected source.
• The source ID, canonical URL, employer, and title are present.
• Location and remote restrictions do not contradict each other.
• Dates parse correctly and valid_through is not before date_posted.
• Compensation has a currency and time unit when an amount is present.
• The record does not match a known closed job or uncertain duplicate.
• Unexpected personal data and unapproved fields are removed.

Quarantine records that fail validation instead of forcing defaults. Track valid-record rate, missing-field rate, duplicate-review rate, unexpected-page rate, closure-confirmation lag, and parser errors by source and parser version.

Run source adapters against a small approved fixture set before deployment. Include active, expired, remote, multi-location, salary-range, missing-field, consent, and error pages. Fixtures let you detect parser drift without repeatedly requesting production pages.

FAQ

Is job data scraping legal?

It depends on the source, jurisdiction, method, contract terms, data, and intended use. Prefer first-party exports, official APIs, and licensed feeds. For public pages, review terms, robots rules, privacy requirements, content rights, and applicable law; obtain qualified advice for a real project.

Is there an API for job listings instead of scraping?

For individual employers, yes. Greenhouse's Job Board API and Lever's Postings API publish open roles without authentication, as of October 2026. There is no single official API that covers every job board, and the large boards do not offer a public job search API. See the Indeed job scraping guide for how Indeed's API options work.

What data should a job scraper collect?

Collect only fields required for the project, such as source ID, URL, title, employer, employment type, location, remote eligibility, compensation, publication and expiry dates, and observation timestamps. Avoid applicant, profile, contact, and inferred sensitive data unless a separate authorized use specifically requires them.

How do you scrape job sites with JavaScript-rendered listings?

First check for an approved API, feed, JSON-LD object, or structured response. If the permitted fields truly require rendering, use a browser with blocked nonessential assets, one stable session, strict navigation limits, and source-specific pacing. Do not use a browser simply because it is convenient.

How do you detect duplicate job postings?

Prefer a source job ID or employer requisition ID. Otherwise combine normalized employer, title, location or remote scope, and source evidence. Treat description similarity as supporting evidence, not a definitive identity key.

Can proxies prevent job scraper blocks?

No. Proxies change network routing; they do not fix missing permission, excessive traffic, broken sessions, stale selectors, or denied access. They help with legitimate regional measurement, stable egress, and worker isolation when the source permits the collection.

How often should job listings be refreshed?

Use the slowest interval that still supports the business decision and complies with the source's rules. Active listings may need periodic checks, while closed records should stop generating work. APIs, webhooks, feeds, conditional requests, and adaptive schedules can reduce unnecessary requests.

Final thoughts

Reliable job data scraping begins with an authorized source and a precise record schema. Prefer ATS exports, APIs, and feeds; validate structured data against the listing; preserve source identity; normalize conservatively; and track every job from discovery through confirmed closure.

When permitted pages require regional or session-stable routing, proxies can support the collection infrastructure. They should remain one controlled component of a job scraper built around source policies, careful pacing, validation, deduplication, and auditable lifecycle data.