All work

CrocoJobs: AI Job-Market Platform

An end-to-end AI platform for the Tunisian job market: seven sources collected and enriched automatically into 20,000+ postings, served as semantic search, salary estimates, live market analytics, a RAG agent and a full career suite, on $0/month infrastructure.

Visit crocojobs.aliaydi.com

Started

July 2026

Timeline

Ongoing since July 2026

Complexity

Very High

Type

Web App, Data, AI

Every screenshot and the video show the production platform and its live data. Personal details (e-mail addresses, phone numbers) are blurred.

CrocoJobs is an AI platform for the Tunisian job market, live at crocojobs.aliaydi.com. It collects postings from Tunisia's leading job boards and five international sources, cleans and enriches them automatically, and turns them into one searchable market: semantic search, salary estimates where employers stay silent, live market statistics, an AI analyst that answers questions from the data, and a career suite that matches a CV to open roles and writes the application.

I designed and built it end to end, from the scraping and data engineering to the machine learning, the LLM agents, the API, the web app and the cloud operations. It runs on its own every night, watches its own health, and costs nothing to host.

At a glance

• 20,000+ postings from 7 sources, in French, English and Arabic, with 5,400+ employers resolved to one identity each.
• AI salary estimates for 10,500+ postings, each with a calibrated range: on salaries it had never seen, 81% fell inside the predicted 80% range.
• 35 analytics marts behind a market-statistics space you can slice by period, region, occupation, industry, seniority and skill.
• Ask AI, a tool-calling RAG agent with 9 grounded tools that cites the postings and datasets behind every answer.
• A career suite: a knowledge base built from your documents, CV-matched recommendations, tailored CVs and letters typeset as PDF, and interview training.
• An admin console with live capacity, a ledger of every pipeline stage, usage and AI-spend analytics, editable schedules and 118 runtime settings.
• $0 a month: every layer runs on a free tier, by design.

The problem

Job hunting in Tunisia is scattered across boards that each show a slice of the market. About four postings in five publish no salary, so candidates apply blind and negotiate blind. Nobody can see which roles are in demand, where, at what pay, or how many people compete for them. Postings mix French, English and Arabic, and the same employer appears under a dozen spellings.

CrocoJobs puts the whole market in one place and makes it legible: every posting searchable by meaning, every missing salary estimated and labelled as an estimate, every trend measured, and the tools to act on it.

Walkthrough

A 90-second tour of the live platform: smart search, an AI salary estimate, market stats, Ask AI answering in French, For You, an application kit and the admin console.

Architecture

The platform is a medallion pipeline with an intelligence layer on top. Raw payloads land untouched (bronze), are normalized into one job model in Postgres (silver), enriched with vectors, salary predictions and occupation labels (gold), then aggregated into analytics marts. A FastAPI service serves all of it to a Next.js app on the edge, and LangGraph agents reason over the same data. Orchestration, configuration, AI providers, observability and identity are shared services every layer uses.

Features

Search that understands the question

• One list for the whole market, with filters for governorate, posting date, occupation (ISCO-08), employer industry, seniority, education, contract and salary, and an archive of closed postings.
• Hybrid search: full-text ranking (French stemming, accent-insensitive, fuzzy) fused with multilingual semantic vectors, so an English query finds French postings.
• Smart search reads plain language into structured filters. "Accountant jobs in Sfax on a permanent contract" becomes Sfax + CDI + accounting, matched by meaning and keywords.

Salaries, even when nobody publishes them

Every posting without a published salary gets an estimate from a quantile model: a median, a likely range and a confidence score. The page says plainly that it is an estimate and explains how good the model is, measured on held-out data. Each posting also shows its occupation, languages, competition (applicants per opening) and similar postings.

Remote and abroad

A separate section covers foreign-currency work: remote jobs open to applicants in Tunisia and jobs abroad. Pay is shown exactly as published, in its own currency, never converted or blended into Tunisian statistics. Postings carry their country, who may apply and whether the employer sponsors a visa, and the section has its own market page.

Employers

Employers are resolved into one identity however many ways they are spelled, using name folding and contact evidence. Each company has a page with its open roles, where it hires and what those roles pay. Confidential "anonymous" postings count in market totals but never in rankings.

Market statistics

An analytics space answers the questions the market never publishes. Everything can be viewed for the postings listed today, for all time, or for a single year or month.

• Demand: postings by governorate, occupation, industry, contract, working time, language and education.
• Salaries: pay by seniority, occupation, industry and skill, with confidence intervals and a clear split between published and estimated pay.
• Competition: how fast applications arrive, the most contested roles, and "hidden gems" that pay above the median with below-typical competition.
• Skills: a skills cloud by field, and the best-paying skills.
• Explore it yourself: pick any dimension and measure, computed live.

Ask AI

Ask AI is an analyst for the job market. It plans which tools to call, queries live postings and marts, and writes a grounded answer in the language of the question (French, English or Arabic), citing the postings and datasets it used. Answers stream as they are written, conversations are kept as threads, and comparisons such as "which of these two roles pays more" are computed in code rather than guessed by the model.

A career suite

Signed in with Google, a candidate gets tools that use their own background:

• Career profile: upload CVs, cover letters or notes (PDF, DOCX, Markdown, text) and AI builds a structured knowledge base of experience, education, projects, certifications, skills and languages. New documents merge in without overwriting what is already there, and everything stays editable.
• For You: every open posting ranked against that profile, with the reasons for each match and the same filters as search.
• Application kits: for any posting, a fit analysis (score, strengths, gaps, how to improve), a tailored one-page CV, a cover letter and a motivation letter, in the posting's language and typeset as PDF. Kits are editable, and can be e-mailed straight from the candidate's own Gmail account.
• Interview training: a quiz or a voice interview for the role, at a difficulty computed from the posting's seniority and the employer's size, graded with a written report.
• Bring your own AI key (Gemini, Groq or OpenRouter), or use credits.

Admin console

The whole platform is governed from one console, with no redeploy needed to change how it behaves:

• Operations: database and storage size against their alert thresholds, each source's coverage, a ledger of every pipeline stage with what it changed, and usage analytics (requests, AI calls by model and purpose, tokens and spend).
• Pipelines: every workflow's schedule, editable from the page, a "run now" button, and its run history.
• Platform settings: 118 settings in fifteen categories (AI provider chains and models, scraping, ingestion, salary prediction, search, kits, feature flags, per-tier limits, credits, schedules, observability), with encrypted secrets and an audit log.
• Users: accounts, roles, usage per account and credit grants.

Built for phones

Every page is designed for phones as well as desktops, in light and dark mode, and charts lay themselves out for the space they actually have.

Data pipelines and AI

Automated ingestion

Advanced scraping pipelines combine modern anti-bot mitigation with resilient extraction techniques. Browser, HTTP, JSON-API and feed connectors all implement one interface, with retries and backoff, rate limiting with jitter, checkpoint and resume, and incremental scraping that fetches only postings it has not seen. A posting is retired only after its board has been walked in full several times without it, so a partial run can never close a live job. Each source's terms are built into its connector: attribution links, API-only access where required, and licence-mandated retention windows.

Processing and data modelling

• Normalization turns free text into fields: salary ranges and foreign pay, French relative dates, ISCO-08 occupations, a 95-skill lexicon in nine fields, seniority, education, contracts, languages and remote work.
• A temporal model records an observation of every posting at every scrape, which is what powers arrival curves, freshness and application velocity.
• Deduplication links the same posting across boards, confirmed by vector similarity computed inside the database.
• Occupation labels come from a title lexicon, then propagate to unlabelled postings through embedding centroids, with inferred labels flagged as such.
• Employer resolution merges spellings, legal forms and country suffixes, with contact evidence accepted only when it is corroborated.

Machine learning

• Salary model: AutoGluon quantile regression (p10, p50, p90) on a log target, tracked and versioned in MLflow. A challenger is promoted only if it beats the champion on postings neither model trained on, and every promotion re-scores the corpus automatically.
• Confidence comes from the width of the predicted range, calibrated so a typical estimate scores 50%; uncertain estimates are stored but not shown.
• Embeddings: one multilingual model (bge-m3) for every vector. Several hosts serve it, and each was measured against the stored vectors (cosine similarity above 0.9997) before it could join, so switching hosts never requires re-embedding.
• Training runs on a self-hosted runner; inference runs in CI with the champion model cached between runs.

RAG and agentic workflows

• The Ask agent is a LangGraph plan loop with native tool calling. Its 9 tools cover hybrid search, CV matching, Tunisian and international market statistics, role competition and comparison, skill demand, company profiles and custom aggregations. Market data is reached only through parameterized, whitelisted queries: the model never writes SQL.
• Each tool is defined once and rendered both as the native tool schema and as the prompt catalogue, so the two can never disagree.
• Evidence is budgeted row by row to fit the model's context, and every turn has a planning deadline after which it answers from what it has gathered.
• Career agents extract profiles as safe, entry-level operations (they can add and update, never delete), write fit analyses, CVs and letters as structured JSON that is escaped into vetted LaTeX templates, and fit every CV to one page.
• Interview agents write quizzes, conduct the interview stage by stage, and grade against weighted criteria computed in code.

LLMOps

• A provider router walks ordered fallback chains across Groq, Gemini, OpenRouter, Cloudflare Workers AI and Hugging Face, with circuit breakers per model and quota awareness, so one exhausted model steps aside alone.
• Every prompt is versioned in the MLflow Prompt Registry, and every AI call is traced with its prompt version, messages, model and token usage.
• Quality is measured: a golden-set evaluation harness for the Ask agent, and an LLM judge that scores production answers for groundedness.
• Privacy by construction: CVs and profiles are sent only to the user's own AI key or to paid models, never to free-tier models.

Data marts and operations

• DuckDB marts are rebuilt after every enrichment, per period, and published compressed to object storage; the API swaps them in without a restart.
• A data-quality gate checks integrity and distribution drift (PSI) on every run before the marts move.
• An operations schema records every pipeline stage, every API request and every AI call (one row per model tried), rolled up hourly into daily usage.
• Nightly housekeeping measures the database and storage, enforces retention, and alerts by failing its own run when a threshold is crossed.

Architecture and engineering standards

SOLID and modularity

CrocoJobs is a monorepo of independent packages around one shared core library, in clear layers: domain entities, repositories, configuration, AI providers, vector store, transforms, enrichment, analytics, data quality and operations. The scraper, the ML jobs, the pipelines and the API are separate packages that all build on that core.

The seams are interfaces: source connectors, browser providers, LLM and embedding providers, a vector store, a storage backend and a job-queue engine. Extending the platform means adding a class, not editing a flow. A new job board is a connector, a new AI host is a provider, and a new kind of background job is a small queue subclass.

Decoupling that made the migrations seamless

CrocoJobs started as a local stack (Docker Compose with Postgres, Airflow, an MLflow server, local GPU models and a reverse proxy) and moved to the cloud without rewriting its core. Because every dependency sat behind an interface:

• local GPU models became hosted inference with no re-embedding, because the vector space was measured to be the same;
• the local disk became object storage behind the same storage interface;
• Airflow became GitHub Actions calling the same pipeline modules;
• approximate vector search in the database gave way to exact in-memory matching, answering in under a millisecond;
• self-managed sessions became Google sign-in through Supabase, verified with public keys, so the API holds no auth secret.

The same seams are what will let the API move again, to Google Cloud Run, when traffic calls for it.

Quality control

• Unit tests: a focused suite of 144 tests guards the parsing and data logic everything downstream depends on (salaries, dates, deduplication, employer identity, data quality, international pay).
• Validation layers: typed Pydantic models at every boundary, schema-enforced AI output with a repair turn, and deterministic post-processing of everything a model writes.
• Gates: the data-quality gate, the champion gate for models, and an evaluation harness for the agent.
• Consistency safeguards: idempotent migrations (28 of them), atomic quota checks so a refused request costs nothing, idempotent refunds enforced by unique indexes, and leases and heartbeats so queued jobs resume after a restart.
• Release checks: every web deploy is gated by a type check, lint, the production build and a check that no development setting leaked into the bundle; the API ships as a container image built from scratch; and a launch simulation replays a crowd against a throwaway database.

Infrastructure and cost

Zero-cost engineering

CrocoJobs costs $0 a month to run. Every layer was chosen for a free tier that fits it:

• Web: Next.js on Cloudflare Workers, with an edge proxy that gives the app and the API one origin.
• API: a FastAPI container on Render.
• Data: Supabase for Postgres with pgvector, authentication and object storage.
• Pipelines: GitHub Actions, plus a self-hosted runner for model training.
• AI: the free tiers of Groq, Gemini, OpenRouter, Cloudflare Workers AI and Hugging Face, chained so that one running out is never an outage.
• MLOps: MLflow hosted on DagsHub for tracking, the model registry, prompts and traces.
• Analytics: a DuckDB file instead of a warehouse.

Staying free is an engineering discipline, not a setting:

• Database traffic cut by about 8×: vector arithmetic runs inside Postgres, vectors travel in binary at half precision, list pages never load descriptions, and timers re-read a table only when its one-row fingerprint changes.
• Compressed artifacts: the marts travel gzipped (7 to 1), and the salary model is cached in CI, which cut an inference run from about three minutes to eight seconds.
• AI quota management: per-model circuit breakers, daily-quota detection that benches only the exhausted model until it resets, and key pools.
• Memory discipline: a full LaTeX typesetting engine runs inside a 512 MB container, scheduled around available memory.
• Retention by design: rollups, observation thinning and capacity alerts keep the database well inside its limits.

From local to cloud

Moving from a home server to managed free tiers meant solving problems the local stack never had:

• Ephemeral disks: nothing is written to the API host. Uploads, generated PDFs and the analytics file live in object storage, and the API fetches its marts at boot.
• Durable background work: kits and profile merges run through database-backed queues with leases, heartbeats and priorities, which survive restarts and deploys without a broker.
• Hosted models: a measured, labelled vector space, so moving embeddings from a local GPU to hosted inference changed nothing for search.
• One origin on the edge: two Cloudflare Workers reproduce the old reverse proxy, forwarding the real client address for fair per-user quotas.
• Security without cookies: bearer tokens verified against public keys, which makes the API immune to cross-site request forgery by design.

From Airflow to GitHub Actions

Airflow orchestrated the local stack well, but it needs a scheduler, a webserver, workers and a metadata database running all the time, which free hosting cannot carry. GitHub Actions provides free compute, logs, secrets, and self-hosted runners for the heaviest jobs.

Because the pipelines were already modules in the core library, the move only replaced the thin orchestration layer. Asset-driven DAGs became workflows chained by events, each one checking that the previous run succeeded. Schedules moved out of YAML into the platform's own settings: the API dispatches each run, claims each slot at most once, and lets an admin change a schedule from the Pipelines page. A ledger row per stage keeps the run history for good, beyond GitHub's log retention.

Security and privacy

Roles are re-read from the database on every request, so a change takes effect at once. Secrets are encrypted at rest, and each user's own AI keys are encrypted with a key derived for that user alone. Quotas are enforced per tier and per action, atomically. The privacy page states what the code does (which processors receive what, and for how long), and it is updated in the same change as the code.

Tech stack

• Languages: Python, TypeScript, SQL.
• Frontend: Next.js (App Router), React, Tailwind CSS, Magic UI, Apache ECharts, TanStack Query, react-markdown, the Web Speech API.
• Backend: FastAPI, Pydantic, SQLAlchemy, Alembic, Uvicorn, server-sent events, Jinja2, LaTeX (Tectonic, XeTeX), pypdf, httpx.
• Data: PostgreSQL, pgvector, full-text search with unaccent and trigram matching, DuckDB, pandas.
• Scraping: Playwright browser automation, HTTP and JSON-API clients, RSS/Atom feeds.
• AI and ML: LangGraph, bge-m3, AutoGluon, MLflow (tracking, model registry, prompt registry, tracing, LLM judges), Ollama, Groq, Google Gemini, OpenRouter, Cloudflare Workers AI, Hugging Face Inference.
• Orchestration and DevOps: GitHub Actions (hosted and self-hosted runners), Apache Airflow (the original orchestrator), Docker, Docker Compose, uv workspaces, Ruff, pytest.
• Cloud: Cloudflare Workers (OpenNext), Render, Supabase (Postgres, Auth, Storage), DagsHub, Google OAuth, the Gmail API, Google Tag Manager.
• Security: JWT verification against JWKS (ES256), Fernet, AES-GCM with per-user HKDF keys.

Outcome

CrocoJobs is live and runs itself. Every night it collects new postings, retires closed ones, enriches and estimates, checks its own data quality, rebuilds its analytics and reports what it did, all on infrastructure that costs nothing. Job seekers get the market in one place, a salary where employers stay silent, answers grounded in real data, and applications tailored to each role. The architecture behind it has already been through a full move from a home server to the cloud without a rewrite, and is ready for the next one.