One scraper, three AI apps: how my products read the internet before they write anything
Matury Online, Smart-Copy and Smart-Edu share a single service that fetches and cleans web pages for AI models. How it works, what it costs, and why an answer with a cited source beats an answer from memory.
Karol Leszczyński · Toruń, Poland
A language model writes fluently even when it has no idea what it’s writing about. For three of my products that was unacceptable. Matury Online grades a student’s exam essay and has to check the facts. Smart-Copy writes commissioned articles and is supposed to build on what has already been published on the topic. Smart-Edu assembles chapters of academic papers with footnotes pointing to real publications. Each of them needed the same thing: fetch a handful to a few dozen pages from the internet, strip out the menus, ads and scripts, and hand the result to the model as source material.
Rather than three copies of the same code, a single separate service came out of it. It runs on its own small machine in AWS, has one address, and all three applications ask it the same question: “give me the text of this page.”
What the service does
It receives a URL and returns clean text. Along the way it decides what kind of page it’s dealing with. Ordinary pages (articles, posts, documents) are read directly, in a fraction of a second. Pages built in the browser (applications where the content only appears after scripts run) are recognized by their telltale markup and opened in a real, headless Chrome browser; the service waits until the text has loaded and only then collects it. That takes 5 to 30 seconds and eats about 100 MB of memory per browser, so four such browsers run on the machine at once.
The page-type detection is a heuristic: it works in roughly 95% of cases. The remaining 5% are pages that look like applications but aren’t (the browser starts up unnecessarily), or applications without the typical markers (the service returns empty text and the client app tries another way).
Three products, three different appetites
| Product | Pages per job | How long it can wait | What for |
|---|---|---|---|
| Matury Online | 5 | up to 2 minutes (a student is watching the screen) | fact-check an essay before grading |
| Smart-Copy | 10–20 | up to 5 minutes (the client gets an email) | pick the 3–8 best sources for an article |
| Smart-Edu | 30–45 per chapter | hours (the paper is generated in the background) | gather academic publications for footnotes |
The difference in patience translates into settings. A student waiting for a grade sees a “checking sources” message and gets the result within tens of seconds, or a grade without fact-checking. A Smart-Copy client has placed an order and will receive the text by email, so the service can read for longer. Smart-Edu generates a paper chapter by chapter, each with three Google queries (a general one, a specific one, and one with “pdf” appended, because PDF documents are more often academic publications), and then the model discards anything that isn’t a peer-reviewed article, a textbook or a research report.
Why a separate service rather than a piece of each app
Four Chrome browsers means 400 MB of memory. Keeping them next to each of the three applications would have tripled the cost and complicated everything for no reason. One dedicated machine costs about $20 a month and covers the current scale. On top of that come Google search queries: the first hundred a day are free, beyond that it’s $5 per thousand, which across three products adds up to $5–10 a month.
The price of this design is a single shared point of failure. When the service is down, all three products lose the ability to read the internet at the same moment. That’s why each of them has a fallback path: Matury Online grades without fact-checking and tells the user so, Smart-Copy writes from the client’s own sources, Smart-Edu puts the chapter back in the queue and returns to it later.
What a year of running this service taught me
The biggest problem wasn’t the technology — it was load spikes. When Smart-Edu is assembling a master’s thesis (45 pages to read) at the same time Smart-Copy receives an order for a long article (15 pages), sixty requests hit four browsers and line up in a queue. The fix turned out to be a priority queue: a job someone is watching jumps ahead of a job that will go out by email anyway.
The second lesson is about trust. An answer with a cited source can be verified with one click; without a source, all that’s left is trusting the model. In Matury Online the student sees the basis on which the grade flags a factual error. In Smart-Edu every footnote leads to a publication the system actually read. That principle later carried over to Cytado, where a citation is pinned to a page number, and if the source isn’t in the corpus, the system says “not found” instead of making one up.
What this means for a company that wants to adopt AI
If an assistant is supposed to answer questions about your products, contracts or documentation, the model alone isn’t enough. You need a layer that reaches the right documents, cleans them and hands them to the model together with information about where they came from. An AI implementation built this way gives answers an employee can verify with one click, and a bill calculated per call, with a cap so the cost never surprises anyone. Implementations start at 5 000 zł net; if you have documents such an assistant should work on, describe them to me and I’ll show you a trial run on your own data.