Tech · Hugging Face
Adding a column or creating a subset for a given filter means that one has to scan and rewrite
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
These two formats play off each other throughout this post and the PRX data pipeline: Lance to build, MDS to stream.
Key facts
- Across repeated decode/encode cycles on both high-resolution (1–2 MP) and lower-resolution (0.25–0.5 MP) real images, the first re-encode at quality 92 is imperceptible
- Each further cycle adds almost nothing, because JPEG quickly converges to a stable state, and even after 10 cycles the images stay well within the imperceptible range, while PNG would be 3–10× larger
- The numbers, averaged over 100 images at quality 92 and measured against the original (higher PSNR and lower LPIPS are better)
- Their smallest training bucket is 512² pixels, and they cap upscaling into a bucket at 4/3 (about 33%), so the effective cutoff is 384² ≈ 147k pixels plus an aspect ratio in [0.5, 2.0]
Summary
In one sentence: they assemble training data from a mix of public and internal datasets, re-caption the images with a VLM, and turn the result into the streamable corpus they trained PRX on. At a high level, the data pipeline looks like this:. The goal was to assemble a large, diverse dataset for pre-training. That is a problem of coverage and diversity, not of per-image perfection. The team assemble their pre-training data from a mix of public and internal datasets. Where a source already comes quality-filtered, deduplicated, and filtered for NSFW content and personal information, they build on that work instead of repeating it at scale. In their experience, what matters most for pre-training is to use long captions that accurately describe everything in the image. The team saw this directly in Part 2, where switching from short captions to long ones substantially improved sample quality.