Read 2026-10-23
datajuicer/data-juicer: technical due diligence, read from its repository
What Borehole read in datajuicer/data-juicer's code and history at bd547b2: its licence, security advisories, releases, review practice, tests, dependencies and operations. Every finding cites the file and line it rests on. The full report also covers the team and asks the questions the code cannot answer.
| Dimension | Rating |
|---|---|
| Architecture & Codebase — including AI provenance | strong |
| Engineering Process & SDLC Maturity | adequate |
| Product & Engineering Maturity | adequate |
| Security & Reliability Posture | adequate |
| Observability & Operations | not assessable |
| Fit with the Acquirer | not assessable |
Architecture & Codebase — including AI provenance
strongdata_juicer/core/executor/ray_executor_partitioned.py is 2,760 lines, 19× this codebase's median file
adequateThe median source file here is 140 lines. 5 files exceed 1,400 lines, the largest being 2,760. Size is not a defect on its own — some problems genuinely live in one place — but these are where merge conflicts, review fatigue and single-owner knowledge concentrate, and they are the first thing that slows an inheriting engineer down.
data_juicer/core/executor/ray_executor_partitioned.py:1 — 2,760 lines (19× median) |
data_juicer/ops/common/hawor_func.py:1 — 2,636 lines (18× median) |
data_juicer/utils/model_utils.py:1 — 2,349 lines (16× median) |
data_juicer/config/config.py:1 — 2,154 lines (15× median) |
data_juicer/core/executor/gpu_memory_probe.py:1 — 1,530 lines (10× median) |
Engineering Process & SDLC Maturity
adequate25 tests are switched off
adequate25 skip markers across 24 of 435 test files. Each is a failure somebody chose to stop seeing; the question is whether anyone is tracking them.
tests/config/test_config.py:719 — skipped test |
tests/core/tracer/test_ray_tracer.py:11 — skipped test |
tests/ops/mapper/test_generate_qa_from_examples_mapper.py:10 — skipped test |
8% of recent commits change more than 2,000 lines
adequate33 of 395 commits in the last 24 months changed over 2,000 lines of code, not counting reformatting, renames, vendored or generated code. A change that size is rarely reviewed line by line. Ask how they were reviewed, and what made them that large.
| 33 commits over 2,000 lines |
Product & Engineering Maturity
adequateFixes are 51% of recent work, up from 25%
adequateIn the last two quarters 51% of commits were fixes, against 25% in the quarters before. Rising fix work usually means quality debt is coming due, and it takes time that new work would have had.
| 54%, 48%, 29%, 22%, 23%, 15%, 37%, 21% |
Security & Reliability Posture
adequate1 Dockerfile runs the application as root
adequateNo USER instruction switches away from root, so the process runs with full privileges inside the container. It is a one-line fix and an ordinary expectation, which is why its absence is worth asking about: it usually says more about whether anyone has reviewed the deployment than about this container specifically.
Dockerfile — runs as root |
The build runs third-party code it does not pin
adequate10 third-party CI actions are referenced by a tag or branch rather than a commit, so they run whatever that tag points to on the day. Tags have been moved to malicious commits in real supply-chain attacks, and a piped script is whatever the server returns that minute. Pinning to a commit is the fix, and it is small.
.github/workflows/deploy_sphinx_docs.yml:70 — action pinned by tag |
.github/workflows/deploy_sphinx_docs.yml:141 — action pinned by tag |
6 shell commands built from strings
adequate6 places run a command through a shell (`shell=True`, `os.system`, `exec` with a template) rather than passing arguments. Where any part comes from outside, it is command injection. Ask where each one's input comes from.
data_juicer/download/downloader.py:216 — command run through a shell |
data_juicer/ops/mapper/video_active_speaker_detect_mapper.py:193 — command run through a shell |
data_juicer/utils/ASD_mapper_utils.py:259 — command run through a shell |
16 deserialisers that can run code
adequate16 places read data with a deserialiser that executes what it reads (pickle). That is safe on data the system wrote itself and remote code execution on anything else. Ask where each one's input comes from.
data_juicer/ops/common/hawor_func.py:1748 — code-executing deserialiser |
data_juicer/ops/common/mano_func.py:26 — code-executing deserialiser |
data_juicer/ops/mapper/video_active_speaker_detect_mapper.py:199 — code-executing deserialiser |
41 locked dependencies have published vulnerabilities, 26 high or critical in what ships
adequate41 of 524 locked dependencies match 120 published advisories in OSV, 52 of them rated high or critical. A lockfile says which version is installed, not whether the flawed code is reachable from this product, so each needs a look. Most are fixed by an upgrade, and the advisory names the version.
3 vendored components with no licence
adequateThird-party code is committed into this repository without any statement of the terms it arrived under. Unknown terms are harder to deal with than inconvenient ones: an awkward licence can be complied with, an absent one cannot. Each of these needs its origin established before the code changes hands.
thirdparty/LLM_ecosystems — 6 tracked files, no licence file |
thirdparty/humanvbench_models — 5 tracked files, no licence file |
thirdparty/models — 4 tracked files, no licence file |
Observability & Operations
not assessableNot rated: only 5 of 16 checks could reach a conclusion here; the rest do not apply to this repository or could not read it.
Fit with the Acquirer
not assessableNot rated: structurally not assessable from a repository.
Read more, or read your own
The full report on datajuicer/data-juicer: every dimension, the questions for the room, and the audit log.
Survey a repository: free on public repositories. Private code is read on your own machine by a collector whose source you can read first.