Read 2026-10-23
vibrantlabsai/ragas: technical due diligence, read from its repository
What Borehole read in vibrantlabsai/ragas's code and history at 298b682: its licence, security advisories, releases, review practice, tests, dependencies and operations. Every finding cites the file and line it rests on. The full report also covers the team and asks the questions the code cannot answer.
| Dimension | Rating |
|---|---|
| Architecture & Codebase — including AI provenance | strong |
| Engineering Process & SDLC Maturity | adequate |
| Product & Engineering Maturity | adequate |
| Security & Reliability Posture | adequate |
| Observability & Operations | not assessable |
| Fit with the Acquirer | not assessable |
What the repository says
SECURITY.md:62: names OpenSSF
Architecture & Codebase — including AI provenance
strongsrc/ragas/metrics/base.py is 1,479 lines, 10× this codebase's median file
adequateThe median source file here is 139 lines. 2 files exceed 1,390 lines, the largest being 1,479. Size is not a defect on its own — some problems genuinely live in one place — but these are where merge conflicts, review fatigue and single-owner knowledge concentrate, and they are the first thing that slows an inheriting engineer down.
src/ragas/metrics/base.py:1 — 1,479 lines (10× median) |
src/ragas/integrations/ag_ui.py:1 — 1,390 lines (10× median) |
Engineering Process & SDLC Maturity
adequate15 tests are switched off
adequate15 skip markers across 9 of 118 test files. Each is a failure somebody chose to stop seeing; the question is whether anyone is tracking them.
tests/e2e/metrics_migration/base_migration_test.py:196 — skipped test |
tests/e2e/metrics_migration/test_answer_correctness_migration.py:395 — skipped test |
tests/e2e/metrics_migration/test_answer_relevancy_migration.py:273 — skipped test |
Product & Engineering Maturity
adequate13% of bug fixes add or change a test
adequateOf 282 fixes in the last 24 months, 37 touched a test. A fix without the test that would have caught the bug can quietly come back, and the suite never learns from what went wrong.
| 37 of 282 |
Security & Reliability Posture
adequateThe build runs third-party code it does not pin
adequate5 third-party CI actions are referenced by a tag or branch rather than a commit, so they run whatever that tag points to on the day. Tags have been moved to malicious commits in real supply-chain attacks, and a piped script is whatever the server returns that minute. Pinning to a commit is the fix, and it is small.
.github/workflows/ci.yaml:27 — action pinned by tag |
.github/workflows/ci.yaml:89 — action pinned by tag |
2 advisories against this project in the last 24 months, 1 high or critical
adequateThe newest was published on 2026-04-20. A project that publishes advisories is telling its users about its fixes, and the count and pace say how much security work it carries. The record is incomplete: NVD: 1 record names ragas with nothing tying it to this repository (CVE-2026-6587).
| 2026-04-20: GHSA-95ww-475f-pr4f (low): RAGAS has SSRF via Multi-Modal Faithfulness Collections Module |
| 2026-03-05: GHSA-v2xr-wvrv-p969 (high), fixed in 0.3.0-rc1: RAGAS has an Arbitrary File Read vulnerability |
Observability & Operations
not assessableNot rated: only 5 of 16 checks could reach a conclusion here; the rest do not apply to this repository or could not read it.
Fit with the Acquirer
not assessableNot rated: structurally not assessable from a repository.
Read more, or read your own
The full report on vibrantlabsai/ragas: every dimension, the questions for the room, and the audit log.
Survey a repository: free on public repositories. Private code is read on your own machine by a collector whose source you can read first.