evidentlyai/evidently: technical due diligence, read from its repository

The median source file here is 62 lines. 8 files exceed 1,000 lines, the largest being 1,902. Size is not a defect on its own — some problems genuinely live in one place — but these are where merge conflicts, review fatigue and single-owner knowledge concentrate, and they are the first thing that slows an inheriting engineer down.

2,423 distinct 8-line blocks appear in two or more files. Copy-paste is the cheapest thing a generator does and among the more expensive things to inherit: a fix has to be applied everywhere the block landed, and nothing in the repository records where that is. Tests, fixtures and documentation examples are excluded — those repeat by design — so this is duplication in code that ships. Blocks are matched after normalising whitespace and only across different files.

The suite is configured to retry failing tests (ui/html-visual-testing/playwright.config.ts). A retry turns a flaky test green without saying why it failed, so a green build means the tests passed eventually. Ask how many retries a typical run needs, and which tests need them.

8 third-party CI actions are referenced by a tag or branch rather than a commit, so they run whatever that tag points to on the day. Tags have been moved to malicious commits in real supply-chain attacks, and a piped script is whatever the server returns that minute. Pinning to a commit is the fix, and it is small.

2 HTTP calls use a client with no timeout (Python `requests` without `timeout=`, or Go's default client). When a dependency slows down, each one holds a worker until the connection gives up, and enough of them take the service down with the dependency.

4 places run a command through a shell (`shell=True`, `os.system`, `exec` with a template) rather than passing arguments. Where any part comes from outside, it is command injection. Ask where each one's input comes from.

The newest was published on 2026-08-17. A project that publishes advisories is telling its users about its fixes, and the count and pace say how much security work it carries. Whether 1 of them are fixed in the code you ship could not be told from this history: its version is only known to be at least 0.7.22. Answer lives in: the release you run, set against each fixed version. The record is incomplete: NVD: 1 record names evidently with nothing tying it to this repository (CVE-2023-54276).

Nothing in the service initialises an error reporter (Sentry, Rollbar, Bugsnag, Honeybadger or an APM agent). An exception in production reaches a person only if someone is reading the logs, which usually means a customer reports it first.