The workflow

name: Evals
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: {distribution: temurin, java-version: 21}
      - run: ./gradlew evalRun
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
Advertisement

Fail on regression

Compare eval output vs baseline. Fail PR if pass rate drops significantly.

Advertisement

Cache dataset

Datasets rarely change per PR. Cache to speed builds.