azure-pipelines-validator

# Azure Pipelines Validator

Validates, lints, and security-scans Azure DevOps Pipeline configurations (`azure-pipelines.yml`, `azure-pipelines.yaml`). Runs four validation layers via a single orchestrator script.

## Basic Usage

```bash
# Full validation (all layers)
bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh azure-pipelines.yml
```

Layers executed in order:
0. **YAML lint** (yamllint) — formatting, indentation, trailing spaces
1. **Syntax validation** — schema, required fields, stages/jobs/steps hierarchy, task format, dependencies
2. **Best practices** — display names, task version pinning, pool image specificity, caching, timeouts
3. **Security scan** — hardcoded secrets/API keys/AWS/Azure credentials, dangerous script patterns, SSL bypasses, container `:latest` tags

### Common Options

```bash
# Targeted runs
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --syntax-only
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only

# Skip layers
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --skip-yaml-lint
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-best-practices
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-security

# Strict mode (fail on warnings)
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict
```

### Individual Scripts

```bash
python3 scripts/validate_syntax.py azure-pipelines.yml
python3 scripts/check_best_practices.py azure-pipelines.yml
python3 scripts/check_security.py azure-pipelines.yml
```

## Output and Error Recovery

```
════════════════════════════════════════════════════════════════════════════════
  Azure Pipelines Validator
════════════════════════════════════════════════════════════════════════════════

[1/3] Running syntax validation...
✓ Syntax validation passed

[2/3] Running best practices check...
SUGGESTIONS (2):
  INFO: Line 15: Job 'BuildJob' should have displayName [missing-displayname]
  💡 Add 'displayName: "Your Job Description"' to job 'BuildJob'

  WARNING: Line 25: Task 'Npm@1' could benefit from caching [missing-cache]
  💡 Add Cache@2 task to cache dependencies and speed up builds

[3/3] Running security scan...
MEDIUM SEVERITY (1):
  MEDIUM: Line 8: Container 'linux' uses ':latest' tag [container-latest-tag]
  🔒 Pin container images to specific versions or SHA digests
```

**When validation fails:**
1. Note the rule code in brackets (e.g., `[missing-displayname]`) — see `references/` for rule details.
2. Fix the flagged line and re-run the same layer (`--syntax-only`, `--security-only`, etc.) to iterate quickly.
3. Run full validation once all targeted fixes are applied to confirm no regressions.
4. For `MEDIUM`/`HIGH` security findings, do not merge until resolved; `INFO` findings are advisory.

## Common Scenarios

### New pipeline validation
```bash
bash scripts/validate_azure_pipelines.sh new-pipeline.yml
```

### Security audit before merge
```bash
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only --strict
```

### Pipeline optimisation
```bash
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices
```

### CI/CD self-validation in Azure Pipelines
```yaml
steps:
- script: |
    bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict
  displayName: 'Validate Pipeline Configuration'
```

## Auto-detection

Run without arguments to auto-detect `azure-pipelines*.yml` files in the current directory (up to 3 levels deep).

## Fetching Live Documentation

The validator performs **static analysis only**. For dynamic lookups (task versions, input parameters, feature docs), use:

```
# Context7 MCP
mcp__context7__resolve-library-id("azure-pipelines")
mcp__context7__get-library-docs(context7CompatibleLibraryID, topic="deployment")

# Or WebSearch / WebFetch
WebSearch("Azure Pipelines Docker@2 task documentation 2025")
WebFetch("https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/docker-v2")
```

## Requirements

- **Python 3.7+**, **Bash**
- **PyYAML** and **yamllint**: auto-installed in a persistent `.venv` if not available system-wide — no manual setup required.

```bash
# Optional manual install
pip3 install PyYAML yamllint
```

## Troubleshooting

| Problem | Fix |
|---|---|
| `ModuleNotFoundError: PyYAML` | `pip3 install PyYAML` |
| `Permission denied` | `chmod +x scripts/*.sh scripts/*.py` |
| Unexpected validation errors | Check `references/azure-pipelines-reference.md` or [Microsoft Learn](https://learn.microsoft.com/en-us/azure/devops/pipelines/) |

## Anti-Patterns

### NEVER skip validation on template files

- **WHY**: Templates called from the main pipeline can contain schema violations, missing parameters, or invalid task references that only surface at runtime. Validating only the entry-point file leaves template errors undetected until the pipeline actually runs.
- **BAD**: Validate only `azure-pipelines.yml` and skip all files under `templates/*.yml`.
- **GOOD**: Run the validator on every `.yml` file in the pipeline directory, including all templates.

### NEVER treat YAML lint warnings as informational

- **WHY**: Azure Pipelines YAML is whitespace-significant. Indentation errors and trailing spaces can silently restructure the pipeline — turning a step into a job property, or merging two separate blocks — without any obvious parse error.
- **BAD**: Ignore `yamllint` warnings such as "trailing spaces" or "wrong indentation" because the pipeline appears to run.
- **GOOD**: Fix all YAML formatting issues before submitting; a clean `yamllint` pass is a prerequisite for a trustworthy pipeline.

### NEVER use the strict mode flag on an unreviewed pipeline

- **WHY**: `--strict` fails on all warnings, which is the correct setting for a CI gate. Applied to a brand-new pipeline with dozens of warnings, it produces so much noise that engineers discard the output entirely and disable validation rather than fix the root causes.
- **BAD**: Run `--strict` on a new pipeline, see 30 warnings, and remove validation from the workflow because "it's too noisy."
- **GOOD**: Run without `--strict` first, fix critical errors, then warnings, then graduate to strict mode as a CI gate.

### NEVER skip security scanning before merging pipeline changes

- **WHY**: A passing syntax validation does not imply a secure pipeline. Security scanning catches hardcoded credentials, injection vectors, and insecure patterns that syntax checks are not designed to detect.
- **BAD**: Merge a pipeline change after syntax validation passes with no security check.
- **GOOD**: Always run `--security` as part of the validation workflow, treating `MEDIUM` and `HIGH` findings as merge blockers.

## References

- `references/azure-pipelines-reference.md` — full YAML syntax reference and rule definitions
- `assets/examples/basic-pipeline.yml` — simple CI pipeline
- `assets/examples/docker-build.yml` — Docker build and push
- `assets/examples/deployment-pipeline.yml` — multi-environment deployment with approval gates
- `assets/examples/multi-platform.yml` — multi-platform build matrix
- `assets/examples/template-example.yml` — reusable templates

```bash
# Test with a bundled example
bash scripts/validate_azure_pipelines.sh assets/examples/basic-pipeline.yml
```

## Extending the Skill

Add custom rules to the appropriate script:

- Syntax rules → `scripts/validate_syntax.py`
- Best practice rules → `scripts/check_best_practices.py`
- Security rules → `scripts/check_security.py`

```python
# Example custom best-practice rule in check_best_practices.py
def _check_custom_rule(self):
    for job in self._get_all_jobs():
        job_name = job.get('job') or job.get('deployment')
        if 'tags' not in pool:
            self.issues.append(BestPracticeIssue(
                'warning',
                self._get_line(job_name),
                f"Job '{job_name}' should specify agent tags",
                'custom-missing-tags',
                "Add 'tags' to pool to select appropriate agents"
            ))
```

---

**Note**: This skill validates pipeline configurations but does not execute pipelines. Use Azure DevOps Pipeline validation or Azure CLI to test actual pipeline execution.