Efficiency improvements on GitHub action "verify all rules"

[MasonMasoff]

Description:

Currently, the GitHub Actions workflow in this repository verifies all detection rules on every run, regardless of whether they have been modified. This can lead to longer CI times, especially as the number of rules grows.

Proposed Improvement:

We have implemented a solution in our fork that leverages git diff to identify and verify only the rules that have been changed in a given pull request or commit. This significantly reduces the time required for CI checks, as only the modified rules are validated rather than the entire set.

Benefits:

• Faster CI runs: Only changed rules are verified, reducing overall execution time.
• Scalability: As the repository grows, this approach will help keep CI times manageable.
• Resource efficiency: Less compute time and resources are used per workflow run.

[threat-punter]

Thanks @MasonMasoff. Can you share your example code? Cheers. @threat-punter

[MasonMasoff]

@threat-punter I seem to be unable to create a PR (after signing the CLA), Ill outline the changes here!

In __main__.py

@classmethod
    def verify_changed(cls):
        """Verify only rules that have been changed using Git.

        This checks Git for any modified .yaral files and only verifies those rules,
        rather than verifying all rules. This is useful for CI/CD workflows to avoid
        hitting rate limits.
        """
        import subprocess
        import os

        http_session = initialize_http_session()

        # Maintain lists of successful and failed YARA-L verification responses.
        verify_rule_successes = []
        verify_rule_errors = []

        # Get list of changed rule files using git
        try:
            # Detect if we're in a GitHub Actions environment
            github_event_name = os.environ.get("GITHUB_EVENT_NAME")
            is_pull_request = github_event_name == "pull_request"

            if is_pull_request:
                # For pull requests, compare with the base branch
                base_sha = os.environ.get("GITHUB_BASE_REF", "")
                if base_sha:
                    LOGGER.info("Running in PR context, comparing with base branch: %s", base_sha)
                    result = subprocess.run(
                        ["git", "diff", "--name-only", f"origin/{base_sha}...HEAD", "*.yaral"],
                        capture_output=True,
                        text=True,
                        cwd=ROOT_DIR,
                        check=True,
                    )
                else:
                    # Fallback to default
                    LOGGER.info("No base branch found, using default comparison")
                    result = subprocess.run(
                        ["git", "diff", "--name-only", "HEAD~1", "HEAD", "*.yaral"],
                        capture_output=True,
                        text=True,
                        cwd=ROOT_DIR,
                        check=True,
                    )
            else:
                # For pushes to main or other branches, compare with the previous commit
                LOGGER.info("Comparing with previous commit")
                result = subprocess.run(
                    ["git", "diff", "--name-only", "HEAD~1", "HEAD", "*.yaral"],
                    capture_output=True,
                    text=True,
                    cwd=ROOT_DIR,
                    check=True,
                )

            changed_files = result.stdout.strip().split("\n")

            # Filter out empty strings from the list
            changed_files = [f for f in changed_files if f]

            if not changed_files:
                LOGGER.info("No rule files have been changed.")
                return

            LOGGER.info("Found %s changed rule files: %s", len(changed_files), changed_files)

            # Convert relative paths to absolute paths
            changed_rule_files = [ROOT_DIR / file for file in changed_files if file.endswith(".yaral")]

            # Verify each changed rule
            for rule_file in changed_rule_files:
                if not rule_file.exists():
                    LOGGER.warning("Rule file %s does not exist, skipping verification", rule_file)
                    continue

                with open(rule_file, "r", encoding="utf-8") as f:
                    rule_text = f.read()

                response = verify_rule(http_session=http_session, rule_text=rule_text)

                if response.get("success") is True:
                    LOGGER.info(
                        "Rule verification succeeded for rule (%s). Response: %s",
                        rule_file,
                        response,
                    )
                    verify_rule_successes.append(rule_file)

                else:
                    verify_rule_errors.append({"rule": rule_file, "response": response})

            LOGGER.info("Rule verification succeeded for %s changed rules", len(verify_rule_successes))

            if verify_rule_errors:
                LOGGER.error("Rule verification failed for %s changed rules", len(verify_rule_errors))
                # Log each rule verification error before raising an exception
                for error in verify_rule_errors:
                    LOGGER.error(
                        "Rule verification failed for rule (%s). Response: %s",
                        error["rule"],
                        json.dumps(error["response"], indent=4),
                    )
                raise RuleVerificationError(f"Rule verification failed for {len(verify_rule_errors)} changed rules")

        except subprocess.CalledProcessError as e:
            LOGGER.error("Failed to get changed files using git: %s", e)
            LOGGER.error("Stdout: %s", e.stdout)
            LOGGER.error("Stderr: %s", e.stderr)
            raise

#### Insert after def verify_all(): definition ####

@rules.command(
    "verify-changed", short_help="Verify only changed rules are valid YARA-L rules."
)
def verify_changed():
    """Verify only changed rules are valid YARA-L rules."""
    LOGGER.info("Attempting to verify only changed rules")
    RuleOperations.verify_changed()

In Run Tests.yml

      # Remove Verify YARA-L rules
      # - name: Verify YARA-L rules
      #   # Verify that all rules are valid YARA-L 2.0 rules via Google SecOps' API
      #   run: python -m content_manager rules verify-all

      - name: Verify Changed YARA-L rules
        # Verify that changed rules are valid YARA-L 2.0 rules via Google SecOps' API
        run: python -m content_manager rules verify-changed

Let me know if you have any questions or modifications!

[barnabys-drew]

@threat-punter This significanltly lowered our action runtime. Great job @MasonMasoff !

[threat-punter]

Hey @MasonMasoff. Thanks for this contribution. I think it makes sense from an efficiency perspective. I wanted to let you know that this is on my radar and I'll give it my full attention as soon as I can. Cheers. @threat-punter

[threat-punter]

I'll take a look at this when I'm back from leave next week. Thanks again for opening this and sorry for the delay.

[MasonMasoff]

No worries at all! We've been testing it internally ,Drew and I are happy to meet and discuss some other changes as well!

[threat-punter]

I'm looking at moving Content Manager to its own GitHub repo before working on new features and enhancements. I'll leave this issue open in the meantime. Thanks again for opening this.