Configuring Validation

We offer code validation for Python and SQL using static code analysis tools. To configure code validation, add a "code" validator to the config. For Python, specify code_lang as python. For SQL, choose code_lang from one of the supported dialects, sqlite, tsql, bigquery, mysql, postgres. If your dialect is not supported or mixed, use ansi as a representation of vanilla SQL.

For example,

post_processors: 
    - validator: code 
      settings: 
        code_lang: python 
        code_columns: [code]

post_processors:
    - validator: code
      settings:
        code_lang: sqlite
        code_columns: [sql_context, sql]

Results

For Python validation, we add the following columns to your dataset:

• <CODE_COL>_is_valid: boolean

• <CODE_COL>_pylint_score: a Pylint score between 0 and 10 indicating code quality (fewer errors, warnings)

• <CODE_COL>_pylint_severity: fatal, error , warning or None.

• <CODE_COL>_pylint_messages: Pylint error messages, sorted by severity.

Example: Python Validator Result

For SQL validation, we add the following columns to your dataset:

• <CODE_COL>_is_valid: boolean

• <CODE_COL>_validator_messages: Error messages explaining why the SQL is invaild.

Example: SQL Validator Result