Using Git version control systems in AWS Glue

Integrate with Git version control systems, such as AWS CodeCommit, GitHub, GitLab, and Bitbucket

Edit AWS Glue jobs in AWS Glue Studio whether you use visual jobs or script jobs and sync them to a repository

Parameterize sources and targets in jobs

Pull jobs from a repository and edit them in AWS Glue Studio

Test jobs by pulling from branches and/or pushing to branches utilizing multi-branch workflows in AWS Glue Studio

Download files from a repository and upload jobs into AWS Glue Studio for cross-account job creation

Use your automation tool of choice (for example, Jenkins, AWS CodeDeploy, etc.)

AWSGlueServiceRole

AWSGlueConsoleFullAccess

glue:UpdateJobFromSourceControl — to be able to update AWS Glue with a job present in a version control system

glue:UpdateSourceControlFromJob — to be able to update the version control system with a job stored in AWS Glue

s3:GetObject — to be able to retrieve the script for the job while pushing to version control system

s3:PutObject — to be able to update the script when pulling a job from a source control system

a repository that has already been created by your administrator

a branch in the repository

a personal access token (for Bitbucket, this is the Repository Access Token)

the username of the repository owner

set permissions in the repository to allow AWS Glue Studio to read and write to the repository

you need a folder and a JSON file of the job definition that matches the job name

For example, see the job definition below. The branch in your repository should contain a path my-visual-job/my-visual-job.json where both the folder and the JSON file match the job name

{
  "name" : "my-visual-job",
  "description" : "",
  "role" : "arn:aws:iam::aws_account_id:role/Rolename",
  "command" : {
    "name" : "glueetl",
    "scriptLocation" : "s3://foldername/scripts/my-visual-job.py",
    "pythonVersion" : "3"
  },
  "codeGenConfigurationNodes" : "{\"node-nodeID\":{\"S3CsvSource\":{\"AdditionalOptions\":{\"EnableSamplePath\":false,\"SamplePath\":\"s3://notebook-test-input/netflix_titles.csv\"},\"Escaper\":\"\",\"Exclusions\":[],\"Name\":\"Amazon S3\",\"OptimizePerformance\":false,\"OutputSchemas\":[{\"Columns\":[{\"Name\":\"show_id\",\"Type\":\"string\"},{\"Name\":\"type\",\"Type\":\"string\"},{\"Name\":\"title\",\"Type\":\"choice\"},{\"Name\":\"director\",\"Type\":\"string\"},{\"Name\":\"cast\",\"Type\":\"string\"},{\"Name\":\"country\",\"Type\":\"string\"},{\"Name\":\"date_added\",\"Type\":\"string\"},{\"Name\":\"release_year\",\"Type\":\"bigint\"},{\"Name\":\"rating\",\"Type\":\"string\"},{\"Name\":\"duration\",\"Type\":\"string\"},{\"Name\":\"listed_in\",\"Type\":\"string\"},{\"Name\":\"description\",\"Type\":\"string\"}]}],\"Paths\":[\"s3://dalamgir-notebook-test-input/netflix_titles.csv\"],\"QuoteChar\":\"quote\",\"Recurse\":true,\"Separator\":\"comma\",\"WithHeader\":true}}}"
}            
          

you need a folder, a JSON file of the job definition, and the script

the folder and JSON file should match the job name. The script name needs to match the scriptLocation in the job definition along with the file extension

For example, in the job definition below, the branch in your repository should contain a path my-script-job/my-script-job.json and my-script-job/my-script-job.py. The script name should match the name in the scriptLocation including the extension of the script

{
  "name" : "my-script-job",
  "description" : "",
  "role" : "arn:aws:iam::aws_account_id:role/Rolename",
  "command" : {
    "name" : "glueetl",
    "scriptLocation" : "s3://foldername/scripts/my-script-job.py",
    "pythonVersion" : "3"
  }
}            
          

AWS Glue currently does not support pushing/pulling from GitLab-Groups.