Integration with Amazon Security Lake - Amazon Detective

The content from the Amazon Detective Administration Guide is now consolidated into the Amazon Detective User Guide. Amazon Detective Administration Guide will reach its end of standard support on May 08, 2024.

Integration with Amazon Security Lake

Amazon Security Lake is a fully managed security data lake service. You can use Security Lake to automatically centralize security data from AWS environments, SaaS providers, on-premises sources, cloud sources, and third-party sources into a purpose-built data lake that's stored in your AWS account. Security Lake helps you analyze security data, so you can get a more complete understanding of your security posture across your entire organization. With Security Lake, you can also improve the protection of your workloads, applications, and data.

Amazon Detective integrates with Amazon Security Lake, which means that you can query and retrieve the raw log data stored by Security Lake.

Using this integration, you can collect logs and events from the following sources which Security Lake natively supports.

  • AWS CloudTrail management events version 1.0

  • Amazon Virtual Private Cloud (Amazon VPC) Flow Logs version 1.0

For details on how Security Lake automatically converts logs and events that come from natively-supported AWS services to the OCSF schema, see the Amazon Security Lake User Guide.

After you integrate Detective with Security Lake, Detective begins pulling raw logs from Security Lake related to AWS CloudTrail management events and Amazon VPC Flow Logs. For more details, see Querying raw logs.

To integrate Detective with Security Lake, complete the following steps:

  1. Before you begin

    Use an Organizations management account to designate a delegated Security Lake administrator for your organization. Make sure that Security Lake is enabled and verify that Security Lake is collecting logs and events from AWS CloudTrail management events and Amazon Virtual Private Cloud (Amazon VPC) Flow Logs.

    In alignment with the Security Reference Architecture, Detective recommends using a Log Archive account and defer from using a Security Tooling account for the Security Lake deployment.

  2. Create a Security Lake subscriber

    To consume logs and events from Amazon Security Lake, you must be a Security Lake subscriber. Follow these steps to grant query access to a Detective account administrator.

  3. Add the required AWS Identity and Access Management (IAM) permissions to your IAM identity.

  4. Accept the Resource Share ARN invitation and enable the integration

Use the AWS CloudFormation template to set up the parameters required to create and manage query access for Security Lake subscribers. For the detailed steps to create a stack, see Create a stack using the AWS CloudFormation template. After you finish creating the stack, enable the integration.

For a demonstration of how to integrate Amazon Detective with Amazon Security Lake using the Detective console, watch the following video:

Before you begin

Security Lake integrates with AWS Organizations to manage log collection across multiple accounts in an organization. To use Security Lake for an organization, your AWS Organizations management account must first designate a delegated Security Lake administrator for your organization. The delegated Security Lake administrator must then enable Security Lake, and enable log and event collection for member accounts in the organization.

Before you integrate Security Lake, with Detective, make sure that Security Lake is enabled for the Security Lake administrator account. For the detailed steps on how to enable Security Lake, see Getting Started in the Amazon Security Lake User Guide.

Also, verify that Security Lake is collecting logs and events from AWS CloudTrail management events and Amazon Virtual Private Cloud (Amazon VPC) Flow Logs. For more details about log collection in Security Lake, see Collecting data from AWS services in the Amazon Security Lake User Guide.

Step 1: Create a Security Lake subscriber

To consume logs and events from Amazon Security Lake, you must be a Security Lake subscriber. A Subscriber can query and access the data that Security Lake collects. A subscriber with query access can query AWS Lake Formation tables directly in an Amazon Simple Storage Service (Amazon S3) bucket by using services such as Amazon Athena. To become a subscriber, the Security Lake administrator has to provide you with subscriber access that lets you query the data lake. For information about how the administrator does this, see Creating a subscriber with query access in the Amazon Security Lake User Guide.

Follow these steps to grant query access to a Detective account administrator.

To create a Detective subscriber in Security Lake
  1. Open the Detective console at https://console.aws.amazon.com/detective/.

  2. In the navigation pane, choose Integrations.

  3. In the Security Lake subscriber pane, note the Account ID and External ID values.

    Ask the Security Lake administrator to use these IDs to:

    • To create a Detective subscriber for you in Security Lake.

    • To configure the subscriber to have query access.

    • To make sure that the Security Lake query subscriber is created with Lake Formation permissions, select Lake Formation as the Data Access Method in the Security Lake console.

    When the Security Lake administrator creates a subscriber for you, Security Lake generates an Amazon Resource Share ARN for you. Ask the administrator to send this ARN to you.

  4. Enter the Resource Share ARN that is provided by the Security Lake administrator in the Security Lake subscriber pane.

  5. After you receive the Resource Share ARN from the Security Lake Administrator, enter the ARN in the Resource Share ARN box in the Security Lake subscriber pane.

Step 2: Add the required IAM permissions to your account

To enable Detective integration with Security Lake, you must attach the following AWS Identity and Access Management (IAM) permissions policy to your IAM identity.

Attach the following inline policies to the role. Replace athena-results-bucket with your Amazon S3 bucket name if you want to use your own Amazon S3 bucket to store the Athena query results. If you want Detective to automatically generate an Amazon S3 bucket to store the Athena query result, remove the entire S3ObjectPermissions from the IAM policy.

If you do not have the required permissions to attach this policy to your IAM identity, contact your AWS administrator. If you have the required permissions but an issue occurs, see Troubleshooting general IAM issues in the IAM User Guide.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListAllMyBuckets" ], "Resource": "*" }, { "Sid": "S3ObjectPermissions", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::<athena-results-bucket>", "arn:aws:s3:::<athena-results-bucket>/*" ] }, { "Effect": "Allow", "Action": [ "glue:GetDatabases", "glue:GetPartitions", "glue:GetTable", "glue:GetTables" ], "Resource": [ "arn:aws:glue:*:<ACCOUNT ID>:database/amazon_security_lake*", "arn:aws:glue:*:<ACCOUNT ID>:table/amazon_security_lake*/amazon_security_lake*", "arn:aws:glue:*:<ACCOUNT ID>:catalog" ] }, { "Effect": "Allow", "Action": [ "athena:BatchGetQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:GetQueryRuntimeStatistics", "athena:GetWorkGroup", "athena:ListQueryExecutions", "athena:StartQueryExecution", "athena:StopQueryExecution", "lakeformation:GetDataAccess" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ssm:GetParametersByPath" ], "Resource": [ "arn:aws:ssm:*:<ACCOUNT ID>:parameter/Detective/SLI/ResourceShareArn", "arn:aws:ssm:*:<ACCOUNT ID>:parameter/Detective/SLI/S3Bucket", "arn:aws:ssm:*:<ACCOUNT ID>:parameter/Detective/SLI/TableNames", "arn:aws:ssm:*:<ACCOUNT ID>:parameter/Detective/SLI/DatabaseName", "arn:aws:ssm:*:<ACCOUNT ID>:parameter/Detective/SLI/StackId" ] }, { "Effect": "Allow", "Action": [ "cloudformation:GetTemplateSummary", "iam:ListRoles" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "organizations:ListDelegatedAdministrators" ], "Resource": "*", "Condition": { "StringEquals": { "organizations:ServicePrincipal": [ "securitylake.amazonaws.com" ] } } } ] }

Step 3: Accept the Resource Share ARN invitation and enable the integration

To access raw data logs from Security Lake, you must accept a Resource Share invitation from the Security Lake account that was created by the Security Lake administrator. You also need AWS Lake Formation permissions to set up cross-account table sharing. In addition, you must create an Amazon Simple Storage Service (Amazon S3) bucket that can receive raw query logs.

In this next step, you’ll use an AWS CloudFormation template to create a stack for: accepting the Resource Share ARN invitation, create required AWS Glue crawler resources, and grant AWS Lake Formation administrator permissions.

To create an AWS CloudFormation stack
  1. Create a new CloudFormation stack using the CloudFormation template. For more details, see Creating a stack using the AWS CloudFormation template.

  2. After you finish creating the stack, choose Enable integration.

Creating a stack using the AWS CloudFormation template

Detective provides an AWS CloudFormation template, which you can use to set up the parameters required to create and manage query access for Security Lake subscribers.

Step 1: Create an AWS CloudFormation service role

You must create an AWS CloudFormation service role to create a stack using the AWS CloudFormation template. If you do not have the required permissions to create a service role, contact the administrator of the Detective administrator account. For more information about the AWS CloudFormation service role, see AWS CloudFormation service role.

  1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane of the IAM console, choose Roles, and then choose Create role.

  3. For Select trusted entity, choose AWS service.

  4. Choose AWS CloudFormation. Then, choose Next.

  5. Enter a name for the role. For example, CFN-DetectiveSecurityLakeIntegration.

  6. Attach the following inline policies to the role. Replace <Account ID> with your AWS Account ID.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "CloudFormationPermission", "Effect": "Allow", "Action": [ "cloudformation:CreateChangeSet" ], "Resource": [ "arn:aws:cloudformation:*:aws:transform/*" ] }, { "Sid": "IamPermissions", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:DetachRolePolicy", "iam:UpdateAssumeRolePolicy", "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:CreatePolicy", "iam:DeletePolicy", "iam:PassRole", "iam:GetRole", "iam:GetRolePolicy" ], "Resource": [ "arn:aws:iam::<ACCOUNT ID>:role/*", "arn:aws:iam::<ACCOUNT ID>:policy/*" ] }, { "Sid": "S3Permissions", "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket*", "s3:PutBucket*", "s3:GetBucket*", "s3:GetObject", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration" ], "Resource": [ "arn:aws:s3:::*" ] }, { "Sid": "LambdaPermissions", "Effect": "Allow", "Action": [ "lambda:CreateFunction", "lambda:DeleteFunction", "lambda:GetFunction", "lambda:TagResource", "lambda:InvokeFunction" ], "Resource": [ "arn:aws:lambda:*:<ACCOUNT ID>:function:*" ] }, { "Sid": "CloudwatchPermissions", "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:DeleteLogGroup", "logs:DescribeLogGroups" ], "Resource": "arn:aws:logs:*:<ACCOUNT ID>:log-group:*" }, { "Sid": "KmsPermission", "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": "arn:aws:kms:*:<ACCOUNT ID>:key/*" } ] }

Step 2: Add permissions to your IAM principal.

You’ll need the following permissions to create a stack using the CloudFormation service role that you created in the preceding step. Add the following IAM policy to the IAM principal that you plan to use to pass the CloudFormation service role. You will assume this IAM principal to create the stack. If you do not have the required permissions to add the IAM policy, contact the administrator of the Detective administrator account.

Note

In the following policy, CFN-DetectiveSecurityLakeIntegration used in this policy refers to the role that you created in the previous Creating an AWS CloudFormation service role step. Change it to the role name that you entered in the preceding step if it’s different.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "PassRole", "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": "arn:aws:iam::<ACCOUNT ID>:role/CFN-DetectiveSecurityLakeIntegration" }, { "Sid": "RestrictCloudFormationAccess", "Effect": "Allow", "Action": [ "cloudformation:CreateStack", "cloudformation:DeleteStack", "cloudformation:UpdateStack" ], "Resource": "arn:aws:cloudformation:*:<ACCOUNT ID>:stack/*", "Condition": { "StringEquals": { "cloudformation:RoleArn": [ "arn:aws:iam::<ACCOUNT ID>:role/CFN-DetectiveSecurityLakeIntegration" ] } } }, { "Sid": "CloudformationDescribeStack", "Effect": "Allow", "Action": [ "cloudformation:DescribeStacks", "cloudformation:DescribeStackEvents", "cloudformation:GetStackPolicy" ], "Resource": "arn:aws:cloudformation:*:<ACCOUNT ID>:stack/*" }, { "Sid": "CloudformationListStacks", "Effect": "Allow", "Action": [ "cloudformation:ListStacks" ], "Resource": "*" }, { "Sid": "CloudWatchPermissions", "Effect": "Allow", "Action": [ "logs:GetLogEvents" ], "Resource": "arn:aws:logs:*:<ACCOUNT ID>:log-group:*" } ] }
Step 3: Specify custom values in the AWS CloudFormation console
  1. Go to the AWS CloudFormation console from Detective.

  2. (Optional) Enter a Stack name. The stack name is auto-filled. You can change the stack name to a name that does not conflict with existing stack names.

  3. Enter the following Parameters.

    • AthenaResultsBucket – If you don't enter values, this template generates an Amazon S3 bucket. If you want to use your own bucket, enter a bucket name to store the Athena query results. If you use your own bucket, make sure that the bucket is in the same Region as the Resource Share ARN. If you use your own bucket, make sure the LakeFormationPrincipals you choose have permissions to write objects to and read objects from the bucket. For more details about bucket permissions, see Query results and recent queries in the Amazon Athena User Guide.

    • DTRegion – This field is pre-filled. Do not change the values in this field.

    • LakeFormationPrincipals – Enter the ARN of the IAM principals (for example, IAM role ARN) that you want to grant access to use the Security Lake integration, separated by commas. These could be your security analysts and security engineers that use Detective.

      You can only use the IAM principals that you previously attached the IAM permissions to in step [Step 2: Add the required IAM permissions to your account].

    • ResourceShareARN – This field is pre-filled. Do not change the values in this field.

  4. Permissions

    IAM role – Select the role that you created in the Creating an AWS CloudFormation Service Role step. Optionally, you can keep it blank if your current IAM role has all the required permissions in the Creating an AWS CloudFormation Service Role step.

  5. Review and check all the I Acknowledge boxes and then click the Create stack button. For more details, review the following IAM resources that will be created.

* ResourceShareAcceptorCustomResourceFunction - ResourceShareAcceptorLambdaRole - ResourceShareAcceptorLogsAccessPolicy * SsmParametersCustomResourceFunction - SsmParametersLambdaRole - SsmParametersLogsAccessPolicy * GlueDatabaseCustomResourceFunction - GlueDatabaseLambdaRole - GlueDatabaseLogsAccessPolicy * GlueTablesCustomResourceFunction - GlueTablesLambdaRole - GlueTablesLogsAccessPolicy

Step 4: Add Amazon S3 bucket policy to IAM principals in LakeFormationPrincipals

(Optional) If you let this template generate an AthenaResultsBucket for you, you must attach the following policy to the IAM principals in LakeFormationPrincipals.

{ "Sid": "S3ObjectPermissions", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::<athena-results-bucket>", "arn:aws:s3:::<athena-results-bucket>/*" ] }

Replace athena-results-bucket with the AthenaResultsBucket name. The AthenaResultsBucket can be found on the AWS CloudFormation console:

  1. Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation.

  2. Click on your Stack.

  3. Click the Resources tab.

  4. Search for the logical ID AthenaResultsBucket and copy its physical ID.

Deleting a CloudFormation stack

If you do not delete the existing stack, new stack creation in the same Region will fail. You can delete a CloudFormation stack by using the CloudFormation console or use the AWS CLI.

To delete the AWS CloudFormation stack (Console)
  1. Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation.

  2. On the Stacks page in the CloudFormation console, select the stack that you want to delete. The stack must be currently running.

  3. In the stack details pane, choose Delete.

  4. Select Delete stack when prompted.

    Note

    The stack deletion operation can't be stopped once the stack deletion has begun. The stack proceeds to the DELETE_IN_PROGRESS state.

After the stack deletion is complete, the stack will be in the DELETE_COMPLETE state.

Troubleshooting stack deletion errors

If you are seeing a permission error with the message Failed to delete stack after clicking the Delete button, your IAM role doesn't have CloudFormation permission to delete a stack. Contact your account administrator to delete the stack.

To delete the CloudFormation stack (AWS CLI)

Enter the following command in the AWS CLI interface:

aws cloudformation delete-stack --stack-name your-stack-name --role-arn arn:aws:iam::<ACCOUNT ID>:role/CFN-DetectiveSecurityLakeIntegration

CFN-DetectiveSecurityLakeIntegration is the service role that you created in the Creating an AWS CloudFormation Service Role step.

Changing the integration configuration

If you want to change any of the parameters that you used to integrate Detective with Security Lake, you can edit them, and then enable the integration again. You can edit the AWS CloudFormation template to re-enable this integration for the following scenarios:

  • To update the Security Lake subscription, you can either create a new subscriber, or the Security Lake administrator can update the data source for the existing subscription.

  • To specify a different Amazon S3 bucket to store the raw query logs.

  • To specify different Lake Formation principals.

When you re-enable Detective integration with Security Lake, you can edit the Resource Share ARN, and view the IAM permissions. To edit the IAM permissions, you can go to the IAM console from Detective. You can also edit the values you previously entered in the AWS CloudFormation template. You must delete the existing CloudFormation stack and re-create it to re-enable the integration.

To re-enable Detective integration with Security Lake
  1. Open the Detective console at https://console.aws.amazon.com/detective/.

  2. In the navigation pane, choose Integrations.

  3. You can edit the integration using either of these steps:

    • In the Security Lake pane, choose Edit.

    • In the Security Lake pane, choose View. In the view page, choose Edit.

  4. Enter a new Resource Share ARN, to access the data sources in a Region.

  5. View the current IAM permissions, and go to the IAM console, if you want to edit the IAM permissions.

  6. Edit the values in the CloudFormation template.

    1. Delete the existing stack first, before creating a new stack. If you do not delete the existing stack and you try to create a new stack in the same Region, your request fails. For more details, see Deleting a CloudFormation stack.

    1. Create a new CloudFormation stack. For more details, see Creating a stack using the AWS CloudFormation template.

  7. Choose Enable integration.

Disabling the integration

If you disable Detective integration with Security Lake, you can no longer query log and event data from Security Lake.

To disable Detective integration with Security Lake
  1. Open the Detective console at https://console.aws.amazon.com/detective/.

  2. In the navigation pane, choose Integrations.

  3. Delete the existing stack. For more details, see Deleting a CloudFormation stack.

  4. In the Disable Security Lake integration pane, choose Disable.

Supported AWS Regions

You can integrate Detective with Security Lake in the following AWS Regions.

Region Name Region Endpoint Protocol;
US East (Ohio) us-east-2

securitylake.us-east-2.amazonaws.com

HTTPS
US East (N. Virginia) us-east-1

securitylake.us-east-1.amazonaws.com

HTTPS
US West (N. California) us-west-1

securitylake.us-west-1.amazonaws.com

HTTPS
US West (Oregon) us-west-2

securitylake.us-west-2.amazonaws.com

HTTPS
Asia Pacific (Mumbai) ap-south-1

securitylake.ap-south-1.amazonaws.com

HTTPS
Asia Pacific (Seoul) ap-northeast-2

securitylake.ap-northeast-2.amazonaws.com

HTTPS
Asia Pacific (Singapore) ap-southeast-1

securitylake.ap-southeast-1.amazonaws.com

HTTPS
Asia Pacific (Sydney) ap-southeast-2

securitylake.ap-southeast-2.amazonaws.com

HTTPS
Asia Pacific (Tokyo) ap-northeast-1

securitylake.ap-northeast-1.amazonaws.com

HTTPS
Canada (Central) ca-central-1

securitylake.ca-central-1.amazonaws.com

HTTPS
Europe (Frankfurt) eu-central-1

securitylake.eu-central-1.amazonaws.com

HTTPS
Europe (Ireland) eu-west-1

securitylake.eu-west-1.amazonaws.com

HTTPS
Europe (London) eu-west-2

securitylake.eu-west-2.amazonaws.com

HTTPS
Europe (Paris) eu-west-3

securitylake.eu-west-3.amazonaws.com

HTTPS
Europe (Stockholm) eu-north-1

securitylake.eu-north-1.amazonaws.com

HTTPS
South America (São Paulo) sa-east-1

securitylake.sa-east-1.amazonaws.com

HTTPS

Querying raw logs in Detective

After you integrate Detective with Security Lake, Detective begins pulling raw logs from Security Lake related to AWS CloudTrail management events and Amazon Virtual Private Cloud (Amazon VPC) Flow Logs.

Note

There are no additional charges to query raw logs in Detective. Usage charges for other AWS Services, including Amazon Athena, still apply at published rates.

AWS CloudTrail management events are available for the following profiles:

  • AWS account

  • AWS user

  • AWS role

  • AWS role Session

  • Amazon EC2 instance

  • Amazon S3 bucket

  • IP address

Amazon VPC FLow Logs are available for the following profiles:

  • Amazon EC2 instance

  • Kubernetes pod

For a demonstration of how to use Amazon Detective with Amazon Security Lake using the Detective console, watch the following video:

To query raw logs for an AWS account
  1. Open the Detective console at https://console.aws.amazon.com/detective/.

  2. In the navigation pane, choose Search and search for an AWS account.

  3. In the Overall API call volume section, choose display details for scope time.

  4. From here, you can start to Query raw logs.


                In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.

In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.


                In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.

From the Query raw logs table, you can Cancel query request, See results in Amazon Athena, and Download results as a comma-separated values (.csv) file.

If you see logs in Detective, but the query returned no results, it could happen because of the following reasons.

  • Raw logs may become available in Detective before showing up in Security Lake log tables. Try again later.

  • Logs may be missing from Security Lake. If you waited for an extended period of time, it indicates that logs are missing from Security Lake. Contact your Security Lake administrator to resolve the issue.

Query raw logs for an AWS role

If you want to understand the activity of an AWS role in a new geolocation, you can do so within the Detective console.

To query raw logs for an AWS role
  1. Open the Detective console at https://console.aws.amazon.com/detective/.

  2. From the Detective Summary page Newly observed geolocations section, note down the AWS role.

  3. In the navigation pane, choose Search and search for the AWS role.

  4. For the AWS role, expand the resource to display the specific API calls that were issued from that IP address by that resource.

  5. Choose the magnifier icon next to the API call that you want to investigate to open the Raw log preview table.

    
                            In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.

In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.

From the Query raw logs table, you can Cancel query request, See results in Amazon Athena, and Download results as a comma-separated values (.csv) file.

Query raw logs for an Amazon EC2 instance

  1. Open the Detective console at https://console.aws.amazon.com/detective/.

  2. In the navigation pane, choose Search and search for an Amazon EC2 instance.

  3. In the Overall VPC Flow volume section, choose the magnifier icon next to the API call that you want to investigate to open the Raw log preview table.

  4. From here, you can start to Query raw logs.

    
                            In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.

In the Raw log preview table, you can view the logs and events retrieved by querying data from Security Lake. For more details about the raw event logs, you can view the data displayed in Amazon Athena.

From the Query raw logs table, you can Cancel query request, See results in Amazon Athena, and Download results as a comma-separated values (.csv) file.