Setting up a log system with AWS OpenSearch (P1)

byGink▻ Tue, 30 Apr 2024

Requirements

Commonly, basic needs for a log system include:

It's easy to operate and maintain, and flexible to expand later depending on the data size.
Able to receive log data from many sources and many different data types.
Ability to assign users with different permissions to access and search log data.

The first 2 points can be solved by using AWS OpenSearch. As it's a managed service by AWS, similar to ElasticSearch but easier to operate and scale up. With OpenSearch, we can combine with log shippers like Filebeat, Fluent-bit, LogStash to aggregate log data from any sources, including server like EC2, GCE, Azure VM... or even container orchestration like GKE, EKS...

The last point, to be able to assign different permissions. Actually OpenSearch has it own internal user system, but the operator must create user account manually and that's not cool at all. Fortunately, we can combine OpenSearch with AWS Cognito for multiple authentication options like Google, Facebook, OpenID, etc.

So let's see how to do it?

Step by Step

1. IAM: Create 2 roles with custom trust policy to interact with OpenSearch

Go to IAM > Roles to create 2 new roles with trust policy and permission as following. We will need these roles to identify our users in OpenSearch.

log-center-master-user-role
log-center-limited-user-role

Trust policy: (the identity-arn-id will be updated later when we create the identity pool in Cognito)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "cognito-identity.amazonaws.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "cognito-identity.amazonaws.com:aud": "identity-arn-id"
                },
                "ForAnyValue:StringLike": {
                    "cognito-identity.amazonaws.com:amr": "authenticated"
                }
            }
        }
    ]
}

Permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "es:ESHttpGet",
            "Resource": "*"
        }
    ]
}

2. Cognito: Create user pool and identity pool to access OpenSearch

2.1. User pool

This is the pool of users can access to OpenSeaarch later. We'll create some groups of users for it.

Amazon Cognito > User pools > Create user pool

Authentication providers: There are a few options but it's better to use Email for authentication.
Multi-factor authentication: May disable MFA (No MFA) for now, we can enable it later.
User account recovery: Can disable Self-service account recovery also. This will not allow our users reset password, only Cognito admin can do it.
Self-service sign-up: Can disable Self-registration also, we gonna create users via Google authentication.
Required attributes: Email and name are obligated.
Message delivery: Let's choose Send email with Cognito.
User pool name: type in the name for this pool.
Hosted authentication pages:
- Should enable to configure domain name.
- If you want to create a custom domain name, remember to add the SSL certificate for it in us-east-1 region first.
Initial app client: When creating user pool via AWS Console, we have to create the client also. Can create any but we gonna delete it later, when connect this pool to OpenSearch.

2.2. Identity pool

OpenSearch will use this pool to identify which user is authenticated. At first, we will create a pool for Guest access, after connect this pool to OpenSearch, it will be disabled automatically and change to Authenticated access.

Then we will specify the user pool created above as the source of users (provider) for this identity pool.

Amazon Cognito > Identity pools > Create identity pool

Guest role: For now, can go to IAM again to create a role without any permission for it. For example: log-center-guest-user-role

After finishing these steps, we should have 3 roles in IAM, 1 user pool and 1 identity pool in Cognito. Great, let's get to the next step to create OpenSearch domain.

3. OpenSearch: Create the domain

This should be the storage for our log system. It should be scalable because log data is commonly huge. For now AWS provide 2 options: cluster and serverless. We will use cluster because the serverless option is quite limited in feature. At this moment, Cognito can't work with serverless OpenSearch, though it might be changed later.

Amazon OpenSearch Service > Domains > Create domain

I recommend to use Standard create for more configurable options. Some following options (for a cheaper price) are only available here.
- 1-AZ
- Domain without standby
- Use Elasticsearch instead of OpenSearch (not recommended, because it sticks with version 7.10 and not being updated anymore)
Custom endpoint:
- Similar to Cognito domain, must create the SSL certificate first.
- One thing different: the cert must be created in the same region with OpenSearch, not us-east-1.
Network: should enable Public access. We will use Fine-grained access control for authentication later.
Fine-grained access control:
- Should enable fine-grained access control.
- Master user: Remember about the master role created above? Go to IAM page, look for its ARN and put it here. Some thing similar to arn:aws:iam::xxx:role/log-center-master-user-role.
Amazon Cognito authentication:
- Should enable Amazon Cognito authentication.
- For Cognito user pool and identity pool, let's use the one we've just created before.
- The role name here shouldn't be changed, let's keep it CognitoAccessForAmazonOpenSearch as AWS Console recommends.
Access policy: It's better to keep Only use fine-grained access control. It can be modified later to limit IP access but we don't need it for now.

Things look good now, let's create it.

4. Update Cognito configuration

The process to create OpenSearch domain will take a while. It will also create a new Client in user pool and disable Guest access in identity pool inside Cognito. Wait for it to complete first then we will continue with Cognito.

4.1. Update identity pool ID in the roles

Remember about the identity-arn-id we've used in trust policy at the first step? Now we have the identity pool in Cognito already, let's update it back to those roles trust policy.

Go to Amazon Cognito > Identity pools, copy the Identity pool ID
Go to IAM > Roles, look for log-center-master-user-role, log-center-limited-user-role, open Trust relationships, click on Edit trust policy.
Replace identity-arn-id with the copied ID.

4.2. Update Cognito identity pool to forward users' role

Back to the Identity pool in Cognito again.
We will see that Guest access has been disabled by OpenSearch. It's ok.
Look at Identity providers, there should be one provider here, which is the user pool connected. Click on it.
Go to Role settings and Edit it with Choose role with preferred_role claim in tokens.
- Why we need this? Because we gonna create some groups of users in the user pool. Each group will have a different role and this preferred_role will forward it to OpenSearch.
- Role resolution: Deny request

4.3. Create some user groups in the user pool.

Go to Amazon Cognito > User pools and choose the created pool.
Open Groups tab, click on Create group.
Create a master group, for example master-users and connect it with log-center-master-user-role.
Back to Users tab, create some users and add them to this group.
Can repeat the same process for a new group for log-center-limited-user-role (optional). It's not required because we will leave this role for users who can login via Google Authentication later.

5. Verification

If you choose to use custom domain name (for both OpenSearch and Cognito), let's go to your DNS service to update the CNAME domain first.
Try to login OpenSearch dashboard with the user created in step 4.3.

If you can login via that user, congratulation! We're done here.

Let's check my next article on how to configure login OpenSearch with Google Authentication.