# AWS Cost Optimization Toolkit Catalog Update Process

This procedure documents how to update the local `catalog-publish` directory and publish changes to the CloudFront-backed S3 catalog.

## Current Catalog Pattern

The catalog is hosted through:

```text
CloudFront Distribution: E10DZO54CZYSZO
S3 Bucket: costopt-toolkit-catalog-catalogbucket-7infyuelt33l
Local publish folder: .\catalog-publish\
```

The S3 bucket is private and is served through CloudFront Origin Access Control. This is not S3 static website hosting.

## Local Folder Layout

Expected local structure:

```text
catalog-publish\
  404.html
  getting-started.html
  index.html
  manifest.json
  README.md

  docs\
    AWS Cost Optimization Remediation Modules Runbook.docx
    index.html

  remediation-tools\
    index.html
    *.yaml
    Security-Services-Audit-Configuration-Cost-Remediator.txt

  report-generator\
    index.html
    Cost-Optimization-Report-Template-V1.yaml

  scripts\
    publish-catalog.ps1
```

## Step 1 - Rebuild or Refresh the Local Catalog Folder

From:

```powershell
D:\MyTools\AWS-Cost-Optimization-Report-Generator\Remediation Tools
```

Run:

```powershell
.\catalog-publish\scripts\publish-catalog.ps1 `
  -SourceRoot "D:\MyTools\AWS-Cost-Optimization-Report-Generator" `
  -CatalogRoot ".\catalog-publish"
```

This refreshes the local catalog folder by copying current remediation YAML files and runbook files into the expected catalog structure.

## Step 2 - Manually Confirm Key Files Exist

```powershell
$Expected = @(
  ".\catalog-publish\getting-started.html",
  ".\catalog-publish\index.html",
  ".\catalog-publish\manifest.json",
  ".\catalog-publish\README.md",
  ".\catalog-publish\docs\index.html",
  ".\catalog-publish\docs\AWS Cost Optimization Remediation Modules Runbook.docx",
  ".\catalog-publish\remediation-tools\index.html",
  ".\catalog-publish\report-generator\index.html",
  ".\catalog-publish\report-generator\Cost-Optimization-Report-Template-V1.yaml"
)

$Expected | ForEach-Object {
    [PSCustomObject]@{
        Path   = $_
        Exists = Test-Path $_
    }
} | Format-Table -AutoSize
```

All rows should show `Exists = True`.

## Step 3 - Confirm Manifest Links Match Real Files

The catalog buttons are driven by `manifest.json`. If the manifest points to a filename that does not exist locally, the catalog will show a Download YAML button that returns a 404.

```powershell
$Manifest = Get-Content .\catalog-publish\manifest.json -Raw | ConvertFrom-Json

$Manifest.tools | ForEach-Object {
    $ExpectedPath = Join-Path ".\catalog-publish" $_.path

    [PSCustomObject]@{
        Tool   = $_.name
        File   = $_.path
        Exists = Test-Path $ExpectedPath
    }
} | Where-Object { $_.Exists -eq $false } | Format-Table -AutoSize
```

Expected result:

```text
No rows returned
```

If rows are returned, either copy the missing file into `catalog-publish\remediation-tools\`, or update the `file` and `path` values in `manifest.json` to match the actual filename.

### Known Filename Mismatches That Occurred

```text
CloudTrail Cost Review:
  Manifest originally expected:
    remediation-tools/CloudTrail-Cost-Review-Remediator.yaml
  Actual file:
    remediation-tools/CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml

KMS Customer Managed Key Review:
  Manifest originally expected:
    remediation-tools/KMS-Customer-Managed-Key-Review.yaml
  Actual file:
    remediation-tools/KMS-Customer-Managed-Key-Review-Remediator.yaml

Security Services Audit / Configuration Cost Review:
  Manifest expected:
    remediation-tools/Security-Services-Audit-Configuration-Cost-Remediator.txt
  Note:
    This is a .txt file, not a .yaml file. The publish process must copy it explicitly.
```

### PowerShell Fix for Known CloudTrail and KMS Mismatches

```powershell
$ManifestPath = ".\catalog-publish\manifest.json"
$Manifest = Get-Content $ManifestPath -Raw | ConvertFrom-Json

$Manifest.tools | Where-Object { $_.name -eq "CloudTrail Cost Review" } | ForEach-Object {
    $_.file = "CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml"
    $_.path = "remediation-tools/CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml"
}

$Manifest.tools | Where-Object { $_.name -eq "KMS Customer Managed Key Review" } | ForEach-Object {
    $_.file = "KMS-Customer-Managed-Key-Review-Remediator.yaml"
    $_.path = "remediation-tools/KMS-Customer-Managed-Key-Review-Remediator.yaml"
}

$Manifest | ConvertTo-Json -Depth 20 | Set-Content $ManifestPath -Encoding UTF8
```

### PowerShell Fix for the Security Services .txt File

```powershell
Copy-Item `
  ".\Security-Services-Audit-Configuration-Cost-Remediator.txt" `
  ".\catalog-publish\remediation-tools\Security-Services-Audit-Configuration-Cost-Remediator.txt" `
  -Force
```

## Step 4 - Confirm YAML Count

```powershell
Get-ChildItem .\catalog-publish\remediation-tools\*.yaml | Measure-Object
```

The count should be close to the number of YAML remediation modules expected in the catalog.

In the current toolkit, a healthy local catalog had:

```text
Count: 60
```

Also confirm the Security Services `.txt` file if it is listed in the manifest:

```powershell
Test-Path .\catalog-publish\remediation-tools\Security-Services-Audit-Configuration-Cost-Remediator.txt
```

## Step 5 - Confirm Folder Index Pages Exist

CloudFront with a private S3 origin does not automatically list folders or resolve folder paths to `index.html`.

These must exist:

```powershell
Test-Path .\catalog-publish\remediation-tools\index.html
Test-Path .\catalog-publish\report-generator\index.html
Test-Path .\catalog-publish\docs\index.html
```

Expected result for each:

```text
True
```

## Step 6 - Confirm Folder Links Use Explicit index.html

With this CloudFront/S3 private-origin pattern, links like `/remediation-tools/` can return 404 even if `remediation-tools/index.html` exists. Use explicit links.

Check the main catalog page:

```powershell
Select-String -Path .\catalog-publish\index.html -Pattern "remediation-tools"
```

Expected link:

```html
href="remediation-tools/index.html"
```

If needed, repair folder links:

```powershell
$IndexPath = ".\catalog-publish\index.html"

(Get-Content $IndexPath -Raw) `
  -replace 'href="remediation-tools/"', 'href="remediation-tools/index.html"' `
  -replace 'href="report-generator/"', 'href="report-generator/index.html"' `
  -replace 'href="docs/"', 'href="docs/index.html"' |
  Set-Content $IndexPath -Encoding UTF8
```


## Required Pre-Publish Gate - Manifest Normalization

This gate prevents a recurring issue where `manifest.json` points to an older or alternate filename even though the correct file exists in `catalog-publish\remediation-tools`.

### Symptom

One or more catalog buttons return a 404 even though the corresponding YAML exists locally.

The manifest validation command returns rows like:

```text
Tool                            File                                                     Exists
----                            ----                                                     ------
CloudTrail Cost Review          remediation-tools/CloudTrail-Cost-Review-Remediator.yaml  False
KMS Customer Managed Key Review remediation-tools/KMS-Customer-Managed-Key-Review.yaml    False
```

### Cause

`manifest.json` can drift from the actual filenames in the `catalog-publish` folder.

This can happen after:

- Extracting a newer or older catalog package.
- Replacing `manifest.json`.
- Manually copying YAML files.
- Renaming module files.
- Refreshing the catalog from a source folder that has slightly different filenames.
- Rebuilding catalog packages before applying local filename corrections.

### Required Fix Before Publishing

Run this normalization step before every S3 sync:

```powershell
$ManifestPath = ".\catalog-publish\manifest.json"
$Manifest = Get-Content $ManifestPath -Raw | ConvertFrom-Json

$Manifest.tools | Where-Object { $_.name -eq "CloudTrail Cost Review" } | ForEach-Object {
    $_.file = "CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml"
    $_.path = "remediation-tools/CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml"
}

$Manifest.tools | Where-Object { $_.name -eq "KMS Customer Managed Key Review" } | ForEach-Object {
    $_.file = "KMS-Customer-Managed-Key-Review-Remediator.yaml"
    $_.path = "remediation-tools/KMS-Customer-Managed-Key-Review-Remediator.yaml"
}

$Manifest | ConvertTo-Json -Depth 20 | Set-Content $ManifestPath -Encoding UTF8
```

### Required Validation After Normalization

Immediately re-run the manifest validation:

```powershell
$Manifest = Get-Content .\catalog-publish\manifest.json -Raw | ConvertFrom-Json

$Manifest.tools | ForEach-Object {
    $ExpectedPath = Join-Path ".\catalog-publish" $_.path

    [PSCustomObject]@{
        Tool   = $_.name
        File   = $_.path
        Exists = Test-Path $ExpectedPath
    }
} | Where-Object { $_.Exists -eq $false } | Format-Table -AutoSize
```

Expected result:

```text
No rows returned
```

Do not run `aws s3 sync --delete` until this validation returns no rows.

### Make the Fix Persistent in publish-catalog.ps1

To prevent the same issue from returning, append the normalization logic to the local publish script:

```powershell
$PublishScript = ".\catalog-publish\scripts\publish-catalog.ps1"

@'

Write-Host "Normalizing known manifest filename mappings..."
$ManifestPath = Join-Path $CatalogRoot "manifest.json"

if (Test-Path $ManifestPath) {
    $Manifest = Get-Content $ManifestPath -Raw | ConvertFrom-Json

    $Manifest.tools | Where-Object { $_.name -eq "CloudTrail Cost Review" } | ForEach-Object {
        $_.file = "CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml"
        $_.path = "remediation-tools/CloudTrail-CloudTrail-Lake-Cost-Review-Remediator.yaml"
    }

    $Manifest.tools | Where-Object { $_.name -eq "KMS Customer Managed Key Review" } | ForEach-Object {
        $_.file = "KMS-Customer-Managed-Key-Review-Remediator.yaml"
        $_.path = "remediation-tools/KMS-Customer-Managed-Key-Review-Remediator.yaml"
    }

    $Manifest | ConvertTo-Json -Depth 20 | Set-Content $ManifestPath -Encoding UTF8
}

'@ | Add-Content $PublishScript
```

Only append this block once. If it already exists in `publish-catalog.ps1`, do not add a duplicate copy.

### Updated Pre-Publish Order

Use this order before every catalog update:

1. Refresh or rebuild `catalog-publish`.
2. Confirm key files exist.
3. Confirm remediation YAML count is healthy.
4. Confirm `Security-Services-Audit-Configuration-Cost-Remediator.txt` exists.
5. Run the manifest normalization step.
6. Run manifest validation.
7. Confirm no missing files are returned.
8. Sync to S3 with `--delete`.
9. Invalidate CloudFront.
10. Hard refresh browser and test buttons.

## Step 7 - Publish to S3

```powershell
aws s3 sync .\catalog-publish\ s3://costopt-toolkit-catalog-catalogbucket-7infyuelt33l/ --delete
```

Important: `--delete` makes the S3 bucket match the local `catalog-publish` folder.

If the local folder is missing YAML files, they will be deleted from S3. This happened during testing when the local `remediation-tools` folder only contained `.keep`.

Always validate the local folder before syncing:

```powershell
Get-ChildItem .\catalog-publish\remediation-tools\*.yaml | Measure-Object
```

And always validate manifest paths before syncing:

```powershell
$Manifest = Get-Content .\catalog-publish\manifest.json -Raw | ConvertFrom-Json

$Manifest.tools | ForEach-Object {
    $ExpectedPath = Join-Path ".\catalog-publish" $_.path

    [PSCustomObject]@{
        Tool   = $_.name
        File   = $_.path
        Exists = Test-Path $ExpectedPath
    }
} | Where-Object { $_.Exists -eq $false } | Format-Table -AutoSize
```

## Step 8 - Invalidate CloudFront

CloudFront caches files. After uploading new catalog content to S3, invalidate CloudFront so users receive the latest version.

```powershell
aws cloudfront create-invalidation `
  --distribution-id E10DZO54CZYSZO `
  --paths "/*"
```

The command returns an invalidation ID and a status of `InProgress`.

## Step 9 - Check Invalidation Status

Replace the invalidation ID with the ID from the previous command.

```powershell
aws cloudfront get-invalidation `
  --distribution-id E10DZO54CZYSZO `
  --id <InvalidationId>
```

Expected final status:

```text
Completed
```

After invalidation completes, do a browser hard refresh with `Ctrl+F5`.

## Step 10 - Browser Validation

Open the CloudFront catalog URL and test:

```text
/
getting-started.html
README.md
manifest.json
docs/index.html
docs/AWS Cost Optimization Remediation Modules Runbook.docx
report-generator/index.html
report-generator/Cost-Optimization-Report-Template-V1.yaml
remediation-tools/index.html
```

Also test:

1. One or two `Download YAML` buttons.
2. The `Runbook DOCX` button.
3. The `Download Report Generator` button.
4. The `Remediation Tools Folder` button.
5. The `Getting Started` button.

## Troubleshooting

### Remediation Tools Folder Button Returns 404

Cause:

The link points to:

```text
/remediation-tools/
```

CloudFront/S3 private origin does not automatically resolve this to:

```text
/remediation-tools/index.html
```

Fix:

Update `index.html` to use:

```html
href="remediation-tools/index.html"
```

Then sync and invalidate CloudFront.

### Download YAML Button Returns 404

Cause:

The manifest path does not match an actual file in `catalog-publish`.

Fix:

Run the manifest validation check and correct `file` and `path` values in `manifest.json`.

### S3 Sync Deletes YAML Files

Cause:

The local `catalog-publish\remediation-tools\` folder is incomplete and `aws s3 sync --delete` removes objects from S3 that are not present locally.

Fix:

Repopulate local files, validate YAML count, validate manifest paths, then sync again.

### CloudFront Still Shows the Old Page

Cause:

CloudFront cached the old object.

Fix:

Run invalidation for `/*`, wait for `Completed`, then hard-refresh the browser.

### README or Manifest Loads but New Button Does Not Work

Cause:

The static page, manifest, or linked folder index is out of sync.

Fix:

Confirm these local files exist and were uploaded:

```text
index.html
manifest.json
remediation-tools/index.html
report-generator/index.html
docs/index.html
```

Then invalidate CloudFront.

## Notes

- Folder-style links such as `/remediation-tools/` do not automatically resolve to `/remediation-tools/index.html` with this CloudFront/S3 private-origin pattern.
- Use explicit links such as `remediation-tools/index.html`, `docs/index.html`, and `report-generator/index.html`.
- The S3 bucket remains private.
- CloudFront is the access layer.
- Cognito or another authentication layer can be added later.