Skip to content

feat: Use Codefresh provided storage for workflow logs #428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
56ec000
workflow logs configuraion
ilia-medvedev-codefresh Mar 20, 2025
81fcfb7
logs
ilia-medvedev-codefresh Mar 20, 2025
dbdf50a
remove Garage
ilia-medvedev-codefresh Mar 20, 2025
fed6bc9
set account id as required for worfklow log store
ilia-medvedev-codefresh Mar 23, 2025
cc7acb8
Add readme section
ilia-medvedev-codefresh Mar 24, 2025
6c29291
Add readme section
ilia-medvedev-codefresh Mar 24, 2025
913f2ae
Add readme section
ilia-medvedev-codefresh Mar 24, 2025
0ef3994
Add readme section
ilia-medvedev-codefresh Mar 24, 2025
6d37931
Add readme section
ilia-medvedev-codefresh Mar 24, 2025
6466c59
merge
ilia-medvedev-codefresh Mar 24, 2025
a74aabd
Update README.md.gotmpl
ilia-medvedev-codefresh Apr 9, 2025
9f3b43f
add archiveLogs true by default
ilia-medvedev-codefresh Apr 15, 2025
6ca5cc0
fix: argocd fullname template; feat: add sources-server and event-rep…
mikhail-klimko Mar 24, 2025
ee2f4f7
chore: update chart NOTES.txt (#431)
mikhail-klimko Mar 25, 2025
5278a2f
feat: bump app-proxy to 1.3389.0 (#432)
denis-codefresh Mar 25, 2025
ee0b072
chore(deps): update codefresh-gitops-operator version to 0.5.2 (#434)
scme0 Mar 26, 2025
e5bccb8
fix: argo-events 2.4.7-1-cap-CR-28072 with fix for eventsource restar…
oleksandr-codefresh Mar 27, 2025
37ee72e
patch - private registry external-argocd support and values structure…
ilia-medvedev-codefresh Mar 31, 2025
332d74b
fix gitops operator values (#439)
ilia-medvedev-codefresh Apr 1, 2025
7099828
feat: sources-server 0.3.15 with rollout rollback query (#442)
oleksandr-codefresh Apr 7, 2025
8401981
feat: argo-cd cm custom rollout action 'pause' 'skip-current-step' (#…
oleksandr-codefresh Apr 8, 2025
7548966
feat: rollout-reporter with external argo rollouts (#441)
mikhail-klimko Apr 10, 2025
902a430
update README.md
mikhail-klimko Apr 10, 2025
5d078de
feat: updated app-proxy to 1.3419.1 (#444)
kim-codefresh Apr 10, 2025
dc17f28
feat: update app-proxy image tags to 1.3430.2 (#447)
ATGardner Apr 15, 2025
f14633a
bump gitops operator version (#451)
scme0 Apr 17, 2025
7bb4136
feat: update argo-cd dependency version to 7.8.23-1-cap-v2.14.9 (#455)
ATGardner Apr 20, 2025
f998fb9
updated debian, cf-cli, kubectl (#454)
vitalii-codefresh Apr 22, 2025
2887699
feat: add gitops-operator-default pulling interval (#456)
kim-codefresh Apr 22, 2025
3a763c9
feat: update setup URL in NOTES.txt to use dynamic values (#449)
plakyda-codefresh Apr 23, 2025
da7cd2c
feat: cf-argocd-extras (0.3.16) - event-reporter with git-source data…
oleksandr-codefresh Apr 23, 2025
3bb1db2
eat: cf-argocd-extras (0.3.17) - events with change revision data (#459)
oleksandr-codefresh Apr 24, 2025
6dba4d0
bumped version to 7.8.23-2-cap-v2.14.9-2025-04-23-4de04dd8 (#461)
reggie-k Apr 24, 2025
c476ec5
app-proxy 1.3451.0 with app resource tree with annotations & labels i…
oleksandr-codefresh Apr 28, 2025
d8e9c0a
updated sealed-secrets (#464)
vitalii-codefresh Apr 30, 2025
9bb9c52
feat: security fixes for Argo Rollouts 1.7.2 (#460)
kostis-codefresh Apr 30, 2025
efa1873
updated nginx (#465)
vitalii-codefresh May 1, 2025
2db91c7
feat: add Argo CD version check and validation in pre-install hook (#…
ATGardner May 4, 2025
681c20a
updated csdp, gitops-operator (#467)
vitalii-codefresh May 5, 2025
45e77a8
fix: update app-proxy image tags to 1.3470.0 - fix file revision vali…
ATGardner May 5, 2025
198da9d
feat: removal of event-reporter from cf argo-cd fork (#466)
oleksandr-codefresh May 7, 2025
8055503
feat: limits enforcement in runtime installation (#457)
andrii-codefresh May 7, 2025
3b6b92b
feat: global constraints (#469)
mikhail-klimko May 12, 2025
9623fef
bump codefresh gitops operator version (#470)
scme0 May 12, 2025
1552447
feat: cf-argocd-extras v0.3.22 with payloadVersion reported (#472)
oleksandr-codefresh May 13, 2025
6de98f2
Feat/revert cr 28342 usage (#474)
andrii-codefresh May 14, 2025
5c551ca
feat: add gzip configuration to internal router configmap (CR-24650) …
olegt-codefresh May 14, 2025
2a825cd
merge main
ilia-medvedev-codefresh May 14, 2025
5a2fe33
add possibility to disable configmap generation by passing null to ar…
ilia-medvedev-codefresh May 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions charts/gitops-runtime/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,6 @@ dependencies:
version: 0.7.1
alias: gitops-operator
condition: gitops-operator.enabled
- name: garage
repository: https://codefresh-io.github.io/garage
alias: garage-workflows-artifact-storage
version: 0.5.0-cf.3
condition: garage-workflows-artifact-storage.enabled
- name: cf-argocd-extras
repository: oci://quay.io/codefresh/charts
version: 0.3.22
23 changes: 15 additions & 8 deletions charts/gitops-runtime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,18 @@ See [Use OCI-based registries](https://helm.sh/docs/topics/registries/)
## Codefresh official documentation:
Prior to running the installation please see the official documentation at: https://codefresh.io/docs/docs/installation/gitops/hybrid-gitops-helm-installation/

## Argo-workflows artifact and log storage
> [!NOTE]
> This version of the chart includes default configuration for storing workflow artifacts and logs in Codefresh provided s3 compatible storage.

If you have your own storage configuration using the default configmap `artifact-repositories` upgrading the chart will override your artifact storage configuration.
To prevent this please set `argo-workflows.controller.workflowDefaults.spec.artifactRepository.configMap` to `artifact-repositories` and `argo-workflows.controller.workflowDefaults.spec.artifactRepository.key`
to the respective key in your configmap identifying the repository.
> [!WARNING]
> It's highly recommended to use your own artifact storage for data privacy reasons.
> Codefresh provided storage has a retention policy of 14 days and limitations on uploaded file sizes.
> Please refer to the official documentation for more details.

## Installation with External ArgoCD

If you want to use an existing ArgoCD installation, you can disable the built-in ArgoCD and configure the GitOps Runtime to use the external ArgoCD.
Expand Down Expand Up @@ -259,6 +271,9 @@ sealed-secrets:
| argo-rollouts.enabled | bool | `true` | |
| argo-rollouts.fullnameOverride | string | `"argo-rollouts"` | |
| argo-rollouts.installCRDs | bool | `true` | |
| argo-workflows.codefreshWorkflowLogs | object | `{"endpoint":"gitops-workflow-logs.codefresh.io","insecure":false}` | Argo workflows logs storage on Codefresh platform settings. Don't change unless instructed by Codefresh support. |
| argo-workflows.controller.workflowDefaults.spec.archiveLogs | bool | `true` | |
| argo-workflows.controller.workflowDefaults.spec.artifactRepositoryRef | object | `{"configMap":"codefresh-workflows-log-store","key":"codefresh-workflows-log-store"}` | By default artifact repository is set to a Codefresh provided repository. For data privacy it is reccommended to set your own artifact repository. For instructions see: https://argo-workflows.readthedocs.io/en/latest/configure-artifact-repository/#configuring-your-artifact-repository |
| argo-workflows.crds.install | bool | `true` | Install and upgrade CRDs |
| argo-workflows.enabled | bool | `true` | |
| argo-workflows.executor.resources.requests.ephemeral-storage | string | `"10Mi"` | |
Expand Down Expand Up @@ -304,14 +319,6 @@ sealed-secrets:
| event-reporters.workflow.sensor.retryStrategy.steps | int | `3` | Number of retries |
| event-reporters.workflow.sensor.tolerations | list | `[]` | |
| event-reporters.workflow.serviceAccount.create | bool | `true` | |
| garage-workflows-artifact-storage | object | `{"deployment":{"kind":"StatefulSet","replicaCount":3},"enabled":false,"fullnameOverride":"garage","garage":{"replicationMode":3},"persistence":{"data":{"size":"100Mi","storageClass":""},"enabled":true,"meta":{"size":"100Mi","storageClass":""}},"resources":{},"tests":{"enabled":false}}` | Builtin Workflows artifacts storage solution. Local S3 backed by local persistence with (PV and PVC) |
| garage-workflows-artifact-storage.deployment.kind | string | `"StatefulSet"` | Only statefulset is supported for Codefresh gitops runtime. Do not change this |
| garage-workflows-artifact-storage.persistence.data | object | `{"size":"100Mi","storageClass":""}` | Volume that stores artifacts and logs for workflows |
| garage-workflows-artifact-storage.persistence.data.storageClass | string | `""` | When empty value empty the default storage class for the cluster will be used |
| garage-workflows-artifact-storage.persistence.meta | object | `{"size":"100Mi","storageClass":""}` | Volume that stores cluster metadata |
| garage-workflows-artifact-storage.persistence.meta.storageClass | string | `""` | When empty value empty the default storage class for the cluster will be used |
| garage-workflows-artifact-storage.resources | object | `{}` | Resources for garage pods. For smaller deployments at least 100m CPU and 1024Mi memory is reccommended. For larger deployments double this size. |
| garage-workflows-artifact-storage.tests | object | `{"enabled":false}` | Helm tests |
| gitops-operator.affinity | object | `{}` | |
| gitops-operator.crds | object | `{"additionalLabels":{},"annotations":{},"install":true,"keep":false}` | Codefresh gitops operator crds |
| gitops-operator.crds.additionalLabels | object | `{}` | Additional labels for gitops operator CRDs |
Expand Down
13 changes: 13 additions & 0 deletions charts/gitops-runtime/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,19 @@ See [Use OCI-based registries](https://helm.sh/docs/topics/registries/)
## Codefresh official documentation:
Prior to running the installation please see the official documentation at: https://codefresh.io/docs/docs/installation/gitops/hybrid-gitops-helm-installation/

## Argo-workflows artifact and log storage
> [!NOTE]
> This version of the chart includes default configuration for storing workflow artifacts and logs in Codefresh provided s3 compatible storage.

If you have your own storage configuration using the default configmap `artifact-repositories` upgrading the chart will override your artifact storage configuration.
To prevent this please set `argo-workflows.controller.workflowDefaults.spec.artifactRepository.configMap` to `artifact-repositories` and `argo-workflows.controller.workflowDefaults.spec.artifactRepository.key`
to the respective key in your configmap identifying the repository.
> [!WARNING]
> It's highly recommended to use your own artifact storage for data privacy reasons.
> Codefresh provided storage has a retention policy of 14 days and limitations on uploaded file sizes.
> Please refer to the official documentation for more details.


## Installation with External ArgoCD

If you want to use an existing ArgoCD installation, you can disable the built-in ArgoCD and configure the GitOps Runtime to use the external ArgoCD.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{{- if hasKey (index .Values "argo-workflows" "controller" "workflowDefaults" "spec" "artifactRepositoryRef") "configMap" }}
{{- if eq (index .Values "argo-workflows" "controller" "workflowDefaults" "spec" "artifactRepositoryRef" "configMap") "codefresh-workflows-log-store" }}
{{- $_ := required "global.codefresh.accountId is required" .Values.global.codefresh.accountId }}
apiVersion: v1
data:
codefresh-workflows-log-store: |
archiveLogs: true
s3:
bucket: {{ .Values.global.codefresh.accountId }}
endpoint: {{ index .Values "argo-workflows" "codefreshWorkflowLogs" "endpoint" }}
insecure: {{ index .Values "argo-workflows" "codefreshWorkflowLogs" "insecure" }}
keyFormat: {{ .Values.global.runtime.name }}/{{ "{{" }}workflow.name{{ "}}" }}/{{ "{{" }}pod.name{{ "}}" }}
accessKeySecret:
name: codefresh-token
key: token
secretKeySecret:
name: codefresh-token
key: token
kind: ConfigMap
metadata:
annotations:
workflows.argoproj.io/default-artifact-repository: codefresh-workflows-log-store
name: codefresh-workflows-log-store
{{- end }}
{{- end }}
2 changes: 1 addition & 1 deletion charts/gitops-runtime/templates/tunnel-client.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ reduce complexity of installation and number or mandatory values to provide for
{{ $runtimeName := required "runtime.name is required" .Values.global.runtime.name }}
{{ $_ := set $tunnelClientContext.Values.tunnel "subdomainPrefix" (printf "%s-%s" $accoundId $runtimeName)}}
{{- include "codefresh-tunnel-client.resources" $tunnelClientContext }}
{{- end }}
{{- end }}
35 changes: 35 additions & 0 deletions charts/gitops-runtime/tests/codefresh-workflow-logs-store.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
suite: codefresh-workflow-logs-store tests
templates:
- codefresh-workflow-log-store.yaml
tests:

- it: Should only create the configmap when the workflowDefaults.artifactRepositoryRef.configMap is set to codefresh-workflows-log-store
template: 'codefresh-workflow-log-store.yaml'
values:
- ./values/mandatory-values.yaml
set:
argo-workflows.controller.workflowDefaults.spec.artifactRepositoryRef.configMap: "some-other-configmap"
asserts:
- hasDocuments:
count: 0

- it: Should only create the configmap when the workflowDefaults.artifactRepositoryRef.configMap is set to codefresh-workflows-log-store
template: 'codefresh-workflow-log-store.yaml'
values:
- ./values/mandatory-values.yaml
set:
argo-workflows.controller.workflowDefaults.spec.artifactRepositoryRef.configMap: "codefresh-workflows-log-store"
asserts:
- hasDocuments:
count: 1

- it: Should fail if the accountId is not set
template: 'codefresh-workflow-log-store.yaml'
values:
- ./values/mandatory-values.yaml
set:
argo-workflows.controller.workflowDefaults.spec.artifactRepositoryRef.configMap: "codefresh-workflows-log-store"
global.codefresh.accountId: ""
asserts:
- failedTemplate:
errorMessage: 'global.codefresh.accountId is required'
44 changes: 12 additions & 32 deletions charts/gitops-runtime/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,18 @@ argo-workflows:
resources:
requests:
ephemeral-storage: 10Mi
controller:
workflowDefaults:
spec:
archiveLogs: true
# -- By default artifact repository is set to a Codefresh provided repository. For data privacy it is reccommended to set your own artifact repository. For instructions see: https://argo-workflows.readthedocs.io/en/latest/configure-artifact-repository/#configuring-your-artifact-repository
artifactRepositoryRef:
configMap: codefresh-workflows-log-store
key: codefresh-workflows-log-store
# -- Argo workflows logs storage on Codefresh platform settings. Don't change unless instructed by Codefresh support.
codefreshWorkflowLogs:
endpoint: gitops-workflow-logs.codefresh.io
insecure: false

#-----------------------------------------------------------------------------------------------------------------------
# Argo rollouts
Expand Down Expand Up @@ -704,38 +716,6 @@ gitops-operator:
cpu: 100m
memory: 128Mi
#-----------------------------------------------------------------------------------------------------------------------
# Garage
#-----------------------------------------------------------------------------------------------------------------------
# -- Builtin Workflows artifacts storage solution. Local S3 backed by local persistence with (PV and PVC)
garage-workflows-artifact-storage:
fullnameOverride: garage
enabled: false
deployment:
# -- Only statefulset is supported for Codefresh gitops runtime. Do not change this
kind: StatefulSet
replicaCount: 3
garage:
#-- Default to 3 replicas, see the replication_mode section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication-mode
replicationMode: 3
persistence:
enabled: true
# -- Volume that stores cluster metadata
meta:
# -- When empty value empty the default storage class for the cluster will be used
storageClass: ""
size: 100Mi
# -- Volume that stores artifacts and logs for workflows
data:
# -- When empty value empty the default storage class for the cluster will be used
storageClass: ""
size: 100Mi
# -- Resources for garage pods. For smaller deployments at least 100m CPU and 1024Mi memory is reccommended. For larger deployments double this size.
resources: {}
# -- Helm tests
tests:
enabled: false

#-----------------------------------------------------------------------------------------------------------------------
# cf-argocd-extras
#-----------------------------------------------------------------------------------------------------------------------
# -- Codefresh extra services for ArgoCD
Expand Down