Proto commits in spinnaker/kleat

These 87 commits are when the Protocol Buffers files have changed:

2021-05-18

Fix wrong type of `providers.aws.bakeryDefaults.baseImages[].virtualizationSettings` (#175) * Change type of `providers.aws.bakeryDefaults.baseImages[].virtualizationSettings` as repeated value Rosco manages `virtualizationSettings` for `aws` as a list of settings where an item contains regional VM settings, like follows. aws: bakeryDefaults: baseImages: - baseImage: id: ubuntu shortDescription: v12.04 detailedDescription: Ubuntu Precise Pangolin v12.04 packageType: deb templateFile: aws-ebs.json virtualizationSettings: - region: us-east-1 virtualizationType: hvm instanceType: t2.micro sourceAmi: ami-d4aed0bc sshUserName: ubuntu spotPrice: "0" spotPriceAutoProduct: Linux/UNIX (Amazon VPC) I have confirmed that testdata also contains wrong schema around `virtualizationSettings`. As long as I could confirm from Rosco implementation, it had `virtualizationSettings` was typed as `List<AWSVirtuzliationSettings>`, therefore, I do believe this is mistake while generating testdata for `aws` provider. static class AWSBakeryDefaults { String awsAccessKey String awsSecretKey String awsSubnetId String awsVpcId Boolean awsAssociatePublicIpAddress String templateFile BakeRequest.VmType defaultVirtualizationType List<AWSOperatingSystemVirtualizationSettings> baseImages = [] } static class AWSOperatingSystemVirtualizationSettings { BakeOptions.BaseImage baseImage List<AWSVirtualizationSettings> virtualizationSettings = [] } https://github.com/spinnaker/rosco/blob/version-0.25.0/rosco-core/src/main/groovy/com/netflix/spinnaker/rosco/providers/aws/config/RoscoAWSConfiguration.groovy#L65 * Fix broken test after changing `providers.aws.bakeryDefaults.baseImages[].virtualizationSettings` as plural * Fix `golangci-lint` suggestions on simplifying the code

The documentation is generated from this commit.

2021-04-12

feat(keel): add support for ManagedDelivery (#145) * feat(keel): add support for ManagedDelivery - add keel configuration - updated other services when keel configuration is provided - update tests with keel configuration * fix linting issues * updates based on feedback - move deck defaulting to code - fix proto comments - cleanups * feat(keel): configure keel based on halconfig setup - cleanup documentation - refactor SQL for better reflecting config options * rebase changes

2021-01-20

Plugin compatibility validation command (#161) * feat(plugins): mapping plugins config in halconfig * feat(plugins): import plugin-tools module * feat(plugins): implement plugins validation subcommand * test(plugins): write unit test for plugins.go * feat(plugins): refactor plugin validation * fix(plugins): cleanup golangci-lint errors * docs(plugins): include description in extensibility.proto

2020-11-16

Add hal to echo converter for microsoft team (#155)

Add microsoft teams notification (#153)

2020-10-05

feat(aws): Adding property externalId to AWS account config for assuming roles. (#146) * feat(aws): adding externalId to aws account config * fix(aws): fix spacing * fix(aws): fix alphabetical order of aws properties.

2020-09-30

feat(storage): add redis and sql to storage options (#143)

2020-08-20

docs(Kleat): attempt to fix markdown table formatting (#133)

Edit kleat ci docs (#132) * docs(Kleat): edit CI config doc in Kleat API. * docs(Kleat): edit CI config doc in Kleat API. * docs(Kleat): edit CI config doc in Kleat API. * docs(Kleat): edit CI config doc in Kleat API. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

2020-08-07

refactor(proto): exclude "mine canary" feature flag from Deck's protos (#129) * refactor(proto): exclude "mine canary" feature flag from Deck's protos Upon further investigation, the "mine canary" feature flag should only be gating the pre-Kayenta legacy canary stages, which we suspect is currently only used by opsmx's custom build. To avoid confusing OSS users, do not permist configuration of this flag with kleat. * fix(proto): remove reserved statement for removed mineCanary field, update Deck_Features proto to exclude it

feat(proto): remove unread Deck changelog proto (#130) * feat(proto): remove unread Deck changelog proto As of 1.21, Deck no longer requires config to render a changelog modal ([source](https://github.com/spinnaker/deck/pull/8353)). * refactor(proto): remove reserved statement for removed changelog field

2020-08-04

Edit kleat canary protos (#127) * Docs(API): edits to Kleat API for canary config fields * Docs(API): edits to Kleat API for canary config fields * Docs(API): edits to Kleat API for canary config fields * Docs(API): edits after first reviews * docs(API): more updates after review * docs(API): more updates after review

2020-07-23

Api docs artifacts (#126) * docs(api): refinements to artifacts config doc * docs(api): refinements to artifacts config doc * docs(api): refinements to artifacts config doc * docs(api): refinements to artifacts config doc * docs(api): refinements to artifacts config doc

2020-07-17

docs(api): Edit Kleat API documentation for Bitbucket artifact config. (#118) * docs(api): Edit Kleat API documentation for Bitbucket artifact config. * docs(api): Edit Kleat API documentation for Bitbucket artifact config.

2020-07-15

Add documentation on nested image paths (#116) * Add documentation on nested image paths * Generate protos/docs

2020-07-07

feat(authn/saml): New optional SAML parameter for signing messages (#114) * feat(authn/saml): New optional SAML parameter for signing messages * feat(authn/saml): Added "unspecified" enum entry * feat(authn/saml): Regenerated code

2020-06-30

feat(proto): add Travis filteredRepositories field (#111) This field was added to Halyard recently: https://github.com/spinnaker/halyard/pull/1730

2020-06-25

fix(deck): create empty providers.kubernetes entry if provider is enabled (#108) As explained in this [PR](https://github.com/spinnaker/deck/pull/8381), the Kubernetes provider does not have any statically configured defaults in Deck. However, the registration of the Kubernetes cloud provider in Deck depends on the truthiness of `providers.kubernetes`.

2020-06-23

feat(travis): Add numberOfJobs and buildResultLimit (#106) These parameters were just added to Halyard, since the time we added Travis support to kleat. Update kleat with them.

2020-06-19

feat(stats): Include stats fields in echo config (#103) * test(echo): Refactor echo tests to handle stats changes The stats block has a property that no other field in the hal config has; even when nothing is specified in the input, we still generate some output. This is because one of the blocks in the stats config is the deployment method, which kleat needs to fill in with details about itself. While this is a departure from the usual situation where empty input leads to empty output, it feels like this is the one case where an exception is warranted given that this output needs to be populated by kleat rather than by the user. To make the diff on that change smaller, refactor the tests now to include this default value for the Stats field (currently nil). * refactor(stats): Use a different message for input/output stats While currently the only configurable stats field is whether it is enabled, we'll soon be adding more fields. Some fields will only be in the output (ie, that Spinnaker is being deployed by kleat and the kleat version) and should not be in the input. To allow us to do this, use a different message for the block in the hal config and in the echo config. Another approach would be to use the same message, but have the fields be ignored in the input. I didn't do that as I don't want those documented as they might add confusion, but am definitely open to further thoughts on this. * feat(stats): Add stats protos Add the remaining fields to the stats protos. This commit does not use these new fields yet, but it does lightly refactor the tests to explicitly include some of the fields as nil in the tests (making the diff in the next commit cleaner). * feat(stats): Include stats fields in echo config This commit includes the newly-added stats fields in the generated echo config, and adds some tests on these fields.

2020-06-17

fix(core): Address ignored fields from test halconfig (#102) * fix(test): Remove unused fields from test halconfig This commit removes fields that are not used by kleat from the test halconfig. * feat(proto): Add some missing protos These are fields that Halyard defaults but doesn not expose commands for editing (though users can still manuall set them). In order to preserve backwards-compatibility for users migrating from Halyard we should still support reading them. Ideally the fields should be defaulted to something sensible in the service, but this will not affect migrating users who already have them specified. * fix(proto): Fix mis-spelled requiredGroupMembership This was incorrectly included as requiredGroupMemberships in a number of protos. * feat(proto): Remove experimental flag As we're not using optional protos, remove the experimental flag as we don't need it anymore. * chore(core): Bump deps and run go mod tidy

fix(pkg): Add snake_case json_names to snake_case monitoring protos (#101)

feat(pkg): add monitoring daemon support (#100) Adds monitoring dameon metric stores, hal to monitoring translation, and test converage.

feat(proto): Add unspecified to enum fields (#98) * style(test): Sort halconfig file It's easier to find fields if it's sorted. Also while making trivial changes, format the markdown files with prettier. * feat(security): Remove unused enum This enum was only used by Halyard to track they type of OAuth provider you had configured; it used this to set other fields when you ran hal commands. A while back I updated the documentation on spinnaker.io to show users how to set the required fields themselves (which is useful both for directly editing your hal config with Halyard, or for early adopters of kleat). We can remove this field as it is not used. I'll add a note to the changelog about this. Existing Halyard users would already have had the other fields filled out based on that field when they ran hal commands, so I'm not including this as a breaking change. New users will need to set their config fields themselves using the documentation on spinnaker.io. * feat(proto): Add unspecified to enum fields We want to distinguish unspecified enums from enums set to their default value. Let's do this by making the default value in each enum be a sentinel UNSPECIFIED value. This means that when the field is not specified, it will be deseriazlied as UNSPECIFIED, and will be omitted from re-serialization. Any valid value will be included in serialization.

2020-06-16

feat(proto): Wrap string and int32 fields where necessary (#97) * feat(kubernetes): Add missing Kubernetes fields A few fields were missing from KubernetesAccount. Add these. * feat(proto): Wrap string and int32 fields where necessary This commit goes through all string and int32 fields, wrapping them where necessary. The only changes necessary were a few string fields. * fix(proto): Fix two optional fields I said I'd update the two existing optional string fields, but forgot to in the last commit. * style(proto): Fix capitalization Co-authored-by: Maggie Neterval <mneterval@google.com> * fix(core): Recompile protos Co-authored-by: Maggie Neterval <mneterval@google.com>

refactor(proto): Wrap all boolean fields (#96) * refactor(proto): Wrap all boolean fields We've decided that all boolean fields should have presence information. Without this, booleans will not be included in the serialized config if they are 'false'. While this is fine for fields that default to false (for now), it breaks for any field that defaults to true. It also would cause issues if any field were changed in the future to default to true. One of the advantages of kleat is that it only includes fields that are explicitly set by the user in its config, so it is easier to change field defaults, as only users who have explicitly set a field will be overriding the default. But if we store a boolean without presence, it's impossible to ever in the future default it to 'true'. It also might be confusing for users that some fields in their hal config are not in their microservice configs. * fix(core): Remove TODO and add documentation Remove a TODO, and also fix the code which should pass through the value when non-nil (not only when true). Also document how to choose a field type. This also re-wraps the file. I didn't otherwise change anythign above the added section, so didn't break this into commits as it should be easy to see what was re-wrapped vs what was added. * test(orca): Fix broken test I fixed the code so explicitly set false is passed through for pipeline templates, but didn't fix the test.

2020-06-15

feat(proto): mark AWS and Google canary accounts rootFolder as optional (#95) * feat(test): add tests for canary rootFolder fields Add empty and non-empty rootFolder field to halconfig.yml. The empty rootFolder will be omitted from kayenta.yml, which is not the behavior we want. The next commit will mark rootFolder as optional since we are considering an empty string a valid value distinct from nil. * feat(proto): mark AWS and Google canary accounts rootFolder as optional

2020-06-09

refactor(config): Clean up some of the config generation (#83) * feat(proto): Add proto for serialized config files In order to make it easier to pass around the serialized form of the config files (as well as the name of each file), make a proto to hold this information. * refactor(config): Add Serializer interface There are currently three different ways to serialize a config to bytes: * Convert to YAML * Write out JavaScript that loads JSON * Write out an env file Create an interface Serializer and write three implementations of it to abstract away these three different ways of serializing. * refactor(config): Generate serialized configs as a proto Add a function GenerateConfigFiles that converts a proto of the typed service configs into a proto of the serialized service configs. Then use this function when generating the service configs from the hal config. * refactor(config): Split hal parsing into its own function Let's expose a function to parse the hal config from a file and then require that we pass a parsed hal config to the function that writes out the service configs. * refactor(config): Remove validate_paths There are only two functions in this module; one is unused after some recent refactoring and the other can be moved to the one place where it is used.

feat(pkg): add Igor support (#82) * feat(proto): add Repository and Artifactory protos Add repository and artifactory protos in preparation for adding Igor config. Although Halyard allows you to configure Fiat permissions on each artifactory search, this does not appear to be consumed in any way in Igor, so I've omitted this field from Kleat. * feat(proto): add Igor service config Only includes artifact templates, unlike Halyard, which includes all configured artifact types, because templates are all that appear to be needed by Igor. * feat(pkg): translate and write Igor config * test(pkg): add unit and end to end tests for Igor Also updates the Echo end-to-end tests, since Echo needs the ci.gcb config. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

2020-06-08

fix(deck): Features should be singular (#80) The deck config expects feature flags to be under the singular 'feature'. (This is a bit confusing because Halyard used plural 'features' in its config, but it ended up singular in the generated config.)

2020-06-05

feat(deck): Generate environment variables for deck (#78) In addition to requiring client-side config provided as settings.js, deck requires some settings to be applied server-side. These are applied by passing environment variables to the deck server process. There does not appear to be a way to pass these as YAML, we we'll need to generate a file of key=value pairs, which will then be used by a configMapGenerator. Rather than manually translate the field names to the UPPER_SNAKE case needed for the variables, I've just explicitly overridden the json name for these fields. (Naming the fields in UPPER_SNAKE didn't work as the underscores were removed in the auto-generated JSON name.)

refactor(pkg): remove roscoMode (#77)

feat(pkg): add Deck support (part 3!) (#76) * feat(pkg): conditionally set Gate basUrl to http or https * feat(pkg): add hal-to-deck tests * refator(proto): do not expose canary.stagesEnabled Upon reflection, I highly doubt any Halyard users are using Kayenta separately from using Spinnaker's built-in canary stages. Let's not expose this parameter until presented with evidence for its necessity. * refactor(proto): remove completed TODOs We have already documented the new /var/secrets convention, and the fact that provider defaults statically provided to Deck will fall back to those of the explicitly specified primaryAccount. * feat(pkg): pass top-level Spinnaker version to Deck Deck displays the top-level Spinnaker version in the UI. Pass top-level Spinnaker version from Halconfig to Deck.

2020-06-03

fix(proto): configure Orca and Echo timezone (#75) * fix(proto): configure timezone in Orca * fix(proto): configure timezone in Echo * refactor(docs): remove completed timezone TODO

feat(pkg): Add Deck support (part 2!) (#74) * fix(cmd): fix error check os.Args actually includes the program name as argument 0, so we actually need to check if len != 3. * fix(proto): add mptV2UI feature flag to Deck proto * test(pkg): add Deck end to end tests Although the final settings.js output will not be YAML, let's add an end to end test for the actual data translation from halconfig to Deck config as yaml-to-yaml, like the other microservices. Also adds a few more TODOs to hal_to_deck that should be addressed in a followup PR but do not want to block on completing the writing of settings.js * feat(internal): write Deck config as JS Kleat can now write Deck's config as a valid JS file.

2020-06-02

feat(proto): add Deck support (part 1!) (#73) * feat(proto): add Bearychat, Email, Google Chat, and Pubsub notifications protos Although these four notification types are not currently configurable with Halyard, they are defaulted to enabled in Deck's halconfig/settings.js. This is because when *any* notification types are configured, Deck assumes that is the exhaustive list of enabled notification types. (If *no* notification types are configured, Deck's default behavior is to enable all of them.) Since it is unclear if it would be reasonable to push the existing Halyard-default-behavior into Deck, let's simply be consistent with notification types in Kleat and allow users to manually enable any types they wish. * feat(proto): add canary UI protos Adds Deck-specific protos to canary.proto. Note that the defaults for `stagesEnabled` and `defaultJudge` depend on those being pushed from Halyard to Deck-Kayenta in [this PR](https://github.com/spinnaker/deck-kayenta/pull/499). * feat(proto): add timezone proto to halconfig At least Deck, Orca, and Echo consume a user-configurable timezone. Unfortunately, Halyard currently writes the timezone to a `global.spinnaker.timezone` field in spinnaker.yml, which is [my fault](https://github.com/spinnaker/halyard/pull/1117), though in my defense it was only after a CR suggestion that I moved the translation from Orca's profile to the global profile... I don't want to further complicate this Deck-focused PR with logic to add timezone to Echo and Orca's protos, so will add a TODO to do that in a followup PR. * feat(proto): update features.proto Updates features.proto to include flags consumed by Deck. Adds TODO for how to best handle roscoMode, which Deck's halconfig/settings.js currently sets to true (Halyard does not expose a command to manually set roscoMode.). * feat(proto): translate Gremlin integration to Gate Also adds tests. * feat(proto): add Deck protos Strongly types all Deck configuration exposed by Halyard, with a few exceptions: * canary.atlasWebComponentsUrl: likely only used by Netflix, unclear why this is Halyard-configurable. * canary.reduxLogger: should only be useful in local development, unclear why this is Halyard-configurable. * defaultInstancePort: defaulted in halconfig/settings to 80. In most places in the code, we also fall back to 80, but there are a few places we fallback to null instead, perhaps to force users to think about making a choice here, for example, when configuring a new application. Let's not expose this property in Kleat until presented with a compelling reason. Also adds several TODOs to handle after some discussion in a followup PR: * Handle top-level version * Handle changelog gist * Handle roscoMode * Add Tencent cloud protos * feat(pkg): add hal to deck translation Adds TODOS for the following: * Replace stubbed gateUrl with actual gateUrl, which may be http or http depending on other settings * Add changelog translation once we determine how we are going to handle * Refactor getDeckProvidersConfig which feels like terrible code, but couldn't find an elegant Go-native way to reduce duplication Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

2020-05-26

feat(rosco): Generate rosco config (#69) * feat(rosco): Add rosco config protos The rosco config is very similar to the clouddriver config, in that it contains the configured cloud providers at the top level. The one difference is that only some cloud providers support baking, so it only contains these providers. * feat(rosco): Generate rosco config This commit adds support for generating a rosco config from a hal config. * feat(rosco): Add rosco output to service configs Also add rosco.yml to the end-to-end test. * docs(rosco): Add note to changelog about baseImages

2020-05-21

feat(gate): Add deck base URL (#68) * refactor(proto): Add baseUrl field to ServiceEnabled and rename We will in some cases need to have kleat set the baseUrl field for a service (in cases where the hal config affects this, such as gate and deck). As this lives under the same config key as whether the service is enabled, let's augment that proto to supply both these fields. Also rename it as it's more general than when it was initially designed. * feat(gate): Add deck base URL Gate needs to know the base URL for deck so that it can handle redirects during auth. Rather than force the user to manually override this in spinnaker.yml, let's output this as part of the gate config.

feat(pkg): include optional service enabled as config property (#67) We have (for now) decided to have whether or not an optional service is enabled be a property of any services that need that information, rather than service discovery information in spinnaker.yml. Let's translate canary.enabled into services.kayenta.enabled for the two services that need to communicate with Kayenta (Gate and Orca).

2020-05-19

feat(pkg): translate and write kayenta.yml from halconfig (#66) * feat(proto): add kayenta to services proto * feat(pkg): translate and write kayenta.yml from halconfig * feat(test): add end-to-end test for kayenta config translation * feat(test): add table tests for kayenta config

2020-05-15

feat(fiat): Generate fiat config (#65) * refactor(security): Split authentication and authorization The security package is fairly large; let's split it into authentication and authorization. One motivation for this is that you can only use a single enum constant once in each package (not counting nested enums), so the likelihood of collision goes up as packages get larger. * refactor(authz): Nest RoleProviderType enum This enum is very specific to the message in which it's used, so let's nest it. * feat(fiat): Add fiat config proto This commit just adds the proto that holds the fiat config; future commits will actually add the logic to generate a fiat config. * feat(fiat): Generate fiat config This commit adds the logic for generating a fiat config from the hal config, and adds a basic-end-to-end test. * test(fiat): Add unit tests to fiat config generation As there is no complex logic in the config generation a simple test of an empty config and a non-empty config should be sufficient.

2020-05-14

feat(proto): add canary protos (#62) * feat(proto): add canary protos * docs(config): update README with canary migration instructions The existing canary section of the halconfig is difficult to represent in proto. Let's require users to migrate each integration block they care about from an item in a list to an entry in a map. * fix(proto): re-run make proto after rebasing off master

test(gate): Add test for gate config generation (#61) * fix(proto): Fix some of the security protos Halyard is writing the edited enums in upper case, so we should make our enum the same. (I was confused because the toString was writing it in lower-case, but I guess Halyard is not using toString as part of serialization, which I guess probably makes sense when you think about it enough.) I also accidentally named a property spring instead of security. Let's also rename the message SpringSecurity and un-nest it as it represents the top-level security.* config settings for Spring (to which I'll be shortly adding another field). * feat(proto): Add basic authentication Halyard currently looks on the service settings file for a flag to determine whether to enable basic authentication; this is different from all the other authentication methods, which are configured together in the same block of the hal config. As I don't anticipate wanting to expose the service settings through kleat (as these are in general related to how a service is deployed rather than how it behaves) let's add a new field to the halconfig so users can still configure it if they want. I suspect this is not very common in any real production set-up (as it would require sharing a username and password with everyone using Spinnaker) but as it was easy to just add the two fields, it seemed better than not supporting it anymore. Users *will* need to move any basic auth config from the service settings file to the hal config. * feat(gate): Write out the gate config to file Also, update the logic for generating the gate config to avoid generating config blocks that are not relevant. In general, I think we should adopt the pattern (and sort of already have) where if we're not just looking something up in the hal config and are actually creating a new struct inline, we should first check to see if we need to create the wrapper struct to avoid writing out a bunch of empty structs. Remove the TODO as upon further investigation we don't need to allow these to be settable in kleat and can rely on the default port and binding. * test(gate): Add test for gate config generation This commit adds the end-to-end test that the gate config is correctly generated when everything is specified. (Of course this isn't really realisitic as you likely won't have every type of auth configured but given that kleat just passes things through it's still a good test.) Next will be the unit tests that verify the particular parts of the config generation that are somewhat complex work properly. * fix(tests): Use more specific comparators in tests Protobufs support a proto.Equal method that uses proto-supported reflection to deeply compare two messages; use this instead of reflect.DeepEqual as go reflection should not be used on protobufs in favor of protobuf-specific reflection. Similarly, in one case we're comparing bytes with reflect.DeepEqual whereas it would be better to just use the specific comparator for bytes. (I remember thinking this when I wrote it but I guess I never changed it.)

2020-05-12

refactor(proto): remove unused feature flags (#59) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(proto): Wrap some proto values (#58) This commit wraps some proto fields in cases where the service needs to make a meaningful distinction between the default value for the type and no value for the type (ie null). In particular, these are: * Booleans that currently (or might reasonably in the future) default to true in the service (and where explicitly passing false would be different from not passing a value) * Strings where empty string is a reasonable value but we currently (or might reasonably in the future) default nulls to something that's not the empty string. It also removes a parameter from the Travis config that was being ignored by the service and so should not be supported in kleat.

2020-05-08

feat(proto): add CI protos (#56) * feat(proto): add CI protos * fix(proto): plus gcb perms, minus jenkins master

2020-05-07

feat(gate): Add initial gate config (#55) * fix(gate): Update SAML proto Halyard did some mappings and transformations that we won't be replicating in kleat. Notably, Halyard parsed the metadata string you passed on the command line to store it in either metadataLocal or metadataRemote; but then when generating the gate config it would just add whichever one was not null to metadataUrl. We'll only support metadataUrl in kleat and users will need to (once) manually update their key from metadataLocal/metadataRemote to metadataUrl. Halyard accepted a redirect URI for SAML and parsed out the port, host, and protocol. We'll require that users supply these separately, which is how Gate expects them. * feat(gate): Add initial gate config This commit adds the ability to configure Gate using kleat. I still need to add tests for gate, as well as do some more extensive comparison with what Halyard outputs to make sure everything is correct here, but this defines all of the types and allows a gate config to be generated.

2020-05-04

fix(oauth2): Update some documentation of the OAuth 2.0 provider (#53) * fix(oauth2): Update some documentation of the OAuth 2.0 provider Improve the documentation with things that I learned while looking into how to improve the spinnaker.io documentation around OAuth 2.0. In particular: * Add better comments to a few fields * Convert clientAuthenticationScheme to an enum as there are only four possible values * Standardize on OAuth 2.0 in documentation * style(proto): Remove accidental newline

2020-05-01

feat(security): Add OAuth2 protos (#52) * fix(proto): Fix package for webhook It's in the security directory, so should be in package security. * feat(security): Add OAuth2 protos This commit adds the protos required for configuring OAuth2 through kleat. As I noted in a prior PR, Halyard does some complex logic here that we'll need to account for. What it does is that, based on your OAuth2 provider, it automatically sets a bunch of these fields. So for example if you choose Google as the OAuthProvider, it sets the accessTokenUrl to https://www.googleapis.com/oauth2/v4/token for you. I think the way we should handle this is to move all that defaulting to Gate. If Gate finds that the OAuth2 provider is Google, and you have not explicitly set an accessTokenUrl it should set it to the one that Halyard would have set. (And similarly for the other providers and fields.) It's only really for the custom provider that you'll need to set all of these fields. The next step after this PR will be to make these changes in Gate. After I do that, I'll come back and update the comments to explain when you do/don't need to set these fields, but need to make the change in Gate first so I understand this enough to document it.

feat(security): Add security to hal config (#51) * feat(security): Add security to hal config This commit adds the security block to the hal config, as well as the configuration nested under it. This includes SSL for gate/deck, as well as authentication and authorization. The one exception is that I left a TODO for OAuth as Halyard looks to be doing some complex logic there and I want to look at separately. * fix(proto): Fix some typos in proto comments Co-Authored-By: Maggie Neterval <mneterval@google.com> Co-authored-by: Maggie Neterval <mneterval@google.com>

feat(orca): Add orca protocol buffers (#50) This commit adds support for configuring orca with kleat.

2020-04-29

feat(front50): Add proto for s3 storage config (#47) * feat(front50): Add proto for s3 storage config Also fix the text that Halyard will create a bucket for you in the GCS proto; kleat will no longer do this. * fix(proto): Remove non-ASCII quotes from docs Let's avoid any encoding issues. * feat(front50): Include s3 proto in appropriate messages I forgot to include the s3 proto in the hal config and the front50 config. Add it there. * fix(s3): Enabled should be a bool Co-Authored-By: Maggie Neterval <mneterval@google.com> Co-authored-by: Maggie Neterval <mneterval@google.com>

fix(proto): change enabled string to bool (#45)

2020-04-28

refactor(proto): Misc cleanup on protos (#42) * fix(proto): Don't delete all generated code if protoc fails We currently delete all the generated code before running protoc so that we can start with an empty directory (avoiding cases where we miss deleting an old generated file). This means that if protoc fails, we delete all the generated code, which can be annoying. Let's instead generate the code into the staging directory and only move it into the final directory if protoc exits successfully. * refactor(proto): Move providers to cloudprovider package * refactor(proto): Make Spinnaker nested under front50 This message is called 'Spinnaker' but the only thing that unifies what is in it is that it contains the things under 'spinnaker' in the front50 config, which makes it seem reasonable to nest it under front50. * refactor(proto): Misc cleanup on protos Add a few missing comments and alphabetize imports. In protos that only contain other non-primitive types, I didn't add a comment on each field. (Ex: Given that the Clouddriver proto has a KubernetesProvider field that is itself commented, it seemed redundant to also add a comment inside the clouddriver proto.)

feat(proto): Use protojson to unmarshal protos (#40) * feat(proto): Use protojson to unmarshal protos Instead of relying on the struct tags that are added by protoc-gen-go, use protojson to marshal/unmarshal objects that implement the proto.Message interface. This has the advantage of correctly marshaling/unmarshaling enums, as well as getting us compatibility with the standard JSON format for proto messages. As there is no standard YAML format for proto messages (and YAML is a superset of JSON), there is no protoyaml package. Make a thin wrapper that converts YAML to JSON before using protojson (and the reverse for marshaling). This is effectively what our yaml.Unmarshal was doing before under the hood (to use the JSON struct tags) so the JSON <-> YAML conversion is unaffected, it's jsut hat the JSON <-> Object conversion now uses protojson instead of go's built-in json. * feat(proto): Add enum types Now that we can marshal and unmarshal enum types, replace types that were made strings uniquely to work around the unmarshaling issues. * chore(kubernetes): Remove version flag from Kubernetes provider The V1 provider will be removed from the codebase by the time that kleat is ready for use. I didn't renumber all the protos to fill the missing number to reduce the size of the diff (and because at some point we'll have to stop doing this). Maybe we can do one last cleanup before some kind of 'alpha' release.

2020-04-27

refactor(proto): Clean up name of protos (#39) * fix(proto): Update proto package names Per the proto style guide, the package of a proto should be the directory in which it lives. As a follow-up to splitting the protos into directories grouping common functionality, update the package in each directory. * refactor(proto): Clean up name of protos Now that the protos are in their own directory/package and are referred to by their package name, we can remove redundant parts of the message name. This also helps generate client code that is more in line with go style. For example, a GCS artifact account is now referred to as 'artifact.GcsAccount' instead of 'artifact.GcsArtifactAccount'. I adopted a style where the top-level proto configuring something lives in that package with (usually) the same name as the package. For example, artifact.Artifacts, ci.Ci, etc. I also changed the message for the Google cloud provider to GoogleComputeEngine to avoid confusion with other Google providers.

2020-04-24

refactor(core): Tidy the proto directory (#37) Move notifications into their own directory and split then up into separate fields. Also, un-nest some places where we still had nested protos.

refactor(proto): Organize protos into directories (#35) This is a first pass, there are still a couple of places I'd like to reorganize but I'd like to get feedback on this first (particularly given the large diff and potential merge conflicts). I grouped the proto fiels into logical groups an put them in their own directories, removing rendundant parts of the filenames. In the case of the top-level configs, I removed Config from the name because go style prefers shorter names and 'config.Front50' is just as clear as (or clearer than) 'config.Front50Config'. We could think about whether to shorten the names of other things as well but I haven't done that in this PR (and they are used less frequently so the diff will be smaller).

feat(pkg): export HalToServiceConfigs method (#34)

feat(front50): Add azs and oracle persistent stores (#33) This also renames the GCS message to GcsPersistentStore to be more in line with how we've named other things. It also removes the top-level persistentStore field, and requires each persistent store to be marked as enabled or disabled (with exactly one being enabled) to match what front50 expects. While this is a slight change, hopefully it will be easy to configure as the kleat config will only require the single one that's being used to be in the config (unlike Halyard which will add fields for everything). I didn't add tests here, as we're about to do some refactoring and potentially will clean up the tests. I'll add tests on this when finishing up the rest of the persistent stores.

feat(echo): Add GCB account and stats (#32) * feat(echo): Add GCB account Echo requires GCB accounts to be in its config file, as it listens to their configured Pub/Sub topics to respond to changes in build status. This commit adds top-level CI field to the hal config, and implements support for GCB accounts. Support for other CI accounts will be done when we implement the rest of the igor config. * feat(stats): Add stats to echo config For now this only passes through whether stats are enabled or not; we'll need to think more about how to handle generating a unique id for the instance, and passing through the Spinnaker version. * fix(gcb): Add comments to GCB protos Co-authored-by: Maggie Neterval <mneterval@google.com> Co-authored-by: Maggie Neterval <mneterval@google.com>

2020-04-23

feat(echo): Add pubsub config (#30) * feat(echo): Add pubsub config This commit adds pubsub config support to kleat. Currently only Google Pubsub is supported in Halyard, so this commit only adds the same to kleat. * fix(echo): Fix typo in pubsub config Co-Authored-By: Maggie Neterval <mneterval@google.com> Co-authored-by: Maggie Neterval <mneterval@google.com>

feat(echo): Add notification config (#29) * refactor(core): Use sigs.k8s.io/yaml instead of ghodss/yaml There has not been a release of ghodss/yaml in more than 3 years; it looks like kubernetes has started maintaining a fork of it with more recent releases, so let's use that. The motivation for this is that I want to use UnmarshalStrict in tests so they catch additional fields that shouldn't be there, but that is not present in the latest release of ghodss/yaml (though is on the master branch there). * feat(echo): Add functionality to generate echo config This commit adds an empty echo config, and adds the functionality for kleat to generate an echo config, and a basic test on that config. The code to generate the config (as well as the test) is somewhat duplicative of the code for clouddriver. For now I'm deferring cleaning that up into better reusable components, but will do that before we add a third microservice. * feat(echo): Add notification config This commit adds support for configuring notifications via the halconfig, and having those be written to the echo config. As artifacts are just passed-through to echo, and the clouddriver work already strongly-typed this, this commit also passes artifacts through to echo. * fix(echo): Address PR review comments I guess I'll have to regenerate the protos manually and push another commit after this one. Co-Authored-By: Maggie Neterval <mneterval@google.com> Co-authored-by: Maggie Neterval <mneterval@google.com>

2020-04-22

feat(docs): document poor performance of sortTagsByDate flag (#28)

2020-03-30

refactor(core): Add package for generated protos (#20) protoc-gen-go recommends adding a go_package option to specify the package where output code should be placed. For now, just setting the package to "client" to match the directory, but now that we have a lot of protos, we should probably think about how to better organize this directory.

2020-03-17

feat(proto): add artifact protos for Helm, HTTP, Maven, Oracle, S3, and templates (#17) * feat(proto): add Helm artifact provider protos * feat(proto): add HTTP artifact provider protos * feat(proto): add Maven artifact provider protos * feat(proto): add Oracle Artifact Provider protos * refactor(protos): disambiguate providers from artifact providers * feat(proto): add S3 artifact provider protos * feat(proto): add artifact template protos

2020-03-16

feat(proto): add GCS, GitHub, GitLab, and Git repo artifact provider protos (#16) * feat(proto): add GCS artifact provider protos * feat(proto): add GitHub artifact provider protos * feat(proto): add GitLab artifact provider protos * feat(proto): add Git repo artifact provider protos

feat(proto): add bitbucket artifact provider protos (#15)

2020-03-11

feat(proto): document or remove Halyard-defaulted values (#14) * feat(proto): document or remove Halyard-defaulted values * fix(docs): reword defaulting note in README

2020-03-10

feat(protos): add DC/OS, Docker Registry, ECS, Huawei Cloud, and Oracle protos (#13) * feat(proto): add DC/OS provider protos * feat(proto): add Docker Registry provider protos * feat(proto): add ECS provider protos * feat(proto): add huaweicloud protos * feat(proto): add oracle protos * refactor(examples): move halconfig.yml and clouddriver.yml into examples directory Per the [recommended golang project structure](https://github.com/golang-standards/project-layout/tree/master/examples), consolidate example input and output into `examples` directory. * refactor(proto): rename persistentstorage to persistent_storage per proto style guide Per Protocol Buffers Style Guide, all proto file names should be [lower_case_snake.proto](https://developers.google.com/protocol-buffers/docs/style). * refactor(proto): specify Provider in top-level provider message names In order to keep each message name as specific as possible, renames each cloud provider top-level provider message to include Provider. For example, `Kubernetes` is now `KubernetesProvider`. * fix(protos): correct typos, fix wording

2020-03-04

fix(proto): consistent spelling of Cloud Foundry

fix(proto): remove license headers

feat(proto): add Cloud Foundry protos

2020-03-02

Add copyright header to all files (#10) I've followed the recommendation of the CDF regarding copyright headers, and attributed it to "The Spinnaker Authors" without a year.

feat(proto): add AWS and Azure provider protos (#9) * refactor(api): clean up protos * feat(proto): add AWS provider protos * feat(proto): add Azure provider protos * Update halconfig.yml Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> * Update api/proto/aws.proto Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> * fix(proto): get rid of bad ’s Co-authored-by: Eric Zimanyi <ezimanyi@google.com>

2020-02-25

feat(appengine): complete appengine protos (#6)

2020-02-12

add providers.google proto (#5)

2020-02-06

remove markdown link

2020-02-04

Add clouddriver proto descriptions (#2) * straw-descriptions for kubernetes account protos * err if halconfig is not a file * Update proto/providers.proto Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> * Update proto/providers.proto Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> * Update proto/providers.proto Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> * Apply suggestions from code review Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> * Update proto/providers.proto Co-Authored-By: Eric Zimanyi <ezimanyi@google.com> Co-authored-by: Eric Zimanyi <ezimanyi@google.com>

2020-02-03

Provider version is a string

2020-01-30

Add google to providers

Finish kubernetes & add google accounts

2020-01-21

fix typo in protobuf name

2020-01-17

begin clouddriver config

2020-01-16

Split up protobufs and add front50 config

Move proto to its own folder

Initial commit