Proto commits in QMSTR/qmstr

These 97 commits are when the Protocol Buffers files have changed:

2020-01-09

[go-qmstr] add module client library Introduce client library for modules to access qmstr-master from modules without directly calling grpc functions.

The documentation is generated from this commit.

2019-10-25

[proto] Send all info nodes at once for each file node It's best practise if we have an info node message for each file node.

[qmstr-master] implement rpc get source file nodes Implement the RPC function to retrieve source filenodes from the graph. This will be called from analyzers.

[proto] Remove fileType from data model Remove the fileType field from data model. It is unreliable because it depends on the builders to be set correctly. Also the kind of information we need, e.g. what file node is a source, can be deduced from the graph.

Adapt report phase

Adapt analysis phase to the new fileNode structure Connect info nodes and diagnostic nodes to the file data nodes instead of the file nodes.

datamodel] timestamp -> string Queries silently change the value of int64 to float, probably on large integers due to overflow.

[datamodel] Seperate file data to another structure That way our data model represents how the filesystem works and reduces complexity. We can store multiple paths pointing to the same file data and keep track of file data that are changing but the path stays the same.

Introduce timestamps Introduce timestamps into the database structure

2019-05-06

Wait also waits for inserts to be done Since adding nodes from the commandline might reference nodes that are not yet written to database we need a way to make sure everything is in db without switching to the next phase.

2019-05-03

Remove BOM service The service is bulky and not scalable.

Add version field in the package node structure

Remove pkg analyzer Pkg-analyzer modifies the build database. That's not allowed anymore in other phases than the build phase. We can use the connect command instead to connect the targets to the packages.

Fix test reporter Add GetProjectNode rpc function in the report service, so it can be available inside the reporters.

Improve rpc Package function Give a more appropriate name to the function, that reflects what the function does. Also provide the package to which we are going to connect the targets.

Html reporter should not fail if a package does not exist Html reporter should create a BOM for each package. If no packages exist then it just returns.

GetPackageNode returns a stream of packages Since the database structure changed, we may have more than one package node in our database. GetPackageΝode returns a stream of packages. If we want to get a specific package node from the database, we can call the function parameterised by a package with the desired name. The function will return a stream with only the desired node. Otherwise it will return a stream with all the packages in the database.

Fix phase when importing snapshot Importing snapshot always activated the current phase. This lead e.g to activating the analysis phase and running all analyzers again when importing a post analysis snapshot.

2019-04-16

Merge in proto repo again Using a separate proto repo introduced more problems developing than it solved.

2019-02-13

Remove the proto files Proto files live their own repo and are picked up by the libraries.

This commit does not contain any .proto files.

2019-01-31

Remove session Session is not used anymore. Session was needed in the old implementation to refer to the package node or the database in general. Now we can just refer to the package node by quering for packageNodeType.

2019-01-25

Add sourceURL field in PackageInformation (#205) This field is to be set by the user (or tool?) in the "metadata" section of the QMSTR yaml config file and it should point to the source of the package being analyzed For example: * http://ftp.gnu.org/gnu/glibc/glibc-ports-2.15.tar.gz * git://git.myproject.org/MyProject

2018-11-30

Use proto enumeration to restrict file type declaration Use enumeration to restrict arbitrary strings in filenode's type field.

2018-10-30

Html reporter shows path not name Use path as we may have targets with the same name and we won't be able to distinguish between them.

2018-10-19

Export and import pushed files Include uploaded source files into the snapshot and restore on import.

Actually write the uploaded data to a file

Actually send the file

Provide PushFileMessage from gcc builder

Add rpc function to push files to the server Upload files to the server. This will be used when the builder captures code from stdin to persist it for analysis.

2018-10-12

(Preparation) Restructure package nodes inserts Package analyzer should send a package node message. It's a preparation step to allow Missing Pieces analyzer to add/append a file node and not get mixed up with the package analyzer.

2018-10-04

Distinguish error and warning in builder Do not create errors for mode not implemented in gcc but produce a warning.

Await pending inserts on export Await the pending inserts to be written to the database before exporting snapshot to make sure having a consistant snapshot. Also use event classes from protobuf.

Don't break the build if qmstr fails Instead of breaking the whole build insert error messages into the graph (package node) to signal errors in the builders.

2018-10-01

Import export logic Persist session and phase in database Export database via snapshot command Load snapshot on startup via --seed flag on start command

Overhaul protobuf/grpc code generation In order to get correct import statments for python3 right now the only way is to have the proto files in a directory structure that corresponds to the module path. Of course this also has affects the generated go code.

Change nodeType to per node predicate Remove nodeType that needs to be set correctly to a predicate named like the actual message. This makes it easier to query and filter by that predicate without looking up the corresponding int to the nodeType. Also this is less error prone because the actual value assigned is irrelevant. Clean up database code To have the node predicates serialized when writing to the database a dummy value is inserted automatically.

define server phase in protobuf Change server phase into enum defined in the protobuf data model. This way it needs to be done once and not clashes can occur.

2018-08-24

Fix versionidentifier Calc short id on the fly. Remove remotes/<remotename> from description. Also show full identifier in detail pages.

Create reports for build configurations Add the build configuration to the taxonomy of the html report.

Store the buildconfig in the package node Store the config in the package node to access it later.

Fix summary rendering Escape double quotes in the summary. Also do not store summary in a field as it is nothing but a short version of the already stored message.

2018-07-13

Use the right proto files Do not use the copied protobuf files anymore.

Add gradle plugin stub The plugin connects to a running qmstr-master and sends some remote log messages. The project is build with gradle and compiles the gRPC java code from the provided proto files (those are copied from the qmstr repository and we need to find a solution to share those files). For convenience the build can create idea project files and install the resulting plugin in your local maven repository.

2018-07-05

Add dependencies to BOM and htmlreporter Add the dependencies to the bill of material and render it to html report.

2018-07-04

Find declared license and map to package info

Implement GetBOM Add bom and corresponding rpc function

2018-06-20

Distinguish between rpc failure and modules failure Print out better failure messages and publish more events to improve feedback to the user.

2018-06-18

Check for type selector in the configuration Package analyzer connects the package node with linked targets but in the case of jabref we want to connect the package node to a jar file. In that case we provide a type selector to override the default type.

2018-06-13

Add info data query

Fix adding file and info nodes

fix path substitution

2018-06-12

Send filenode rpc function definition

Let analyzers send back info nodes

Merge remote-tracking branch 'origin/wip/giasemi/85_getFileNodes' into demodemo

Merge branch 'wip/markus/83_sending_infonodes' into demodemo

2018-06-11

Adapt new implementation of quering file nodes Analyze function is not anymore called for each file node. Now it's more generic and can be used for the analysis of both file nodes and package nodes.

Implement file node query Add an rpc function which queries depending on a filenode and returns a slice of filenode. This way we can query for different attribute sets of a filenode.

2018-06-08

Add token to request and analyzer to infonode Add missing parts: - token for enforcing analyzer order - analyzer node to infonode

2018-06-07

Add SendInfoNodes grpc function

2018-06-06

Implement Confidence score Confidence score is used as a normalised ranking of info nodes.

2018-06-05

Implement trust level Trust levels are configured and assigned to analysis modules to specify each module's reliability.

2018-06-01

Introduce event publication and subscription In order to give more feedback to clients the master will emit events that the clients can subscribe to. This will be used to show more out put on long running qmstrctl commands like analyze and report.

Add error message to status

2018-05-28

Add status command repair wait Repair the wait command to wait until init phase is done. Add status command to print out the current master server phase.

2018-05-22

Add the connection between targets and package node in the DB Insert the package's targets in the database.

Move package node access to control service We want to be able to have access to package node in all phases.

2018-05-09

Move proto buffer files to generic location Since the protobuf files will be used to generate java and python code they should not be in a go specific directory.

2018-05-03

Inject output directory into module configuration For each module inject an output directory into the module configuration.

2018-04-30

Pass name to modules via configure

Pass cachedir via configuration response

2018-04-17

Fix reporting rpc Only one response will be sent. Also adapt function name to reflect its purpose.

Map project metadata to package data in html reporter

Introduce session id To support more than one package node add a session id to the master and the packagenode.

Pass package name to html reporter

2018-04-10

Use constants for nodetype To not use magic numbers use constants for the nodetypes

Add declaration for package node. The package node will reference all targets build as well as metadata about the build e.g. a name, declared license and project URL.

2018-03-27

Introduce reporting service Add implementation of the reporting. Similar to the analysis reporters are configured via the global qmstr configuration and are started from the master server.

2018-03-14

Migrate scancode analyzer Migrate the scancode analyzer to the new service model. Improve query for infonodes to just one query, to let the database do as much as possible.

Implement analysis phase Implement analysis phase and adapt spdx analyzer to it.

Implement build phase Implement the build phase and adapt gcc-compiler to it.

Add node type predicate

Adapt CLI to new service model

Introduce new service structure and datamodel Introduce new service structure with separate service definitions for each phase (build, analyze and report)

Change data structure Create dgraph data with protobuf so they can be used from other file types. Also data structure is now more generic.

2018-02-22

Service definition for using global config Since the analysis and report will be defined in a global configuration file passed to the master at runtime the Analyze and Report functions do not need to pass this information.

2018-02-19

Add reporting to buildservice Add report function to build service definition.

2018-02-14

Do not fail if file is not hashable Insert dummy hash but do not fail if hashing fails.

2018-02-08

Add path substitution to analyzer message Since this might be usabe for all analyzers make it part of the AnalysisMessage and not the generic configuration.

Run the analyzers - Add very basic spdx identifier analyzer that finds SPDX-License Identifiers in the first 100 lines of a source file. - Run analyzers on all nodes that are selected via the node type.

Add analysis Add Analyse function stub generalize database queries introduce function to get all nodes of a certain type from the db

Add Analysis definition

2018-01-30

Store gathered information in dgraph Calculate hashes for all files to identify them later. Find link libraries to also calculate hashes for them and store them as dependencies for linked targets. Store dependency information in the graph database.

2018-01-29

Handle linking libraries To not overload the wrapper with work collect information about linking libraries and search paths only and send to master server. Deeper analysis of the linked libraries is master server's duty.

2018-01-23

Add gcc compiler analysis Add functionality to analyze gcc commandline, create a BuildMessage and send it to the master server.

Introduce ability to shutdown master from client

2018-01-22

Introduce remote logging Extend build service to send log messages to the master server from the wrapper. Implement io.Writer to be used as logging sink that sends all logs via the buildservice's Log function.

2018-01-19

Add qmstr-master server stub Add a stub for the master server that instructs the generation of the grpc service code.