Proto commits in scalableminds/webknossos

These 99 commits are when the Protocol Buffers files have changed:

2025-04-24

implement hideUnregisteredSegments and only-show-this-segment etc in context menu

remaining actions

more user state update actions

rename userState to userStates in proto; implement TreeGroupsExpandedState, UpdateTreeVisibility

2025-04-22

snapshots

renamings

backend: add UpdateSegmentVisibilityVolumeAction, UpdateSegmentGroupVisibilityVolumeAction

include userState in skeleton+volume proto

2025-04-17

WIP: Per-User State in Annotations: Camera, Visibility Toggles

2025-04-03

Improve volume and segment index performance (#8460) * WIP: Improve segment index performance * implementation with triple cast on every element, not faster than scala * measure what else is slow * wip cleanup, measure other stuff * wip: use volume bucket buffer, fossil multi-put * logging * WIP rewrite segmentIndexBuffer to include caching, use Set * use new segment index buffer functionality * WIP fossil multi-get for buckets * fix setting volumeBucketDataHasChanged * do not parallelize prefill * use batched multi-get and multi-put * connect to temporaryStore, compress if needed * load buckets from temporarystore, remote fallback layer * load buckets from fallback layer in one request * some cleanup * readerOnly segmentIndexBuffer * cleanup * cleanup * format, remove logging * debug wrong volume values for editable mapping (id seems to be mapped with outdated mapping?) * request correct version of editable mapping data * changelog, migration * cleanup * make cpp types more explicit * clean up cpp code * const, try/catch * clang-format * size_t for index * Do not cache EditableMappingBucketProvider across versions * pr feedback part 1; consistent volumeLayer naming * use fossil multi-get when requesting multiple segmentIds from segment index * use map for more efficient lookups during gathering segmentIndex values * add one more conversion from set to seq for less map overhead * Update webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeSegmentIndexBuffer.scala Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * Update webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeSegmentIndexBuffer.scala Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

The documentation is generated from this commit.

2025-03-26

use batched multi-get and multi-put

The documentation is generated from this commit.

2025-03-24

wip: use volume bucket buffer, fossil multi-put

Distributed Tree Storage (#8423) * WIP: distributed skeleton tree storage * add count_skeleton_keys tool * implement loading and saving with external skeleton bodies * changelog, migration guide * migrate trees previously stored in the main tracing * add missing shutdown * remove unused imports in count skeletons script * pr feedback, take care of revertToVersion with distributed skeletons * typo * Fix duplicated line in nmlgenerator * do not try to flush deleted trees * extract the info if skeleton is stored with external tree bodies to single variable. reset it during duplicate and merge * remove hasExternalTreeBody tree property, use centralized one for all * cleanup, unify save method signature * refresh e2e snapshots (new bool in skeletons) * add storedWithExternalTreeBodies to frontend skeleton type * fix js type * use proto index 1 and 2 * add wrapper class SkeletonTracingWithUpdatedTreeIds to avoid the tuple --------- Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com>

2025-03-20

Support rendering more dtypes (#8325) * adapt histogram code in scala to support int8 * avoid copying bucket data on receive if it's unnecessary * use ACE for shader editor and make the editor movable * make int8 data layers work * format backend * also support int8 segmentations (including rendering of negative values) * fix ad hoc meshing for negative segment ids * support uint32 color layers * fix inverted supportFractionals interpretation in histogram view * remove unused getRgbaAtIndex implementation * make straightforward adaptions for int16 * tmp: don't sanitize layer names and auto-start shader editor * wip: make dirty adaptions to read int16 data in shader * wip: support int16 with packingDegree=2 * refactor * more refactoring and fix some dtypes * also allow local puppeteer execution * add new screenshot tests for dtype datasets * speed up new dtype screenshot tests by parallelizing them * fix typing * format * also support int32 color textures * fix rendering for very large float values * more refactoring in preparation of segmentation tests * tmp: restrict screenshot test on new specs * also add segmentation tests * handle segmentation ids as uint in shader instead of vec4 (negative ids are handled as if they would be positive) * add test capability for special segment ids * tmp: disable olvy * fix all the things :tm: * iterate on tests * format * clean up and fix typing * fix int32 segmentation (and add segment test) * fix uint32 segments * don't store 3D viewport in dtype screenshot tests * avoid page reload between specs when the same DS is tested * add/fix (u)int64 support and add tests * clean up * refactor screenshot test parameters * more clean up * more clean up * refactor getRgbaAtXYIndex and fix the case for multiple data textures per layer * simplify enabling/disabling name sanitizing for glsl texture names * remove more obsolete code * improve comments * re-enable olvy * linting * remove channelCountToFormat heuristic in favor of explicitly defining pixel format * also set alpha for segmentation layers in tests * more clean up in tests * readd noUnusedImports in biome * remove unused imports * remove unused getChannelCount * refactor packingDegree * add dtype screenshots * fix ad hoc meshing in backend for remaining dtypes * adapt nightly action so that it also triggers the screenshot tests if the commit contains a specific phrase (then it will be executed on the current branch) * disable most ci checks * skip rest of workflow properly * rely on manual triggers instead * sanitize subdomain differently than tag name * undo partial screenshot tests * add await to page.close * fix histograms (necessary for screenshot tests to work) * make td controls invisible in screenshots tests * move checkBrowserCredentials * properly close puppeteer pages * make buttons invisible when screenshotting 3d viewport * prototype hardcoded permutations * add int data types to supported segmentation dtypes * add default intensity range support to backend * WIP: Support more dtypes in thumbnail service * fix default intensity ranges * remove debugging helpers * fix backend warning * also use precomputed frequency permutation and refactor * revert default intensity range for int64 and uint64 layers to 0, 255 as this is consistent with the frontend and add comment accordingly * refactor the remaining pattern calculation in shader * update most screenshots * temporarily disable ci * fix param in evaluate * rename method * fix uint24 rendering * fix uint16 * update screenshots * reorder (u)int types * incorporate pr feedback * DRY some test code * move disableLayerNameSanitization to dev flags * incorporate pr feedback * linting * update screenshots * replace ROI2017_wkw_fallback variant with no-fallback * remove unused screenshots * debug output amount of changed pixels * bump allowedThreshold to 0.5% for screenshot tests * hopefully fix wrong segment rendering on some hardware * bump one last time again :tm: * update screenshot * clean up * re-enable ci * remove superfluous (and incorrect) extrema definition in FindDataService.scala * update changelog * use correct value range for segmentation layers when dealing with largest segment id * sort imports * fix linting * fix comparison * fix editing largest segment id * fix tests * avoid null in file name and use without_fallback instead * save database access * support additional volume dtypes in proto serialization and deserialization * reduce number of field number duplications for ElementClassProto * disregard bytes per element in proto serialization and deserialization * support merging signed volume annotation layers * fix toString & equals copy past mistake in new signed Segment Integer implementations * format backend * fix picking and setting negative segment ids * improve logic of warnAboutInvalidSegmentId saga * fix crash when brushing in int64 DS * fix redundant line in changelog * fix endless recursion * Remove compilation errors -> make elementclass fromProto case complete * catch case unrecognized explicitly in elementclass fromProto * make ElementClass.toProto return a box with a failure in case of unsupported dtypes (floating point types) * format backend * remove elementClassToProto implicit --------- Co-authored-by: Michael Büßemeyer <frameworklinux+MichaelBuessemeyer@users.noreply.github.com>

2025-03-17

Merge branch 'master' into proofreading-identity

2025-03-04

Look up datasource infos by annotationId rather than tracingId (#8405) * WIP: look up datasource infos by annotationId rather than tracingId * propagate annotationid to more spots * in task creation, set ids on wk-core side * iterate on task creation * iterate on task creation * tasks * pass in Boxes again * more task creation * tasks * fix more usages of annotationId * save annotation base * fix compiler errors * no need for dataset name in annotation proto * use passed ids in tracingstore * fix task creation from files * fix merging annotations * fix duplicate, merge hybrid * fix merging editable mappings * changelog * fix compound annotation view * implement pr feedback * fix task creation with base annotation

2025-01-28

Merge branch 'master' into proofreading-identity

2025-01-27

Unified annotation versioning (#7917) * unify logging * cleanup * init fossildb migration * update frontend to newest annotation stats schema and fix typing & linting * include annotation stats per layer in time tracking api response json * continue if a layer’s stats couldn’t be updated * add some function stubs to migration * wip migration: read annotations from postgres * fetch annotation info from postgres * iterate on migration * wip migrate updates * migrate updates in stupid concat strategy * migrate skeleton + volume proto * restructure migration code * cleanup * add two frontend TODO comments * migration: renamings for update actions, insert actionTracingId * migration: add generaetd code * migration: set tracing proto fields, create annotationProto * migrate volume buckets * migration: annotation layer name, volume segment index * migrate editable mapping distributed elements * migration: remove morton index, migrate editablemappinginfo objects * migration: switch from orjson to msgspec * migration: add cli args, actually write to target fossilDB * parallelize, add postgres connection parameter * avoid redundant calls in task creation (regression introduced in cba70af92fd0f02d08ee250341461fa985142324) * migration: humanize time diff add previous-start argument * migration: wip handle compacted update actions, add error logging + counting * fix removeAllButLastUpdateTracingAction * adapt some todo comments * change renameTracing to use old editAnnotation route * more comments * fix merge conflicts * fix types for editAnnotation * handle UpdatTdCamera updates, migration: mark more as compacted * fix parsing CompactVolumeUpdateActions * fix migration for proofreading, add some more error messages to backend * when initializing an annotation in the front-end use annotation.version for that * remove max handling of versions * wip: allow updating annotation name from dashboard * migration use passed pg password, add ETR, remove debug logging * fix version handling in model initialization etc * adapt the types so that update actions that need isolation cannot be grouped with others * remove downsampling feature volume annotations * use editableMappingId == volumeTracingId constraint * fix deletion of layer * fix importVolumeData * improve labels for proofreading entries in version restore view * remove/fix some more todo comments * pass tracingId directly to update actions instead of magically adding it later to some (but not all) actions * get rid of unused-tracing-id and superfluous passing of relevant tracing ids * fix stuck 'Activating Mapping' message * do not include name in annotationProto and updateAnnotationMetadata action. make description non-optional, emptystring means empty * add version parameter to editableMapping agglomerateIdsForSegments * option to count versions * migration: add verbose flag, log also to file * add error for missing layers * migration: randomize order with md5 hash * get rid of tracing.version in frontend * fix linting * remove remaining version properties * fix save saga spec * fix skeletontracing saga spec * mock annotation proto in specs * fix wkstore adapter spec * fix more incorrectly passed transactionIds; fix save_reducer spec * fix last frontend tests :tm: * fix linting * format backend * WIP: migration: merge updates in a less naive way * incorporate version parameter in agglomeratesForSegments route; refactor some GET parameters in api module * migration: iron out reverts, merge updates by timestamp * migration: adapt to changed annotationProto format * small cleanup in migration * fix version restore preview for editable mappings * fix that restoring to another version broke the loading of mesh files * fix ordering in updateActionLog * migration: wip select which to materialize * reload page when switching to a version in the version restore view that has different layers (including debugging code because it does not work properly) * revert temporary sorting fix in FE * respect earliestAccessibleVersion in version restore view * fix editing of annotation name * fix that newest version was also requested even when no annotation exists (e.g., in dashboard or in DS view mode) * fix tooltip for segment stats * require passing annotationid to some tracingstore routes * Revert "require passing annotationid to some tracingstore routes" This reverts commit 856f11b9f52b041280155b7c9f8d18af354aff9f. * introduce skeletonMayHavePendingUpdates property to annotation proto to handle special case in migration * Revert "Revert "require passing annotationid to some tracingstore routes"" This reverts commit f24d12357a5fd15a6913a157322582d81de725c0. * set skeletonMayHavePendingUpdates to None after applying updates * migration: set the bool only where it makes sense * migration: set earliest_accessible_version for editable mapping annotations * don't fail on loading addSegmentIndex and compactVolumeActions * checkpoints for migration * cleanup in migration * pass annotationId to routes that require it (back-end cannot find the annotation on version restore, though) * also require annotation id in volume data request * fix annotationId param for data route * use annotation id also for access request * fix switched ids * adapt e2e tests * unused import * fix that version restore view didnt open when loading a version that had no skeleton layer; also use ensureWkReady in most places instead of actively waiting on wk-ready action * remove unused imports * use toasts related to proofreading undo * add custom start time argument to migration * set editableMappingsMayHavePendingUpdates in migration * fetch extra editable mapping updates if needed * fix downloading older volumes. changelog + migration guide. * fix assertion in volume download * remove debug logging * fix typing and CI * fix requesting volume data from older version * if allDataZip is requested with annotationId, use that one also for the access check * fix test * clean up editAnnotation * rename to getUnversionedAnnotationInformation * fix annotation stats and linting * fix superfluous request with empty annotation id * fix type check test * resnap freshshots * add e2e test for updating metadata * clean up typing of update actions * update todo comment * unused import * save one lookup of annotationProto * fix sandbox view * more verbose bucket loading errors, include Failure message * set editableMappingsMayHavePendingUpdates to None after are updates are applied * implement migrationf feedback part1 * Update MIGRATIONS.unreleased.md Co-authored-by: Norman Rzepka <code@normanrz.com> * wip: clean-up older migration run in case of reverts. use new fossilDB putMultipleVersions * migration: fix cleanup of previous, use multi-version put * pin versions in requirements.txt * iterate on migration guide * bump fossildb version to 0.1.33 (master_504) * migration: fix mapping id lookup * rename type to typ in AnnotationLayerProto and DeleteLayerAnnotationAction * fix applying extraSkeletonActions on older requested annotation versions * remove debug logging * WIP: switch migration from threading to multiprocessing * re-add isViewOnlyChange for skeleton actions where it got lost * reformat comment for TokenContext Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * unify function naming in RpcRequest * fix userToken param, remove stateless services from Singleton list * rename some functions, remove unused imports * fix bucketMutatingActions lookup, rename emptystring to empty string * reorder functions in AnnotationWithTracings * correctly clear editableMappings map, fix typo * rename + rephrase some things for pr feedback * add some error messages for editable mapping routes * add comment on merging editable mappings, move up saveSkeleton call in mergedFromIds * Remove superfluous empty list in SequenceUtils.splitAndIsolate * extract EditableMappingMergeService * incorporate pr feedback * extract actionTracingId in spec * remove unused updateActionStatistics route * remove superfluous for/yield * fix merge conflicts * Revert "WIP: switch migration from threading to multiprocessing" This reverts commit a8babc04cba6ad2ff14e0cd9730234d5d6743884. * refresh snapshots after merge * WIP: make migration faster using multi-get/put * prepare merge for import sorting pr * fix merge related problems * make biome ignore some type imports because it produces invalid syntax, otherwise * more batching for migration * include actionTracingId, actionTimestamp in compact action writes * Fix annotation duplicate * decrease batch sizes to avoid grpc message size limits * Fix viewing compound annotations * improve test for regroupByIsolationSensitiveActions, add ordering assertion * bump fossildb to 0.1.34/master__510 * clear all tracings when initialization is (re-)initialized and put initialization actions into a special batch action to avoid inconsistent inbetween-states * block UI when previewing or restoring to another version * fix brushing by waiting for init volume action * fix that the actual restore button was missing in the version restore view (only download was possible); fix achieved by moving isWkReady to store and conditioning the view on that * additional logging if verbose * use version parameter when requesting mapping (frontend and backend) * store initialAllowUpdate so that we can clearly revert that boolean after closing the version restore view * add some error messages, migration: attempt with lower batch size for segment index * auto-expand version group that contains the current version * fix closing version restore view when the currently previewed version has a different layer set from the newest version * lint * when reverting distributed elements of volumetracings/editableMappings, handle the layer not existing before the revert * incorporate pr feedback * rename function (pr feedback) * fix ordering of extra updates * append rather than prepend, pendingAnnotationupdates have higher version numbers * add explaining comment * put editable mapping updates to correct id when merging annotations * rephrase migration guide * fix duplicate updates if addLayer is in the history * don't access dataSource.id.name (fixes toast when previewing older version) * clear potential get variables after closing version restore view * refactor how allowUpdate is reset; fix incorrectly captured initialAllowUpdate value in useWillUnmount * restore modified condition * use UnsupportedOperationException in VolumeTracingLayer * format * unused import * fix type error --------- Co-authored-by: Michael Büßemeyer <frameworklinux+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: Norman Rzepka <code@normanrz.com> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com>

in backend, make baseMappingName optional

2025-01-02

fix bucketMutatingActions lookup, rename emptystring to empty string

2024-12-16

rename type to typ in AnnotationLayerProto and DeleteLayerAnnotationAction

2024-12-12

unused import

2024-12-11

set editableMappingsMayHavePendingUpdates in migration

2024-12-09

set skeletonMayHavePendingUpdates to None after applying updates

introduce skeletonMayHavePendingUpdates property to annotation proto to handle special case in migration

Revert "require passing annotationid to some tracingstore routes" This reverts commit 856f11b9f52b041280155b7c9f8d18af354aff9f.

require passing annotationid to some tracingstore routes

2024-12-04

Merge branch 'master' of github.com:scalableminds/webknossos into unified-annotation-versioning

2024-12-03

Warn user when using precomputed meshes after brushing (#8218) * Add key in volumetracing to store if volume bucket data has changed * show warning in context menu for loading precomputed meshes when bucket data has changed * avoid unnecessary rerenders in context menu * ensure SET_VOLUME_BUCKET_DATA_HAS_CHANGED is dispatched when buckets are changed * add missing file * fix linting * rename load_mesh_menu_item_label file * use action creator instead of hardcoding action * fix that volumeBucketDataHashChanged was not handled when the server sent it * add comments * update snapshots * add changelog entry * remove unintentional changelog entry * change action property from layerName to tracingId --------- Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Michael Büßemeyer <frameworklinux+MichaelBuessemeyer@users.noreply.github.com>

2024-12-02

do not include name in annotationProto and updateAnnotationMetadata action. make description non-optional, emptystring means empty

2024-11-18

cleanup

remove outdated comments

2024-11-05

earliestAccessibleVersion; always fetch all layers; wip resetToBase

2024-10-30

rename annotation.proto layers to annotationLayers, simplify duplicate api

2024-10-28

spelling of layer type proto

2024-10-23

Merge branch 'master' into unified-annotation-versioning

2024-10-21

Rename Resolution to Mag (#8111) * start to replace term resolution * rename resolution to mag in backend where possible * rename resolution restrictions * rename resolution info to mag info * rename in dataset accessor, a.o. * rename in frontend code * rename in docs * rename node resolution to mag * adjust mag slider * fix tests * add explaining sentences * refresh snapshots * improve explanation in info tab and status bar * adapt param name * remove application conf edits * lint * rename columns + table in sql, add migration * adapt test db after schema change * changelog * slightly adjust changelog entry to lower expectations * adjust docs and revert some changes * adjust neuroglancer docs * Update webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeTracingDownsampling.scala Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * address review * fix previous version in db evolution * adjust version in schema.sql * also adapt schema version in reversion * adjust form names * adjust get params for /createExplorational * remove unused messages --------- Co-authored-by: Florian M <florian@scm.io> Co-authored-by: Florian M <fm3@users.noreply.github.com> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Tom Herold <tom.herold@scalableminds.com>

2024-09-24

Merge branch 'master' into unified-annotation-versioning

Metadata for Segments & Trees (#7875) * WIP: user-defined key-value properties for skeletons * adapt UpdateNode test * also for trees,segments. add to backend NMLWriter * parse string list values too * allow to create string tags for segments; styling tweaks * don't use inline tags; instead use a resizable two-split-pane and a table * tweak layout so that the details pane only occupies as much space as needed and not more than the user-specified (via drag) height * allow to edit values in segment details table * refactor * add property UI to skeleton trees * add missing file * fix tests * fix spinner margin in dashboard * allow user to resize pane larger than needed * fix styling in light theme * filter duplicate keys in action creator * implement NML parsing and serialization of user defined properties for trees * adapt parser and serialize for userDefinedProperties; add test * start unifying metadata tables * remove superfluous type wrapper * integrate dashboards metadata table into tree hierarchy view (styling + string-based props) * also integrate with segments tab * make adding and removing properties work properly * also support number and string array as property types * format * re-order props * extract InputWithUpdateOnBlur to fix cyclic dependency * temporarily disable ci * fix metadata editing for segments * change trash to close icon in dashboard * give metadata table rounded corners * warn user when keys are not unique; avoid frequent rerenders via memo * iterate on validation, styling and tag input * try to fix stringListValue: [] error * don't save duplicate keys in case the user ignores the validation errors * auto-focus just added key fields * remove context-menu way for adding metadata; iterate on icon in segment and tree list * change dashboard metadata table style * also adapt validation styling * make some props optional * move multiple-trees-selected note into details pane * make fields read only if no update permissions exist * fix theader style for segments tab * store height for details pane in local storage * fix lost highlighting of segments that are hovered in the data viewports * also print statistic of previous runs in wk dev benchmark * move isCentered logic into SegmentListItem to avoid rerenders * fix type error * fix linting * rename UserDefinedProperty to MetadataEntry * wrap metadata in parent tag in nml * rename userDefinedProperty/ies to metadata and metadata entry * clean up * rename metadata-table-wrapper class * more renaming * remove debug code * clarify comments * delete unused InputNumberWithUpdateOnBlur * minor clean up * re-enable ci * fix compact_toggle_actions.spec.ts and make it type check * update snapshots * remove todo comment * make tooltip color consistent orange * add tooltip to tag icon * Update frontend/stylesheets/_variables.less Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * integrate feedback * rename record to entry * adapt parseTimestamp to defaultValue options interface * also adapt _parseTreeType * rename MetadataEntry to MetadataEntryProto * also rename APIMetadata to APIMetadataEntry and add comments about the distinction * fix e2e tests * deduplicate metadata entries when applying update actions * changelog, deduplicate for segments too * remove unused imports in frontend code * integrate backend pr feedback * refresh snapshots * readd border around empty metadata placeholder in dashboard --------- Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Michael Büßemeyer <frameworklinux+MichaelBuessemeyer@users.noreply.github.com>

2024-09-13

Merge branch 'master' of github.com:scalableminds/webknossos into unified-annotation-versioning

2024-09-12

Merge branch 'master' into unified-annotation-versioning

2024-08-29

Make organization name its id, drop the mongo id (#7386) * WIP: make organization name its id, drop the mongo id * more usages * Update LokiClient.scala * fix compiler errors after merge * prune scala usages of organizationName * fix migration guide * prune frontend usages of organizationName * prune frontend usages of organization.name * fix data type of orga references * sql check, fix delete organization * WIP: Add migration and revision * format backend * enable backend to parse old analytics schema for ingest events * format backend * update schema version * finish first version of migration and revision * fix schema & migration - give explicit name to constraints to make the name more consistent and less implicit * add migration for analyticsEvents' eventProperties * add organizationName to LinkedLayerIdentifier to make it backwards compatible * add legacy route to list datasets supporting the old get param organizationName when filtering for specific datasets & bump api version * dont use direct get on box and throw jserror when no orgaid or organame is defined in analytics event * format backend * remove comments from evolution files & fix schema version * fix migration, remove old now duplicate unique index * readd unique contraint in reversion * in reversion: migrate local join orga, open dataset & uplaod dataset back * in reversion: fix fk constraints and views * in reversion: fix user infos view * migrate new code pulled in by merge to use orgaId and not name * format backend * update schema version & include warning in reversion * rename organizationDisplayName to organizationName * add changelog entry * add migration entry * remove auto generated message files for data and tracingstore * update e2e db fixtures / initial db data * rename left over orgDisplayName variable to orgName * update snapshots to use new e2e your name last names * fix typo and format code * more relaxed analytics json reads * drop default for id column * add missing redirect to versioned.routes * rename migration, undo changing worker job args --------- Co-authored-by: Norman Rzepka <code@normanrz.com> Co-authored-by: Michael Büßemeyer <michael.buessemeyer@student.hpi.de> Co-authored-by: Michael Büßemeyer <MichaelBuessemeyer@users.noreply.github.com>

2024-08-21

Merge branch 'master' into unified-annotation-versioning

2024-08-06

Refactor Skeletontree tab as Antd <Tree /> component (#7819) * refactor tree hierarchy view as functional component * WIP refacator as antd <Tree> * restore checkbox and expaneded keys * restored expand/collapse all behaviors * re-use <DotIcon> components but with rgba values * restore search * fix drag and drop for single tree * fix multi select * remove react-sortable-tree library * updated styling and text wrap * make tree height/width smaller * collapse sup groups * added shift selection to skeleton tree * updated changelog * Update frontend/javascripts/oxalis/view/right-border-tabs/skeleton_tab_view.tsx Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * Update frontend/javascripts/oxalis/view/right-border-tabs/tree_hierarchy_view_helpers.ts Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * Update frontend/javascripts/oxalis/view/right-border-tabs/tree_hierarchy_view.tsx Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * Update frontend/javascripts/oxalis/view/right-border-tabs/tree_hierarchy_view.tsx Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * apply PR feedback * fix tooltip * add autofocus to tree search * fix active tree updates highlighting tree * fix shift selection * fix crash on tree deletion * apply darker background colors for trees in dark mode * disable node drag icon * disabled checkboxes for empty groups * use darker colors for <Tree> components in dark mode * fix multi select drag and drop * formatting * fix CSS alignment of tree checkboxes * use antd themeing instead of CSS overrides * format * Persist skeleton group expansion state (#7939) * move skeleton expanded/collapsed group state to store * WIP: expand and collapse all skeleton groups * fix expand/collapse skeleton groups * expand group if tree is dropped * clean up code * lint * address review * address review 2/2 * remove console.log * lint * add small condition for ondrop * persist isExpanded state in backend * make groups default expanded if field is absent in nml (backend only) * make groups default expanded if for written nmls (backend only) * fix backend treegroup update action tests * add isExpanded to frontend nml support * update snapshots * omit object creation in var * omit useless return statement in map * add isExpanded prop to tree groups in backend testing fixtures (dummies) * add test to backend nml test to test properly setting default to expanded * add changelog * WIP: address review * fix bug where drag and drop of empty groups was failing * remove comment --------- Co-authored-by: Michael Büßemeyer <michael.buessemeyer@student.hpi.de> Co-authored-by: Philipp Otto <philipp.4096@gmail.com> * fix tree sorting * fix off-by-one error in shift select * override <Tree> scroll handle in dark mode * use scrollTo(auto) * format * fix typo --------- Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: Charlie Meister <charliemeister@proton.me> Co-authored-by: Michael Büßemeyer <michael.buessemeyer@student.hpi.de> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com>

2024-07-31

In Proofreading, Load Oversegmentation, Perform Merges Eagerly in Frontend (#7654) * add datastore route to list all agglomerate ids * remove unused file access * types * Apply hdf5 mappings in frontend [WIP] * Fix segment and agglomerate id mixup in proofreading saga * add route to tracingstore * edit mapping after merge and set it in redux store * Fix mapping initialization * WIP: implement split in frontend * Only load subset of mapping for segments visible 5s after page load * small fix * cleanup, time measurements, throttle mapping requests to 500ms -> currently too laggy * Update redux-saga to use throttle with effectChannels * Only refresh mapping if bucket data changed. Fix partial mapping updates after master merge. * Fix how proofreading actions update the mapping in the frontend * disable most ci checks * misc improvements * refine shouldUseDataStore logic * fix type error * fix some type errors * fix more type errors * use NumberLike type for number and bigint at various places * more NumberLike usages * fix lots of typescript errors (related to bigint and to redux) * fix more type errors * fix the last type errors * restore proper NumberLike definition and fix remaining type errors * remove unused imports * fix race condition which could cause mapping to not be properly initialized upon reload * only pass isCentered instead of centeredSegmentId to SegmentListItems * fix react warning when closing context menu for the first time * add debug code for rare scenario where segment id is not an integer * fix TS error * fix that hovered unmapped segment id would not update sometimes * avoid two map look ups * add dryUpdate step before saving proofreading update actions * undo provoking the error * make selective segment visibility in proofreading an option * fix NaN value after mapping unknown segment key * avoid parallel executions of updateHdf5Mapping and also cancel updateHdf5Mapping if wk enters a busy state to avoid using outdated values * always compare known segment ids to newest mapping instead of 'previous' one that isn't updated when doing proofreading operations * don't crash completely when segment mapping is not known yet and user initiates proofreading action * remove artificial delays * improve logging * fix ts error * remove debugging code for NaN mapped id * tweak hovering * rename MIN_CUT_AGGLOMERATE actions * adapt cutFromNeighbors to magic mapping approach * avoid roundtrip for mesh reloading by using newest mapping * tweak hovering (II) * extract code into gatherInfoForOperation * fix context menu for skeletons in 3d viewport * make agglomerate-skeleton-proofreading compatible with magic mappings * don't run rendering code for context menu when its not open * remove unused previousMappingObject * dynamically switch between local and remotely applied mappings when switching to/from proofreading tool etc. (unfortunately, buggy) * auto-reload page if dev-proxy fails * fix missing reload when disabling/re-enabling mapping (now too many reloads are performed) * extract finishMappingInitialization action * extract ensureMappingsAreLoadedAndRequestedMappingExists * rename mappingIsEditable to hasEditableMapping * introduce BucketRetrievalSource to better cancel/restart bucket watchers and reload the layer when necessary * introduce clearMappingAction * make sure that getBucketRetrievalSource doesn't create new instances all the time when multiple volume layers exist * move cuckoo modules into libs/cuckoo * use cuckoo hashing for gpu-based mapping instead of binary search (proper uint64 support needs to be tested) * add missing module * remove logging * make use of protobuf shortcut getHasEditableMapping * Update frontend/javascripts/oxalis/model/sagas/mapping_saga.ts Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * Update frontend/javascripts/oxalis/model/sagas/proofread_saga.ts Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * mappingIsEditable -> hasEditableMapping to fix compilation error * remove unnecessary braces * incorporate some pr feedback * more pr feedback * add comment to cuckoo table 64 bit * also implement cuckoo table for uint32 keys and values * fix invalid initialization of mapping texture when mapping is disabled * fix that mapping was applied twice * fix uint32 cuckoo implementation (still hardcoded to always use 32 bit) * use 64 bit look up when necessary in shader * remove unused mappingSize * avoid toolbar rerendering * only write necessary changes to cuckoo table instead of rewriting it from scratch every time * ensure diffing of previous and new mapping is fast be providing cached diff operation for which the cache is manually set by the mapping saga * eagerly compute value set when reasonable to avoid clustering of that computation and to improve FPS * add missing worker * implement maintenance of value set for entire data cube to avoid recomputation * Revert maintenance of value set because it didn't help with performance This reverts commit c7bb508be78302adb11f0321074322065db5ba78. * remove todo because it was tried in c7bb508be7 and reverted afterwards * remove some time logging * optimize/combine some set operations (2x as fast now) * move and rename diffSets function * further optimization of set operations (in total 4x faster compared to the initial version) * remove unused code * remove superfluous parameter * add renaming todo * delete create_set_from_array webworker as it didn't show an improved FPS rate (therefore, the overhead of passing the data doesn't seem reasonable) * delete debugging code from cuckoo modules * move attemptMappingLookUp glsl code * add some comments * reactivate CI checks * remove unused code * remove obsolete imports * simplify editableMapping check * fix ts problems * fix cyclic dependencies * fix 64 bit mapping rendering * remove UI warnings that the merger mode doesn't support 64 bit, because now it does * fix linting * fix proto related tests * fix more specs * fix pullqueue spec * refactor setNumberLike in cuckoo tables and update some todo comments * remove unused route, change tracingstore mapping route to proto ListOfLong * toSet * remove unused getAgglomerateIdForSegmentId * fix rendering bug outside of viewport on some GPUs * adapt agglomeratesForSegments route to new protobuf interface * link 64-bit issue (#6921) in todo comments * introduce mappingIsPartial uniform * mention #7895 in todo comment * remove commented code * avoid expensive console.log for large mappings * disable most ci checks * fix incorrect hash table size and use hashCombine twice to fix poor capacity utilization due to suboptimal hash distribution * fix cuckootable rehash (did redundant work) and adapt max iterations parameter * if many inserts are done for the mapping, flush the table at the end * fix weak hashing for power-of-two table sizes * refactor diminished hash capacity tweak; clean up and extend tests so that maxing out capacity is tested, too * clean up reloadData related code in mapping_saga * fix serializing bigint to protobuf long * fix toggling of json mappings * group consecutive actions in action logger middleware; add debug logging for dispatched actions * fix that forceful disabled tool wasn't re-enabled when possible (e.g., after toggling segmentation opacity) * mention 64 bit support issue in comment * fix broken mapping of ids by sorting the input ids for the server * add comment about sorting * don't map ids dynamically in segment list view (instead segment items are created with the mapped id if a mapping exists); see #7919 as a follow-up * test reaching critical capacity and remove todo comment * remove some todo comments regarding mapId code that might return unmapped ids if the mapping is partial (impact should be low, add comments for it) * remove more todos and fix toggling of hdf mappings when no volume tracing exists * cast to number when sorting bigint * use bigint in proofreading_saga where sensible and cast to number as late as possible (e.g., in action creators, in REST API etc) * disable more ci stuff * try to handle most ids as Number and cast to Number only when dealing with mapping object and buckets * show zoom warning for agglomerate files only when the mapping is applied remotely * fix that supervoxel highlighting of mesh stays active after leaving proofreading tool (fixes #7867) * fix that after changing the color of a mesh via the segments tab the mesh is always highlighted after initial hover (fixes #7845) * remove unused imports * fix proper type of values returned from getAgglomeratesForSegmentsFrom* * fix incorrect bigint check and refactor to avoid the same mistake in the future * fix unnecessary type adaption that failed on null * fix another sorting bug * integrate pr feedback * rename cuckoo table to CuckooTableVec3 * unify 64 bit todo comments * forceful -> forcefully * refactor eager value set computation * use 0s to initialize mapping uniforms * refactor/fix mapId logic for unmapped ids * fix selective visibility for alpha != 0.2 * only emit soft errors when a data value could not be mapped * fix mapping message hiding too early/never; fix disabled message in mapping UI * misc console stuff * also skip texture updates for cuckoo table when lots of unsets need to be done * show short user notification when segmentation layer is reloaded * highlight whole segment mesh on hover even when geometry is not merged (i.e., super-voxels are highlightable) if not in proofreading tool * use current mag when reading segment ids in proofreading (unless agglomerate skeletons are used) * remove last todo comments * update changelog * remove console.log * re-enable ci checks * fix linting --------- Co-authored-by: Daniel Werner <daniel.werner@scalableminds.com> Co-authored-by: Charlie Meister <charlie.meister@student.hpi.de> Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com>

2024-07-29

Merge branch 'master' into unified-annotation-versioning

2024-07-24

Save expanded/collapsed state of segment groups (#7928) * WIP: add option to expand and collapse all children of segment group * recusively open subgroups * add two menu entries and hide/show them as useful * remove comments * WIP: move expanded groups to store * WIP: allow expanding and collapsing groups again * keep state of collapsed or expanded subgroups if parents is collapsed * remove comments etc * improve comments * dont use treedata any more * persist expanded state of segment groups * fix bug that root group expansion collapsed everything * fix bug that groups are collapsed upon subgroup expansion * hopefully improve performance by updating only in getDerivedStateFromProps * improve some comments and var names, lint * improve performance * WIP: address review * add changelog * lint * add comment and rename method --------- Co-authored-by: Michael Büßemeyer <michael.buessemeyer@student.hpi.de>

2024-07-08

wip: revert editable mappings

sparse versions

2024-06-20

error case

update actions

2024-06-18

WIP: Unified Annotation Versioning

2024-03-11

Lock mapping once the volume annotation was changed by the user (#7549) * make segmentation output layer name for neuron detection configurable * pin mapping once user did some volume annotation * some minor clean up * use and enforce pin mapping action in backend * undo temporary changes * reload available agglomerate mappings names if not fetched yet * add changelog entry * fix tests * add test for pin mapping functionality * clean up code * fix assertion * disable proofreading tool in case a mapping is pinned and not editable * pin mapping upon interpolation & floodfill - show info toast upon pinning the mapping * update proofread tool disable message * fix tests * Ask user for confirmation before pinning mapping * pin mapping / no mapping upon first volume annotation action - Don't pin json mapping - Don't allow changing state of mapping while pinned is set * disable merger mode once a mapping is pinned * don't do quick select if user aborts the pinning of the active mapping * fix tests * improve user informing message * remove unused import * apply feedback * apply pr feedback * properly catch user abort via modal * Let server saved annotation take precedence over the URL configuration if mapping is pinned * pin "no active mapping" in case a JSON mapping is active & fix linting * disable volume annotations tools while json mapping is active and rename pinned to locked * small code clean up * rename pinned to locked also in backend + proto * apply feedback & further replace word "pinned" with "locked" * further replace "pin" with "locked" * fix bucket fill ignoring use decision whether to lock the mapping * format --------- Co-authored-by: Florian M <florian@scm.io> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Philipp Otto <philipp.4096@gmail.com>

2024-01-17

Next round Dataset spelling (#7558)

2023-09-18

Add option to hide tree edges (#7296) * add option to hide tree edges * rename variable * add edgesAreVisible in skeleton proto * new update action in backend * fix tests and add missing arguments * fix backend tests * set correct default for edge visibility in frontend * update doc string * refresh snapshots * apply pr feedback --------- Co-authored-by: Florian M <florian@scm.io> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com>

Add UI for segment statistics (volume and bbox) (#7249) * very much WIP: first steps towards route and frontend * add url params * show segment size in context menu without unit * start tests * add format in multiple dimensions (early implementation) and resp. tests * show volume in context menu of segment * add reload button for segment volume * add segment statistics entry in group menu and style icons * WIP: start implementing segment statistics modal * move statistics modal outside of context menu * show segment sizes in modal * lint * add table to statistics modal * export segment sizes to CSV * make linter happier before I leave * revert unintentional change * WIP: add segment bounding box route * calculate real segment bbox * performance optimization: filter out inner bucket positions * remove unused import * make statistics routes post, allow for multiple segment ids * rename method * do zero check without type conversion * Revert "do zero check without type conversion" This reverts commit 93647699ebb7e9a373f43e9f483c4233be768eec. * compare to segment id correctly * find group segments recursively * very much WIP: use batch request in frontend * use batch requests in frontend * use bounding box route * show bounding box in statistics modal * add bounding box to modal * add bounding boxes to modal and CSV and add spin while loading data * move cyclic dependency * WIP: add bounding box to indiv. segment context menu in viewport * add bounding box size to indiv. segment context menu * improve bounding box info in indiv. segment context menu * improve dependencies for requests of bounding box info * minor changes to make code ready for review * lint * add changelog * add keys to table rows * fix errors for datasets with fallback layers * very much WIP: started to address review * [WIP because of failing tests] address review * add tests * use clean way to find out a segments parent group * revert change in test.sh * trying to fix backend merge conflicts * fix imports after merge * fix saving before opening modal * address re-review * improve performance of segment list tab * improve csv export and change positioning of export modal --------- Co-authored-by: Florian M <florian@scm.io> Co-authored-by: Philipp Otto <philipp.4096@gmail.com>

2023-09-05

ND Datasets (#7136) * add UI for 4th dimension and mock 4D data when requesting buckets * adapt bucket address space to incorporate higher dimension * clean up imports * add slider to Q input field; tune mocked 4D data retrieval * change max value of number slider * make skeletons compatible with higher dimensions * fix tests; improve typing; create CubeEntry instances lazily * fix linting * use n-dimensional generalization instead of fourth coordinate everywhere * fix resampling of buckets when annotating volume data in higher dimensions * add additionalCoordinates parameter to several other places * hardcode additionalCoords initialization * tackle more todo comments * adapt additionalCoordinates input UI to arbitrary dimensions; use two additional coordinates as hardcoded ones * dynamically generate shaders for nodes and edges to deal with arbitrary ND data * make edges interpolate from alpha 1 to 0 if edge partner is on other dimension * Add additional coordinates for 4+D requests * begin integration of backend additionalCoordinate interface * change frontend format for additional coordinates from number-array to array of objects (analoguous to backend) * Add additional coords to db * Fix naming * remove hardcoded additional coordinates related to DataCube class * properly initialize flycam with additional coordinates * make that arrow keys in vector input increment/decrement element where cursor is at * fix onChange in additional-coordinate-input * use bounds of additional unified coordinates in slider * don't hardcode additional coordinate length in skeleton.ts; guard against underspecified additional coords when setting them in the flycam * fix rendering of skeletons in higher dimensions * export/import additional coordinates in NML (front-end import/export only) * ensure api.getDataValue receives additionalCoordinates parameter everywhere * also fallback to stores additionalCoordinates when they are not provided to api * remove second-last todo comment * Add additional coords as paramters for volume actions * Add editPositionAdditionalCoordinates to tracings * Implement n-dimensional morton code * Start implementation of volume tracing n-d fossildb access * Add additional coordinates to skeleton nodes * fix runtime exception when no additional coordinates exist; fix initialization of flycam * remove todo comment * rename additionalCoords to additionalCoordinates everywhere * fix typing * fix delayed update * hotfix missing additionalCoordinates for volume layer * pass additionalCoordinates everywhere SegmentItem- and MeshItem-related around * clicking on a segment list should activate its additionalCoordinates if they exist * fix linting * fix cyclic dependencies * rename to someAdditionalCoordinates * also store additional coordinates in json-encoded url state * show dimensions as addonBefore in AdditionalCoordinates input box * clean up * also maintain editPositionAdditionalCoordinates in frontend * Add additional coordinates to tracings * Read additional coordinate bounds from zarr header * Store additional coordinate as a vec2int in proto * store additionalCoordinates for volume and skeleton layers * Increment evolution number * fix cube.spec and volume_annotation_sampling.spec * fix wkstore_adapter spec * fix volumetracing_saga.spec * fix nml.spec.ts * fix url manager spec * fix skeleton spec by ensuring singletons are set up * fix saga_integration.spec * Remove nd morton code, use 3d morton code always * update documentation package to fix parser crash * Apply suggestions from code review Co-authored-by: Daniel <daniel.werner@scalableminds.com> * DRY assertion * fix camel case * undo changes in v2 api * avoid several THREE.BufferAttribute casts * add comment about additionalCoordinates = null in bucket pickers * add emit-rerender comment * use bounds[0] as a fallback value when additional Coordinates are missing in an update action * warn when meshing >3d segmentations * fix initialization of additional coordinates from url state * change additional coordinates format in xml to sth like additionalCoordinate-t=10 * remove unneeded functions in nml_helpers * fix scala compilation on CI * format * format backend * test nml import/export with additional coordinates * add test for setting additional coords in flycam * Some sort of sorting * Increment evolution number * Add comment in datasetarray, remove todos * fix linting * Update NML import/export for additional coordinates * Update snapshots * Sort additional coordinates when building bucket key * Update snapshots again * fix yarn.lock * Fix volume tracing bucket key not including additional coordinates because the datalayer did not have them * fix styling of additional-coordinate UI in light theme * also serialize additional coordinates for editPosition and layer itself into NML in frontend * show additional coords in context menu and status bar * Add test for bucket keys with additional coordinates (and nml) * Rename additionalCorrdinate to definition * Assert additional coordinates are the same when merging * Add additional coordinate tests * Update changelog * Remove TODOs * add warning for ai-quick-select for n-d datasets * enforce valid values for additional coordinates in UI * fix tests * fix nml test and add a new one for additional coords * Lint backend * remove t from axis order, allow mag paths relative to dataset dir * improve and fix snapshot tests * prevent user from exporting/downloading layers with nd data * Fix simple refactorings * Rename additionalCoordinateDefinition to additionalAxis * Rename additionalCoordinateRequest to additionalCoordinate * Continue renaming * rename additionalCoordinateWithBounds to AdditionalAxis and adapt to naming change in backend * Do not require migration of bucket keys * update snapshots * Lint backend * Rename Segment->additionalCoordinates to anchorPositionadd.. * Remove unused method * Remove exception throwing * Fix renaming issues * Fix upload of annotations --------- Co-authored-by: frcroth <frcroth@gmail.com> Co-authored-by: frcroth <frcroth@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com> Co-authored-by: Florian M <florian@scm.io> Co-authored-by: Florian M <fm3@users.noreply.github.com>

2023-06-13

Segment to Bucket Index for Volume Annotations (#7063) * WIP: Segment to Bucket Index for Volume Annotations * no chunking * get * column family, collect segment id * decompress bucket before collecting segments * load previous bucket * diff segment id sets * add/remove from segment index * fix update version, shutdown client * WIP: recompute segment index for existing volume tracing * update segment index on duplicate + downsample * import * WIP: segment volume route * Add route to compute volume segment volume * comment out logging * measure bucket saving perf * use buffer to avoid superfluous updates inside of one update group * use java.util.strem also for uint64 case * update segment index on volume revert * no segment index for old tracings or tracings with fallback segmentation * code cleanup * write index also when uploading zip * fix segment index when merging tracings * changelog + migration guide * snapshots * adapt frontend type * cleanup * extract canHaveSegmentIndex for a little less duplication * PR feedback

2023-06-07

Disable editing of super voxel skeletons in skeleton mode (#7086) * Introduce tree type (in frontend) and add it to some selector functions - prototype * Use deleteNodeAsUserAction also in context menu * Merge branch 'master' of github.com:scalableminds/webknossos into proofreading-skeleton-modification * Disallow agglomerate skeleton modifications, except for edge creation and removal in proofreading mode. Better error messages. * type skeletontracing_reducer.spec.ts * type nml.spec.js * type skeletontracing_saga.spec * fix linting * fix test and typing * fix types for mocked.reRequire'd functions and add build_info fixture * Update frontend/javascripts/oxalis/model/sagas/proofread_saga.ts Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> * fix bug that volume min-cut was allowed in proofreading mode * Distinguish agglomerate skeletons in tree list using proofreading icon. Include tree type in NML in frontend. Improve typing for coalesce function. * Merge branch 'master' of github.com:scalableminds/webknossos into proofreading-skeleton-modification * update nml snapshot * update changelog * Add type enum for trees in skeleton protobuf and nml * add tree type to update actions * Merge branch 'master' into proofreading-skeleton-modification * update snapshots * Apply suggestions from code review Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> * Merge branch 'master' of github.com:scalableminds/webknossos into proofreading-skeleton-modification * apply suggestion from code review * Merge branch 'proofreading-skeleton-modification' of github.com:scalableminds/webknossos into proofreading-skeleton-modification * Merge branch 'master' into proofreading-skeleton-modification

2023-05-10

Merge editable mappings (#7026) * WIP: Merge Editable Mappings * pass remote fallback layer * adapt usages * gather update actions to merge in * updater * agglomerate id for segment id * find agglomerate id by position * duplicate update actions, fix versioning * log warnings only if relyOnAgglomerateIds. add some debug logging * add perf logging * batch fetching of update actions * Fix fetching update actions * temporarily disable ci checks * logging, fix duplicate version * pass version param * adapt to new fossildb get multiple keys api * fix pagination * adapt the merge modal * cleanup * changelog * rename method, fix merge project, verbose tooltip * make import trees buttons non-primary, extract 24 to constant * pr feedback on naming + comment * update fossildb version info * adapt docker-compose to new fossildb schema

2023-04-27

Add segment groups (#6966) * Add segment groups * Implement nml writing for segment groups * Implement nml parsing for segment groups * implement first rough version for segment groups * clean up typing and integrate autosizer * optimize getDerivedStateFromProps * tune virtual scrolling * temporarily disable most CI checks * allow to create new segment groups (not saved, yet) * send update actions when groupId of segment changed * Update action to allow changing of group id * Adapt wrapping for segment groups * Extract group merge handling from tree groups and implement for segment groups * Fix faulty test * prepare saving of segment groups in front-end * fix updateSegmentGroups action in backend * fix scrolling for virtualized tree list * also make dnd work for groups and respect dropToGap parameter * clean up * add missing description for version history entry for updateSegmentGroups * linting * clean up imports * implement renaming for segment groups * implement deletion of segment groups * implement undo/redo for segment groups * fix that groups could be sent to nirvana by dragging them above root group * remove obsolete expandedGroupIds * fix that segment groups were invisible when no segment existed * clean up * more clean up * remove unused imports/types * simplify rename-group code * implement removeMissingGroupsFromSegments and more improvements * add better batching for recursive deletion of groups and try to make it compatible with undo (still work in progress) * clean up batch actions and fix that undo of recursive-deletion of group didn't correctly save the original group id * add tests for segment group deletion with undo * clean up and also add better types for batch updates of skeletons+groups * update yarn.lock * format * misc clean up * adapt wording in dnd import modal * add (failing) test for fixing a undo/redo bug * fix undo/redo bug related to segment groups * update snapshots * update changelog * incorporate pr feedback * Update frontend/javascripts/oxalis/model/reducers/volumetracing_reducer.ts Co-authored-by: Daniel <daniel.werner@scalableminds.com> * Update frontend/javascripts/oxalis/model/reducers/volumetracing_reducer_helpers.ts Co-authored-by: Daniel <daniel.werner@scalableminds.com> * fix that changing the color of a segment without any isosurfaces crashes wk * forbid renaming when segment or groups are being renamed to fix selection bug * fix --- title * distinguish between segment and tree groups when importing NML * fix that segment could get -1 as group when being dragged to root * delete segment tooltip as it was annoying during dnd operations and also not that useful * let user confirm before removing root (segment or tree) group * always sort segment list by groupId and segmentId * improve wording for downloading visible trees * fix that segment or group was sometimes moved to incorrect parent * clean up mapping with root group * refactor moving of skeleton and segment groups; integrate moving of groups into skeleton context menu * rename createMenu to menu where appropriate * implement move-segment-to-group via context menu (similar to the new move feature for trees) * generalize tree group to group in var names * Update frontend/javascripts/oxalis/api/api_latest.ts Co-authored-by: Daniel <daniel.werner@scalableminds.com> --------- Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com>

2023-04-26

Store editable mappings in multiple fossildb columns+keys (#6903) * WIP: Store editable mappings in multiple fossildb columns+keys * wip: propagate new format * agglomerate skeleton * no longer apply updates lazily * WIP: store segment to agglomerate * puzzle together segment to agglomerate from chunks * use proto for remote segment to agglomerate lookup * WIP: update chunked segment to agglomerate * save segment to agglomerate * fix saving segment to agglomerate chunk * cleanup fetching segmentToAgglomerate; count up versions * WIP: pass buffers * WIP: EditableMappingUpdater with buffers * use said buffers to store updated keys before flushing after update group * duplicate route * WIP: move to lazy applying of updates again * fix merge action endless loop * rename proto file * Add FossilDB Migration * measure time, add more caching * add agglomerate to graph cache * migration perf * More caching, jgrapht connected components, remove dead code * cleanup * remove debug logging * remove time logging from migration script * list versions instead of getMultipleVersions * separate migrating history from rest * pr feedback * undo dev changes * add args to migrate script * cleanup logging * migration guide * pr feedback on comment + migration guide wording * do not cache Fox.failures, add logging to split actions * add segment id assertions * more logging * warning if edge is absent * make actions with invalid segment ids no-ops * log more warnings, tryo around mapping in datastore, make zero segment split/merge no-op * remove unused import * don't retry bucket look up on data store when there's an editable mapping * remove dev ci skips --------- Co-authored-by: Philipp Otto <philipp.4096@gmail.com>

2023-04-05

Implement nml writing for segment groups

Add segment groups

2022-10-18

Look up annotation dataset by tracing id instead of (broken) organizationName field (#6548) * Fix organization name field for annotations of foreign-orga datasets * undo unnecessary change * pretty * [WIP] ask wk for datasource by tracing id instead * implement routes on wk-side * include annotation temporary store in lookup * add comment in proto definition file * add some caching * use correct tracing id when downsampling a volume tracing * changelog * pass dataset name to nml writer * fix parameter order

2022-09-26

Make Largest Segment ID Optional (#6414) * [WIP] Make Largest Segment ID Optional * restore accidentally deleted files * make explored layer’s largestSegmentId None * make front-end compatible with nullable largestSegmentId and adapt UI so that the user is forced to input it if necessary * make id optional also for volume tracings * don't interpret 0 as null for largestSegmentId * allow null also in volume update action * fix linting * allow dynamic imports in tsconfig * temporary fix for mappings * clean up * rename max cell to largest segment id to be consistent with backend and datasource properties * adapt largest segment id behavior for zarr import UI * also rename maximum segment id modal to largest segment id modal * improve toast handling * fix default segment id when loading a empty volume tracing without fallback * adapt tests related to largest segment id * fix backend test * remove assertion * extend documentation about largest segment id * link to dataset settings page in enter-largest-segment-id modal * update changelog * add rel='noreferrer' * PR Feedback / Refactoring / Comments * Update CHANGELOG.unreleased.md Co-authored-by: Daniel <daniel.werner@scalableminds.com> * Apply suggestions from code review Co-authored-by: Daniel <daniel.werner@scalableminds.com> * incorporate pr feedback * use correct max value for largest segment id input * show warning when configuring segmentation layer without largest segment id; fine tune wording and UI * fix linting * make largestSegmentId optional in new N5SegmentationLayer class * remove dependency on largestSegmentId in mapping code, because reassignment of IDs is not necessary with the new custom color mechanism anymore * fix that the create-cell button was hidden when the active segment id was 0 Co-authored-by: Philipp Otto <philipp.4096@googlemail.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com>

2022-09-21

Allow custom colors for volume annotation segments (#6372) * [WIP] allow custom colors for volume annotation segments * pretty * include color in segment update actions * fix compile errors * backend tests * set color to none if no color was included in the update action * Custom segment colors with cuckoo hashing (#6417) * resurrect cuckoo table implementation * adapt CuckooTable to entries consisting of (number, vec3) and finish proof-of-concept for writing and reading that hash table for segment colors * fall back to old color generation if custom color could not be looked up * add UI to select segment color * correctly save segment color to back-end * hook up custom segment colors from store with texture * improve performance of cuckoo updates * fix isDisabled prop for interpolation button * convert usage of vec4 color to vec3 * remove type property for uniforms since it's not needed anymore since three r76 * adapt tests to new cuckootable interface * clean up a bit * fix cuckoo tests by using broader value range for tests and using uint32array instead of float * try to get uint32 texture working (int texture only works when defined as byte, but that will clip values to 8 bit) * dirty proof of concept for uint32 texture * restore custom color rendering * fix tests * DRY mocks in tests * more clean up * implement unset for cuckoo table to allow resetting a segment color * extend tests a bit * don't hardcode values for cuckoo shader code * fix linting * also show segment patterns for custom segment colors * show correct colors (even when mapped) in segment list * also use correct color for mesh and synchronize the mesh with the segment color * fix linting * clean up some typing * make custom segment colors compatible with multiple segmentation layers * fix linting * fix segment color icon for non-tracing layers * don't allow changing colors for segments of non-tracing segmentation layer * remove unused code for low-level texture setup * remove logging from cuckoo table implementation * clean up cuckoo_table.spec.ts * fix React does not recognize the prop on a DOM element. warning when opening context menu * don't rely on random IDs in merger mode * dont use the term color in the merger mode implementation as it is incorrect and confusing * remove shuffleColorOfCurrentTree functionality for merger mode (let's wait for user feedback to see whether this is really needed) * also reset stored representative when deleting last node of tree in merger mode * clean up some mapping related code * allow changing segment colors for non-tracing segmentation layers (necessary for json mappings); the colors won't be saved, though * use custom segment color feature when loading a mapping which defines hue values * clean up typing for withMappingActivationConfirmation * remove old custom color implementation for JSON mappings * remove attempt to integrate ACE editor * move the cuckoo table maintenance into the LayerRenderingManager to avoid redundant updates and ensure diffing with the correct segments * fix linting * fix that shader won't render when all volume layers are invisible (but at least one exists) * don't fill int texture with 255 by default * revert temporary changes in test.sh * skip shader tests which try to parse glsl 3000 syntax (and fail) * fix some tests by having isosurface-related sagas wait for the scene controller to be initialized (which doesn't happen in some tests); therefore fixing null access to the scene controller * skip mapping-related texture requirement test * fix that seeds were not initialized correctly when no segmentation is visible initially * explain why some glsl tests are skipped * remove isMappingSupported distinction and incorporate custom colors texture into rendering logic * fix skipped test in rendering logic spec * implement compaction for update segment UpdateActions * explain texture format etc better * clean up test * make getSegmentsForLayer non-nullable * fix shader tests by avoiding hex uint notation * explain isInt workaround * remove one window mock; enforce correct type of other window mock; remove lots of @ts errors * explain disabled logic * fix linting * fix linting * initialize color picker properly even when a custom segment color was not defined yet * don't share material for chunks of mesh, since the opacity has to be controlled individually for the loading animation * fix linting * fix crash when anchor position of segment is null * try to debug buggy restore * fix bug with repeated sets that caused a key to be appear multiple times; also add regression test * fix that mapping.mappingColors were still used in toolbar and segments tab; clean up * fix linting * avoid that all segments in segment list rerender when mouse is moved * make custom colors work when using setMapping via frontend API * add analytics for custom colors in setMapping * memoize diffing of segments; refactor memoization of diffing trees * add docs for segment list * remove obsolete type parameter for uniform Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Philipp Otto <philipp.4096@googlemail.com>

2022-08-04

[WIP] allow custom colors for volume annotation segments

2022-06-22

Editable Mappings aka Supervoxel Proofreading (#6195) * [WIP] editable mappings * query agglomerate skeleton * first update actions * column families * pipe update actions to fossildb * logic for applying updates on load * some logic for gathering relevant ampping * some logic for generating skeleton * logic for applying update actions in order * resolve segment position to segment id * mapData * fetch unmapped data from datastore * bytes conversion * fix unsigned int to long * apply merge update action * split update action * enable bounding box tool in skeleton-only annotations * avoid loading agglomerate skeletons twice * add proofread tool, add merge and split agglomerate update actions, clean up update actions and add missing descriptions for version view, add first version of proofreading saga * fix json serialization, create stubs for querying agglomerate graph + largest agglomerate id * fix some naming, enable proofread saga * in update actions, use positions and mag * fetch agglomerate ids for segment ids * generate AgglomerateGraph from hdf file * also send mag and agglomerate ids in editable mapping update actions * include agglomerate id in skeleton uri, directly set editable mapping in route, add bool mappingIsEditable * persist mappingName in volume annotation and restore mapping on load * make mapping editable, lazily (before first proofreading action) * add new save queue for mappings and improve save queue typing * restructure routes for compatibility * implement separate save queues for editable mappings, initialize and push update actions for editable mappings * fix route, token and missing-buckets header parsing * fix editable mapping initialization * updateEditableMapping takes list of groups * affinities are float * load editable agglomerate skeleton from tracingstore * fix that source and target were in different trees * some backend fixes * Add logging, fix update action order, store as lists in fossildb * agglomerate skeleton node ids start at one * remap tree and node ids if there ids below the minimum * localize node ids in agglomerate skeleton * correctly set new mapping name after creating an editable mapping * fix tests * allow to select nodes when proofread tool is active * cache materialized editable mappings * speed up ci * [WIP] enable ad-hoc meshes for editable mapping annotations * depend only on the cache key * agglomerate graph resilience for agglomerate id 0 * fix highest lowest resolution confusion * delete the edge even if one graph gets all nodes * editable mapping meshes * make isosurface service not strictly depend on agglomerate service * guard for agglomerate service being null * use tracingstore for ad-hoc meshes * fix mesh positions * fix editable mapping page reload * caching for unmapped data, skip buckets outside of bbox * load meshes when proofreading * perf optimization for ad-hoc meshing * fix window configuration variables and allow to disable proofreading using meshes * make coarse agglomerate more transparent, exclude it from ray tracing and try to fix transparency issues * use data store to request ad-hoc meshes if mapping is not yet editable * reload meshes after proofreading action * enable context menu in 3d view * fix loading of segments on first click, properly reload agglomerate skeletons after proofreading action * fix bucket position access * parallelize buckets again for larger cuboids * reuse binary data service code for loading editably-mapped data * reduce isosurface cube size according to mag, make cube size window configurable * fix linting * cleanup * measure time for json conversions of editable mapping * add proto definition for AgglomerateGraph, EditableMapping * add proto conversion methods, measure speed * cleanup, method naming * fully change AgglomerateGraph to proto, clean up backend code * extract AlfuFoxCache, add cache for largest agglomerate id query * minor backend code cleanup * re-enable CI checks * remove unused code * migration guide * [WIP] prevent making editable mapping after volume brushing * increase agglomerate skeleton max edge limit by one order of magnitude since the larger value has been tested thoroughly in production * make proofreading click in 3d view work, adapt proofreading defaults, reload agglomerate skeletons even if mesh proofreading is disabled * improve version typing, show warning when trying to restore older editable mapping version * disable proofreading tool for non-hybrid annotations * disallow to change mapping after editable mapping was activated * hide volume modification tools after editable mapping was activated * fix linting, only remove meshes that are not immediately loaded again * show empty placeholder instead of misleading info rows in td view context menu * add created timestamp to editable mappings * first round of refactoring, commenting, tweaking * fix null reference in isosurface service * assert volume buckets empty * backend pr feedback (part 1) * add some comments to proofreading saga * pr feedback (part 2) * PR feedback part 1/x * PR feedback 2/x * disable loading of fine over segmentation meshes in proximity by default * avoid requesting mapping when loading brushed volume data * PR feedback 3/x, ensure agglomerate mapping is active when proofreading, fix tests * fix tests for real git co l4-spine-heads elf * fix linting * fix tool cycling and move proofread tool to the right * fix tooltips for navbar buttons, disable merge mode if editable mapping is active, PR feedback * undo last action if editable mapping creation fails, add more explanation to proofreading tool * instead of reloading agglomerate skeletons after proofreading action, simply rename them, fixing multiple bugs * fix ad-hoc mesh loading for editable mappings * increase ad-hoc batch limit 8-fold since cube size edge length per dimension was halved * hide and disable proofreading tool if no agglomerate views are available * fix merge * fix agglomerate too large error message for editable mappings * Apply suggestions from code review Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> * update changelog and pretty * fix typing * fix proofreading when selecting far away node in 3d view * try one-sided mesh rendering to avoid transparency artifacts * disable undo/redo during proofreading and show warning toast * warn if it looks like the user wanted to proofread but the proofread tool was not active Co-authored-by: Daniel Werner <daniel.werner@scalableminds.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com>

2022-02-17

Clean up backend util package (#6048) * Clean up backend util package * rename usages * also rename proto geometry classes * imports * adapt test suite

2021-11-01

Implement Segments lists (#5696) * add alert box to show currently converted datasets * Update frontend/javascripts/dashboard/dataset_view.js Co-authored-by: Daniel <daniel.werner@scalableminds.com> * pr feedback * pretty frontend code * also show successfull datasets in dashboard, tweak styling, automatically poll, adapt tooltips etc. * update changelog * extract some variables, such as RECENT_DATASET_DAY_THRESHOLD, NonBreakingSpace and MAX_JOBS_TO_DISPLAY * change RECENT_DATASET_DAY_THRESHOLD to 3 * adapt the icon colors * add first simple version of adding anotated segments to the UI * WIP: add removing covered buckets of segments in case they are overdrawn * fix removing covered buckets when segments are overdrawn * add some position to each segment and always keep it valid * fix tests * adding undo/redo support and fix updating of segment position * make segment housekeeping async to be compatible with unfetched backend data * try to add segment housekeeping when backend data is merged with already existing frontend data * clean up code and add comments * [WIP] add backend update actions for volume segment lists * Persist segment list in backend, apply update actions * add json format for new update actions * fix json keys for update actions in backend * remove code which maintains the bounding boxes in which one segment can be found * remove more obsolete code and comments * remove more obsolete code and comments * fix action names * fix deprecation warning for threejs's setDynamic * use DiffableMap for storing the segment map; fix multiple bugs in persisting and loading segments on/from backend * fix linting * merge mesh list with segments list * allow to rename segments * rewrite logic for activating a cell in the sidebar and highlight segments if they are centered and/or belong to the active cell id * tune styling * add segments to segment list when selecting a segment or when computing a mesh for it * remove obsolete cells property in volume tracing * refactor store state by introducing localSegmentationData which contains isosurfaces, currentMeshFile etc by layer * make segment lists compatible with view mode and multiple segmentation layers * fix flow and linting * fix tests * clean up mesh UI a bit * remove some stl import related code * Backend Test should delete segment 5 (the one that actually exists in the fixture) * fix segment update for volume tracings * fix redux saga's debounce typing * also add segments when hovernig a segment for more than 500ms; implement bidirectional hovering of segments * make some output less noisy * refactor root saga handling a bit * refresh snapshots * fix segment list in undo state * improve clicking of segment and add tooltip for segment id * don't allow editing a segment in read-only * only show ellipsis on hover and add tooltip which explains right-click option * improve UI text * rename meshes view to segments view * refactor entire segments component into smaller subcomponents * fix react warning and close context menu after usage * wording * hoist more functions within segments view * allow ad-hoc mesh computation via data context menu * add mesh quality UI, improve performance and tweak UI further (e.g., fixed header) * finish mesh quality UI and hook it up to the store and use the values accordingly * fix wrong clamping and neighbor computation for ad hoc meshing * clean up * properly use active mapping in segments view * add ad-hoc meshing to context menu in view mode and optimize some performance * render nice UI if no segments or no mesh files exist; also fix select boxes if initial preferred mag does not exist * extract segments list item into own module (and add new folder within right-border-tabs) * don't crash segments component when segmentation layer is invisible * fix isosurface neighbor definition in back-end * improve wording; immediately add segment on brush start; make context click target larger * instead of auto-adding a segment on hover, only do this when clicking a segment with the move tool instead * don't load centered cell when selecting mesh file and clean up consequentially obsolete function * update changelog * fix linting * fix saga tests * incorporate pr feedback (I) * undo changes in volume_annotation_sampling.js * update comment for autoDestroy * Update frontend/javascripts/oxalis/model/reducers/volumetracing_reducer.js * also maintain creationTime for segments; render them in a tooltip * fix linting * fix flow * fix exception in maintaining undo state for skeleton-only annotation with segmentation layer * backend pr feedback (use protobuf convenience method addSegments) * cancel saga which loads ad-hoc mesh if the mesh was removed * mock date to fix flaky timestamp in test Co-authored-by: Youri K <youri.kaminsky@student.hpi.de> Co-authored-by: Youri K <youri-k@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com> Co-authored-by: Philipp Otto <philipp.4096@googlemail.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Florian M <florian@scm.io>

2021-01-04

Generate agglomerate skeletons from voxelytics agglomerate file (#4958) * add route for agglomerate skeletons * read more arrays, move proto to datastore * convert edges + nodes to skeletontracing object * add shortcut to load agglomerate skeleton * add configurable edge count limit * fix proto paths and show skeletons * use agglomerate id as tree id * agglomerate id to int * skip tests * fix config access * add to standalone datastore conf * small renaming * add error logging, use skeleton defaults * remove unsafe route, add message, indent routes file * fix some todos * fix option * blur mapping dropdown after selection * replace fox by box as the call is actually synchronous * fix error messages * fix remaining frontend todos * readd full CI tests * send request to fallback layer if available, deactivate in view mode * add missing parentheses * Apply suggestions from code review Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> * address PR feedback * add hotfix for bug where tree ids that are too large break webKnossos, add button to import agglomerate skeleton * use shift + middle click as a shortcut and disable skeleton loading in volume-only annotations * Update frontend/javascripts/oxalis/model/reducers/skeletontracing_reducer_helpers.js Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Daniel Werner <daniel.werner@scalableminds.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com>

2020-12-21

Grüße, The Butcher

This commit does not contain any .proto files.

Grüße, The Butcher

This commit does not contain any .proto files.

Grüße, The Butcher

This commit does not contain any .proto files.

2020-11-09

Multi-Resolution Volume Tracings (#4755) * keep fixed zoom during td rotation to planes * refactor 3d rotation method to keep the zoom and simplify it * fix position and up vector not getting interpolated correctly * add changelog entry * Apply suggestions from code review Co-authored-by: Daniel <daniel.werner@scalableminds.com> * remove zReductionFactor * backend: support anisotropic resolutions in volume bucket keys * fix 3d rotation preset * remove useless check for valid values * remove reassignment of width and height * do volume annotation in all resolutions * Update frontend/javascripts/oxalis/controller/camera_controller.js Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> * made code pretty * reduce max brush size * disallow trace tool in higher resolutions * enable brushing in high again * do not block resolution loading * annotate z many slices at once when z resolution is > 1 * fixing a few tests * hopefully fixed all tests * Add info about annotating multiple slices * limit area that gets autofilled * [WIP] lazy downscaling of volume buckets * lookup-based live downsampling * [WIP] downsample volume tracings on upload * [WIP] downscale during upload * fix offsets, switch to mode * determine which mags to downsample * adapt nml unit test suite * fix backend dummy tracing * move function to get number of slices to accessor * WIP: applying flood fill to all resolutions * [WIP] track mags in volumetracing object * fix calculating error during downsampling * fix flood fill for all source resolutions * fixed flood fill for all resolutions * handle differing resolutions during volume merging + upload * add up and downsampling of LabeledVoxelsMap * add unlinkFallback route * changelog * cleanup backend code * fixed up and downsampling and added tests * added method to apply a voxel map * fixed upsampling and added regression test * add comments to sampling methods * make copylayer work in multi resolutions * while upsampling annotate mutliple slices * Add todo about refactoring labeling voxels * adjusted voxel labeling method to label in all resolutions the segmentation layer has * added some comments with todos * Update frontend/javascripts/oxalis/model/accessors/volumetracing_accessor.js * always label in all resolutions when using api; show stack-icon when multi-slice-annotating * more comments * replace old unlinkFallbackLayer with new API for that (keep old UpdateAction for backwards compatibility) * fix wrong parameter format * refactor resolution handling to deal with sparse resolutions better (WIP/1) * undo temporary new_resolutions change * continue refactoring resolution handling to deal with sparse resolutions better (WIP/2) --> brush, paint-bucket and copy-segmentation tool seem to work * fix picking segment color when being in a not-existing segmentation mag * ensure that prefetch saga, bucket picker and pull queue don't try to load buckets for not existing mags * ensure legacy getResolutions method provides dense resolution set; clean up * fix initialization and volume annotation sampling spec * fix CI * update snapshots * tune warning text * make merger mode work in higher resolutions * show a status indicator in the viewports if a layer cannot be rendered due to the zoom step * reduce tooltip rerenderings by making styles constant * fix enabled 'Render Missing Data Black' for missing mags * add issue links to todo comments * implement some feedback * Apply suggestions from code review Co-authored-by: Daniel <daniel.werner@scalableminds.com> * more PR feedback * also implement getIndexOrClosestHigherIndex for ResolutionInfo and use where appropriate * return true in isVolumeTraceToolDisallowed if no volume tracing exists * fix front-end merge conflicts * reduce usages of getResolutionByIndexWithFallback and make that function correct (remove isotropic fallback in favor of failing for unclear cases * simplify is2DVoxelInsideBucket method in Bucket * Apply suggestions from code review Co-authored-by: Daniel <daniel.werner@scalableminds.com> * integrate more PR feedback; resolve merge conflict in changelog; rename missing to unrenderable etc * refactor applyLabeledVoxelMapToAllMissingResolutions * further refactoring of applyLabeledVoxelMapToAllMissingResolutions * remove unused import * fix flow after upgrade * improve comment for LabeledVoxelsMap * fix uniform definion * fix getDataValue for missing resolution * Improve performance of volume annotation tools (#4848) * tmp: use LabeledVoxelsMap for brushing and TypedArray for brushing map (first measurements yield 20x better performance) * restore all of the brush functionality; create LabeledVoxelMap in current mag; further performance optimizations * clean up VoxelIterator a bit * more clean up * clean up and fix CI * further clean up * remove todo * use unzoomed centroid when updating direction * fix contour drawing in mag > 1 * fix flow * fix circle-drawing for anisotropic mags * update changelog * rename VoxelIterator to VoxelBuffer2D * optimize draw circle method by comparing squared dist with squared radius * rename variables and change comment for bx and by * remove obsolete comment * add another comment to fillCircle call * ensure that rendered volume magnification is also the mag in which is annotated - also ensure that the volume toolbar is disabled properly when the segmentation cannot be rendered for some reason * Update frontend/javascripts/oxalis/model/sagas/volumetracing_saga.js Co-authored-by: Daniel <daniel.werner@scalableminds.com> * fix issues from merging master * fix upload/download * fix issues after merge * disable continuous drawing for now since its performance needs to be adapted to the multi-resolution feature * fix linting * adapt headline in import modal * change tracing to annotation * change magnification to resolution in user-facing strings and some internal ones * make resolution-warning less verbose * try to fix e.trigger is not a function * fix linting * fix bug in applyLabeledVoxelMap * fix flow * show fallback-not-included warning only when there is fallback data * fix anisotropic bucket adresses * treat empty resolution list as no resolution list * change Boolean() to != null * fix flow in model initialization * ensure volume layer is saved before reloading; fixes #4857 * format * disable copy-segmentation feature for higher mags due to performance * fix some merge-related problems * fix styling of multi-slice-icon * throw an exception if the resolution could not be looked up instead of defaulting to 0,0,0 * look up volume bucket resolution by zoom step ** 2 rather than index * fix that toolbar rerendered on every state change * improve handling of volume-is-disabled case in toolbar * replace exception by fox.failure if volume resolution lookup fails * show data type and resolutions of layer in tooltip next to layer name (therefore, the dtype tag is removed) * fix syntax error * Enforce TaskType Resolution Restrictions for Multi-Res Volume Tasks (#4891) * [WIP] enforce task type magnification settings in tasks * pass allowedMagnifications to rpc methods * pass magnifications as query string * respect resolution restrictions for volume tracings. * add option to downsample existing volume tracing * remove outdated assertion, update resolution list after downsampling * sleep 10s * remove sleep and bump dev-proxy-timeout to 5 min * fix task creation with resolution restrictions * use inclusive check * when uploading zips with differing resolution sets, fail * remove unused import * refresh snapshots (tracings contain organization name) * write header.wkw to every mag in volume download, fix anisotropic directory names * add resolution-samenss assertion in volume dnd import. fix flow types * fix restriction passing in initial data * Add button to trigger downsampling of volume annotation. The reload button next to the active-resolution-indicator only appears for explorative annotations with volume data. It opens a modal which explains the action and its consequences. While the operation is running, the modal blocks further usages of webKnossos. After the operation has completed, the complete page is reloaded. * fix parameter passing in duplicate rpc * avoid undefined behavior of bucket iterator by checking hasNext. do not relabel when “merging” only one volume * make tasktype resolution restrictions immutable. sort resolution list * update migration guide * remove debug output * improve initiate-volume-downsampling modal and move its button to the layer settings * show toast when mag-restriction is violated instead of prohibiting the zoom change in the first place * don't allow editing mag-restrictions in already created tasktype * remove some more debug output, unify variable names * forbid creating new nodes or using volume tools when the current mag was forbidden by the task type * clean up * Update MIGRATIONS.unreleased.md Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> * fix styling of button-link * use ListBuffer instead of HashSet as per pr feedback * integrate some feedback * Apply suggestions from code review Co-authored-by: Daniel <daniel.werner@scalableminds.com> * remove unnecessary SET_TOOL dependence * do map ids if initialLargestSegmentId != 0 Co-authored-by: Philipp Otto <philipp.4096@googlemail.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com> Co-authored-by: Michael Büßemeyer <michael.buessemeyer@student.hpi.de> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com> Co-authored-by: Philipp Otto <philippotto@users.noreply.github.com> Co-authored-by: Youri K <youri.kaminsky@student.hpi.de> Co-authored-by: Philipp Otto <philipp.4096@googlemail.com>

2020-09-11

handle differing resolutions during volume merging + upload

2020-09-10

[WIP] track mags in volumetracing object

2020-09-08

determine which mags to downsample

2020-07-06

Support Multiple User Bounding Boxes (#4536) * [WIP] multiple user bounding boxes * [WIP] integrate new user bounding boxes into nml * add update actions for user bounding boxes * update volume tracing update actions * ensure task bounding boxes get copied when duplicating * add optional color attribute * add multiple user bounding boxes to UI * render all user bounding boxes * add name and visibility to user bounding boxes * add updateactions to save user bounding boxes * small type fixes * fix linting * fix nml serialization test * removed logging * include right dependency for tests * refresh snapshots * fix flow * implement pr feedback * add old userbb to userbboxes array * adjust ui * fix always the same color bug * move old user bounding box to new array * prevent losing old, unapplied user bounding boxes * revert duplication of adding old userBBox to new array (frontend) * for backwards compatiblity in old tracings * proper backwards compatiliblity * use 0 as default for old user bbox * Add user boudning boxes to import in tracings * avoid id duplication in user bbox import * also export the visiblity of user bounding boxes * apply feedback * add comments * consistent naming of methods * remove unsued method * rename method * merge userboundingboxes for skeletons * correctly convert task bounding box when duplicating volumes * merge skeleton user bounding boxes by their bounds, color, name * changed migration of old user bounding box * the new version triggers an update action to enforce consistency * fix linting * udpate nml snapshot * dispatch user bounding box update after model initialization * revert changes for migrating old user bbox * add tracing migration service to safely remove all old user bounding boxes * update snapshots * set initial color for moved bounding boxes * update nml test spec * changelog + docs * transform task bounding boxes if uploaded as explorational * fix backend tests * add taskboundingboxes to nml parsing and import them as user bounding boxes * fix empty.max exception in bbox parsing * backend color generator range from 0 to 1 * use range of [0,1] for user bb color parts * have fixed size of boudning box setting component independent from visibility of the bb * fix and update nml snapshot tests * only update name of user bb on blur or on enter * also add duplicates when import an nml * also merge duplicate boxes when merging tracings in backend * format Co-authored-by: Youri K <youri.kaminsky@student.hpi.de> Co-authored-by: Michael Büßemeyer <michael.buessemeyer@student.hpi.de> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Daniel <daniel.werner@scalableminds.com>

2019-10-10

add backend part for storing diameter properties

2019-04-08

Persist tree visibility (#3942) * add visibility to trees and responding update actions #3666 * adapt behavior when called with no isVisible attribute #3503 * read and update isVisible per tree * implement 'smart' compaction logic for toggling tree groups * add update-tree-visibility updateAction #3942 * make tree visibility persistence work * extract compaction logic into separate modules * fix flow * fix linting * add comments and more typing * update changelog * fix existing tests * fix existing tests * tmp: add new tests for compact toggle actions * make compact-toggle tests work * extract defaultState into own module; fix flow for compact-toggle-tests * Update frontend/javascripts/oxalis/model/helpers/compaction/compact_toggle_actions.js Co-Authored-By: philippotto <philippotto@users.noreply.github.com> * incorporate further feedback * add comment to clarify null handling of isVisible * use tree.isVisible = true if isVisible is undefined * don't render invisible trees when loading tracing * change tree visibility recursively #3942 * improve readability * fix missing handling of empty groups when compacting toggle actions

2019-02-06

Add Volume Tasks (#3712) * [WIP] volume tasks * fix volume task creation + requesting. TODO: download * allow to choose volume tasks via UI for simple task creation * enforce page reload when switching from/to/between volume tasks; show tracing type in dashboard * add tracing type to task type * multi-task download: fetch volume data * add enums to evolutions * scalafmt * add volume data to multi-task zip downloads * scalafmt * add tasktype tracing type to test db * validate tracing type when reading task type from db * adapt front-end to volume task types * update some flow types * update snapshots * update changelog + migrations * add error handling for the as-yet unsupported volume task features * disable nml upload if volume task is selected * remove superfluous return type * update evolution numbers * persist volume task bounding box in nml * read volume task bounding box from nml after re-upload * scalafmt * don't clip rendering of color layer in volume tasks * undo change to saving skeleton bounding box to nml * don't clip segmentation outside task bounding box * fix typo * Update CHANGELOG.md * apply PR feedback * remove typ attribute from task * update snapshots

2018-10-17

Separate DataStore and TracingStore (#3281) * [WIP] tracingstore module * update schema to register datastores * move towards tracing store module * tracingstore gets its own config * add a tracingstore to the test db * provide volumetracing data route * fix standalone tracingstore * standalone tracingstore not enabled by default * fix local tracingstore, sacrificing standalone-one for now * use tracing store in front-end (not yet for volume tracings) * refactor buffer slicing out of pullqueue and into wkstore adapter * [WIP] set volume properties on upload * in volume tracing upload, set fallback-layer dependent values * rename some files, separate datastore/tracingstore wk communication clients * fix token auth for tracing store * implement segmentation fallback logic in front-end * move proto files to tracingstore. update docker/circleci files * remove obsolete route api/tracingstore since it is supplied in annotation info request * fix proto path in test.sh * fix lint * update schema version in schema.sql * update schema version in schema.sql * update changelog + migration guide * Tracingstore health (#3295) re-add tracingstore listOne route for health check * fix health check for tracing store * update snapshots * pretty code * fix wk store adapter spec * fix pullqueue spec * post-merge fix for application.conf * update evolution number * make evolution numbers unique, fix variable unpacking in version_list.js * fix schema version numbers * update changelog + migrations.md * incorporate PR feedback * fix update action log route for volumes * lint

2018-10-08

backend part for versioned volume tracings (#3317) * [WIP] versioned volume tracings #3146 * [WIP] get multiple version of a tracing * add modified timestamp and first version of updateActionLog #3146 * [WIP] add versioned bucket iterator to enable revertToVersion action #3146 * enable version view for volume tracings, allow to load older volume versions, refactor version view * fix compactVolumeUpdateActions * remove modifiedTimestamp as it is no longer needed #3146 * repair scrolling for now in version view, add description for updateBucket update action * fix updateAction name * revertingToVersion works for buckets previously covered by the volume tracing #3146 * small version view design improvements * fix restore of volumeTracings which are larger than there predecessor #3146 * disable volume version view for now * fix flow * implement pr feedback #3317

2018-05-22

Add support for groups of trees (#2546) * introduce tree groups and show them in the tree tab view (#2514) * simplify tree group handling, add possibility to rename/delete group, fix tree group drag/drop, fix tests, linter and flow (#2514) * make it possible to submit new tree name using enter (#2514) * distinguish correctly between group and groupId (#2514) * renew snapshots after antd and react update (#2514) * make react key for groups and trees truly unique, add groups to update actions, parse and serialize groups from NMLs, add tests (#2514) * add more tests to test nml parser group errors (#2514) * small fixes, autofocus, auto-save when editing groups (#2514) * avoid constant rerender of save button * use virtualized react-sortable-tree component instead of antd tree for performance reasons, update webpack, cleanup (#2546) * store treegroups with skeletontracings, update with updateActions * Write treegroups to nml in backend * read + write tree groups to NML in backend * backend: make groupId an Integer * change groupId from string to number (#2546) * use correct id for trees with no group (#2546) * if updateaction treegroup id is null, set to None, rather than keeping the old * rename groupId to id in nml group tags (#2546) * incorporate PR feedback, enhance design of sortable-tree component (#2546) * incorporate more PR feedback, add workaround for scrolling bug (#2546)

2018-03-06

Merge branch 'master' of github.com:scalableminds/webknossos into team-structure * 'master' of github.com:scalableminds/webknossos: use new fossildb version for local datastore zip volume with iterator instead of list clean up volume download WIP: pagination for volume data downloads Update README.md remove unnecessary console.log fix tree sorting in tracing view by changing antd's menu prop from onSelect to onClick fixes 2373 updated snapshots adds discussionBoard feature toggle (#2370) remove sampleValue from application.conf add /api/features api exposing part of the config allow users to access their own time tracking data allow multiple datastore auth tokens ensure that getNRandomColors always returns n colors even for n > 500 fix usage of fossildb in docker-compose (#2358) CI smoke test: don't daemonize, use circleCI background (#2357) fix regression in AbstractTreeViewer due to new EdgeCollection data structure (fixes #2353) added dataset sorting to dataset gallery view #2322 updated the default sort order of datasets on dashboard

2018-02-28

WIP: pagination for volume data downloads

2018-01-11

Add fossildb (#2235) * add fossildb download and executable * refactoring fossildb download * fossildb: adding external version file * add sbt replacement that starts fossildb * run dockerized fossildb in compose * add fossildb * docker-compose: fix fossildb host in dev service * Update run.sh * Update sbt * Update run.sh * Update sbt * move fossilDB-usingt datastore code from braingames-libs repo to wk * update protobuf plugin version (no more startling start-up exception) * remove cmd.sh, again

2017-12-25

organize scala packages in datastore (part 2)

reorganize packages in datastore (WIP)

integrate braingames-datastore subproject into webknossos-datastore

2017-12-21

create braingames-datastore subdirectory in preparation for libs assimilation

2017-10-23

add user bounding box to skeleton and volume tracings

2017-09-27

rename skeleton tree timestamp to createdTimestamp

2017-09-18

Fixing volume tracings, WIP

2017-09-10

WIP: proto integration, compiling, but not tested

WIP transition to protobuf: update actions operate on generated classes

2017-09-07

WIP wontcompile: protocol buffers for keyvaluestore