Proto commits in graphcore/tensorflow

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2022-11-07

FP8 Conv Ops Summary: FP8 conv implementation that tries to maintain the usual conv APIs. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Subscribers: awf Maniphest Tasks: T65648 Differential Revision: https://phabricator.sourcevertex.net/D76354

The documentation is generated from this commit.

2022-09-05

FP8 Matmul custom op Summary: Provide the FP8 * FP8 matmul operation Test Plan: included Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jackh, alfiee Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Maniphest Tasks: T65647 Differential Revision: https://phabricator.sourcevertex.net/D72812

2022-08-04

Pass the executable options to serielize/deserialize Summary: Make sure the engine is re-created with correct options. Fix T67132 TF1.15 Only Test Plan: Poprun tests pass CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, christiana, frederikm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, christiana, frederikm Maniphest Tasks: T67132 Differential Revision: https://phabricator.sourcevertex.net/D72622

Pass the executable options to serielize/deserialize Summary: Make sure the engine is re-created with correct options. Fix T67132 Test Plan: Poprun tests pass CI Reviewers: christiana, frederikm, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: christiana, frederikm, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T67132 Differential Revision: https://phabricator.sourcevertex.net/D72619

2022-07-12

Implement ConvertToF8/ConvertFromF8 custom ops Summary: This commit adds two custom ops for converting to and from f8 tensors represented by tuple of u8 data and u8 metadata scalar. Fix tuple support in `HloPoplarTestBase::ExecuteNoHloPasses` for f8_test. Fix T65650. Test Plan: CI, new numeric f8_test Reviewers: georgep, samuelh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: samuelh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T65650 Differential Revision: https://phabricator.sourcevertex.net/D71192

2022-06-15

Replace selects which could be computed in compile time with mask fusion Summary: This commit adds new path - mask_finder that searches for select with condition we can compute at compile time. If this select has broadcast of constant as one of its true/false operands, we can replace such select with a sequence of poplar copy() Fix T36290. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk, gauthamg Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk Subscribers: georgep Maniphest Tasks: T36290 Differential Revision: https://phabricator.sourcevertex.net/D41764

2022-06-06

Use remote buffers to store entry computation arguments and results, when available. Summary: What's changed: - Use remote buffers instead of data streams to handle entry computation arguments and results. - Saves HEXOPT space when there are large streams. - Disabled This opens the option to only copy exactly what we need, instead of unconditionally copying everything. Ref T63600 Test Plan: CI tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, gauthamg, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, gauthamg, babakk Subscribers: babakk, tomm Maniphest Tasks: T63600 Differential Revision: https://phabricator.sourcevertex.net/D67730

2022-05-10

Add allow-non-inplace flag to fusion config Summary: This commit adds flag to fusion config allowing fusions to indicate if they support both inplace and non-inplace variants of lowering. Fix T61776. Test Plan: CI, no functional changes Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, babakk Maniphest Tasks: T61776 Differential Revision: https://phabricator.sourcevertex.net/D66706

Add allow-non-inplace flag to fusion config Summary: This commit adds flag to fusion config allowing fusions to indicate if they support both inplace and non-inplace variants of lowering. Fix T61776. Test Plan: CI, no functional changes Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, babakk Maniphest Tasks: T61776 Differential Revision: https://phabricator.sourcevertex.net/D66706

2022-04-26

Make inputs to tensor lists uninitialised Summary: To prevent copies before the while loops mark these inputs as uninitialised Reviewers: vladimirm, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, yanislavd Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, yanislavd Subscribers: babakk Maniphest Tasks: T53525 Differential Revision: https://phabricator.sourcevertex.net/D64827

Make inputs to tensor lists uninitialised Summary: To prevent copies before the while loops mark these inputs as uninitialised Reviewers: vladimirm, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, yanislavd Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, yanislavd Subscribers: babakk Maniphest Tasks: T53525 Differential Revision: https://phabricator.sourcevertex.net/D64827

2022-04-25

Add softmax and stable softmax as ipu ops. Summary: Allow users to target the poplibs softmax and stable softmax from the python frontend. This doesn't replace the TF2XLA softmax because I'm not sure whether it's always better. Ref T59577 Test Plan: Added a softmax test. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm Subscribers: babakk, georgep Maniphest Tasks: T59577 Differential Revision: https://phabricator.sourcevertex.net/D65228

Add a `StaticMultiUpdateAdd` instruction Summary: Add a multi update add instruction, in which the update indices are a static attribute. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk Subscribers: babakk, georgep Maniphest Tasks: T54057 Differential Revision: https://phabricator.sourcevertex.net/D63985

2022-04-22

Add softmax and stable softmax as ipu ops. Summary: Allow users to target the poplibs softmax and stable softmax from the python frontend. This doesn't replace the TF2XLA softmax because I'm not sure whether it's always better. Ref T59577 Test Plan: Added a softmax test. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm Subscribers: babakk, georgep Maniphest Tasks: T59577 Differential Revision: https://phabricator.sourcevertex.net/D65228

2022-04-21

Add a `StaticMultiUpdateAdd` instruction Summary: Add a multi update add instruction, in which the update indices are a static attribute. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, babakk Subscribers: babakk, georgep Maniphest Tasks: T54057 Differential Revision: https://phabricator.sourcevertex.net/D63985

2022-03-23

Merge branch 'poplar/r2.5/release' into poplar/r2.6/release

2022-03-15

Add a StaticMultiSlice HLO instruction Summary: Add a StaticMultiSlice HLO instruction Test Plan: Numerical test Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm, alfiee, jackh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Subscribers: harrym Maniphest Tasks: T54057 Differential Revision: https://phabricator.sourcevertex.net/D62944

Add a StaticMultiSlice HLO instruction Summary: Add a StaticMultiSlice HLO instruction Test Plan: Numerical test Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm, alfiee, jackh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Subscribers: harrym Maniphest Tasks: T54057 Differential Revision: https://phabricator.sourcevertex.net/D62944

2022-02-25

Add CloneMethod_DeduceNewOrderOrBypass Summary: This commit adds deducing bypass method, allowing to pass input as is unless it has an allocation target. Ref T51153. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, samuelh, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, samuelh Maniphest Tasks: T51153 Differential Revision: https://phabricator.sourcevertex.net/D61820

Add CloneMethod_DeduceNewOrderOrBypass Summary: This commit adds deducing bypass method, allowing to pass input as is unless it has an allocation target. Ref T51153. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, samuelh, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, samuelh Maniphest Tasks: T51153 Differential Revision: https://phabricator.sourcevertex.net/D61820

2022-02-21

Provide option to optimise for latency Summary: Provide an option to aim to reduce number of packets at all costs. Extend all our visitors to have an {Ap/Pre}pendToSequence and always force {In/Out}feed programs to be added by these methods `TF2.5 only` version of D60642 lint Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, jackh, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jackh Subscribers: alfiee, marieanne, vladimirm, harrym, babakk, jackh Maniphest Tasks: T54941 Differential Revision: https://phabricator.sourcevertex.net/D61243

Provide option to optimise for latency Summary: Provide an option to aim to reduce number of packets at all costs. Extend all our visitors to have an {Ap/Pre}pendToSequence and always force {In/Out}feed programs to be added by these methods `TF1.15 only` Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, jackh, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jackh, babakk Subscribers: jackh, babakk, harrym, vladimirm, marieanne Maniphest Tasks: T54941 Differential Revision: https://phabricator.sourcevertex.net/D60642

Adding config option for controlling how much tile memory a dynamic-slice can use before being replaced. Summary: Adding `ipu_config.slices.replace_dynamic_slice_threshold` config option for controlling how much tile memory a dynamic-slice can use before being considered for replacement. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, samuelh, vladimirm, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: godfrey.da.costa, samho Maniphest Tasks: T54306 Differential Revision: https://phabricator.sourcevertex.net/D61016

Adding config option for controlling how much tile memory a dynamic-slice can use before being replaced. Summary: Adding `ipu_config.slices.replace_dynamic_slice_threshold` config option for controlling how much tile memory a dynamic-slice can use before being considered for replacement. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, samuelh, vladimirm, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: godfrey.da.costa, samho Maniphest Tasks: T54306 Differential Revision: https://phabricator.sourcevertex.net/D61016

2022-02-03

Add deferred allocation for deduce/bypass copies Summary: Explicit copies prevent loop parameters from having proper layout. This commit adds deferred allocation support for copies. Fix T55199. Test Plan: CI, new test Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, samuelh, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T55199 Differential Revision: https://phabricator.sourcevertex.net/D59864

Add deferred allocation for deduce/bypass copies Summary: Explicit copies prevent loop parameters from having proper layout. This commit adds deferred allocation support for copies. Fix T55199. Test Plan: CI, new test Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, samuelh, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T55199 Differential Revision: https://phabricator.sourcevertex.net/D59864

2022-01-26

Adding TF op for gcl::allReduceWithinReplica Summary: Adding op/kernel/inst and poplar op def for gcl::allReduceWithinReplica, follows the usual pattern. Adds the python function within_replicas.all_reduce which accepts a list of sharded tensors and returns the reduced results gathered over all the shards. Test Plan: New tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee, yanislavd, vladimirm, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, yanislavd, vladimirm, samuelh Maniphest Tasks: T53767 Differential Revision: https://phabricator.sourcevertex.net/D59164

2022-01-25

Adding TF op for gcl::allReduceWithinReplica Summary: Adding op/kernel/inst and poplar op def for gcl::allReduceWithinReplica, follows the usual pattern. Adds the python function within_replicas.all_reduce which accepts a list of sharded tensors and returns the reduced results gathered over all the shards. Test Plan: New tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee, yanislavd, vladimirm, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, yanislavd, vladimirm, samuelh Maniphest Tasks: T53767 Differential Revision: https://phabricator.sourcevertex.net/D59164

Adding TF op for gcl:reduceScatterWithinReplica Summary: Initial op/kernel/inst/opdef for calling gcl::reduceScatterWithinReplica. Follows the usual pattern. Adds the python function within_replicas.reduce_scater which accepts a list of sharded tensors and returns a tuple of reduced results scattered over the shards. TF2.4 Only (TF1 - D59367) Test Plan: New tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Maniphest Tasks: T52884 Differential Revision: https://phabricator.sourcevertex.net/D58826

Adding initial ops for targetting gcl::AllGatherWithinReplica call. Summary: Initial op/kernel/inst/opdef for calling gcl::AllGatherWithinReplica. Follows the usual pattern. Adds the python function `within_replicas.all_gather` which accepts a list of sharded tensors and returns a gathered tensor for each shard via a tuple. Test Plan: New tests. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, dominicm, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: vladimirm, georgep Maniphest Tasks: T52751 Differential Revision: https://phabricator.sourcevertex.net/D58119

Some bugprone and performance fixes Summary: Linter fixes Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: jakeh, vladimirm, babakk, georgep Maniphest Tasks: T53781 Differential Revision: https://phabricator.sourcevertex.net/D58742

2022-01-21

TF1 - Adding TF op for gcl:reduceScatterWithinReplica Summary: Initial op/kernel/inst/opdef for calling gcl::reduceScatterWithinReplica. Follows the usual pattern. Adds the python function within_replicas.reduce_scater which accepts a list of sharded tensors and returns a tuple of reduced results scattered over the shards. TF1.15 Only (TF2 - D58826) Original diff failed to Merge due to BUILD differences but also had a runtime difference in the python API (tensor.ref doesn't exist in TF1) Test Plan: New tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee Subscribers: alfiee Maniphest Tasks: T52884 Differential Revision: https://phabricator.sourcevertex.net/D59367

2022-01-18

Adding initial ops for targetting gcl::AllGatherWithinReplica call. Summary: Initial op/kernel/inst/opdef for calling gcl::AllGatherWithinReplica. Follows the usual pattern. Adds the python function `within_replicas.all_gather` which accepts a list of sharded tensors and returns a gathered tensor for each shard via a tuple. Test Plan: New tests. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, dominicm, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: vladimirm, georgep Maniphest Tasks: T52751 Differential Revision: https://phabricator.sourcevertex.net/D58119

2022-01-07

Merge branch 'poplar/r2.4/release' into poplar/r2.5/release

2021-12-21

Target poplibs GeluErf Summary: What's changed: - Target poplibs gelu_erf. TF2.4 Only TF2 version of D57755. Resolves T52832 Test Plan: CI + new test case. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: georgep Maniphest Tasks: T52832 Differential Revision: https://phabricator.sourcevertex.net/D58115

Target poplibs GeluErf Summary: What's changed: - Target poplibs gelu_erf. TF1.15 Only Another diff for TF2 Ref T52832 Test Plan: CI + new test case. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: georgep Maniphest Tasks: T52832 Differential Revision: https://phabricator.sourcevertex.net/D57755

2021-12-15

Add ipu.control_flow_ops.barrier Summary: Add a barrier op to force control flow. Ref T52106 TF2.4 Only Test Plan: CI, added new tests Reviewers: babakk, samuelh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: babakk, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T52106 Differential Revision: https://phabricator.sourcevertex.net/D57600

Add ipu.control_flow_ops.barrier Summary: Add a barrier op to force control flow. Ref T52106 TF1.15 Only Test Plan: CI, added new tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, babakk, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, babakk Subscribers: vladimirm, douglaso Maniphest Tasks: T52106 Differential Revision: https://phabricator.sourcevertex.net/D57602

2021-12-03

Add copy config and clone method to backend config Summary: This commit allow copy instruction to specify which clone method it would like to use for the output tensor. Fix T51151. Test Plan: CI, new test, check tile balance for copies with CloneMethod_PreserveOrderUnlessAliases Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, jackh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, jackh Subscribers: jackh, georgep Maniphest Tasks: T51151 Differential Revision: https://phabricator.sourcevertex.net/D56691

Add copy config and clone method to backend config Summary: This commit allow copy instruction to specify which clone method it would like to use for the output tensor. Fix T51151. Test Plan: CI, new test, check tile balance for copies with CloneMethod_PreserveOrderUnlessAliases Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, jackh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, jackh Subscribers: jackh, georgep Maniphest Tasks: T51151 Differential Revision: https://phabricator.sourcevertex.net/D56691

2021-12-02

Fuse scale with reduction Summary: Fixes T42432 popops supports fusing single element, f32 scale with reduction having one of the following types ADD, LOG_ADD, SQUARE_ADD Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: georgep Maniphest Tasks: T42432 Differential Revision: https://phabricator.sourcevertex.net/D55667

Fuse scale with reduction Summary: Fixes T42432 popops supports fusing single element, f32 scale with reduction having one of the following types ADD, LOG_ADD, SQUARE_ADD Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: georgep Maniphest Tasks: T42432 Differential Revision: https://phabricator.sourcevertex.net/D55667

2021-11-30

Rename `IpuInterCopy` as `InterIpuCopy` Summary: This commit renames all mentions of `IpuInterCopy` in camel-case, kebab- case and snake-case with their `InterIpuCopy` counterpart, which sounds better, and is easier to search. Fixes T49914. Test Plan: Use the existing tests. This is an aesthetic change only. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T49914 Differential Revision: https://phabricator.sourcevertex.net/D56748

Rename `IpuInterCopy` as `InterIpuCopy` Summary: This commit renames all mentions of `IpuInterCopy` in camel-case, kebab- case and snake-case with their `InterIpuCopy` counterpart, which sounds better, and is easier to search. Fixes T49914. Test Plan: Use the existing tests. This is an aesthetic change only. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T49914 Differential Revision: https://phabricator.sourcevertex.net/D56748

Revert InterIpuCopy rename Summary: This reverts commit c34fbba7204f. Test Plan: revert-hammer Reviewers: Subscribers:

Revert InterIpuCopy rename Summary: This reverts commit 5e8286bbb96b. Test Plan: revert-hammer Reviewers: Subscribers:

Rename `IpuInterCopy` as `InterIpuCopy` Summary: This commit renames all mentions of `IpuInterCopy` in camel-case, kebab- case and snake-case with their `InterIpuCopy` counterpart, which sounds better, and is easier to search. Fixes T49914. Test Plan: Use the existing tests. This is an aesthetic change only. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T49914 Differential Revision: https://phabricator.sourcevertex.net/D56224

Rename `IpuInterCopy` as `InterIpuCopy` Summary: This commit renames all mentions of `IpuInterCopy` in camel-case, kebab- case and snake-case with their `InterIpuCopy` counterpart, which sounds better, and is easier to search. Fixes T49914. Test Plan: Use the existing tests. This is an aesthetic change only. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T49914 Differential Revision: https://phabricator.sourcevertex.net/D56224

2021-11-29

Revert “Fuse scale with reduction” Summary: This reverts commit 872c25960fc50e216984f1c9f415fb3bfe94d7a8. Test Plan: revert-hammer Reviewers: Subscribers:

Revert “Fuse scale with reduction” Summary: This reverts commit db36495955742f40dc5ce7eafe1bf9096479ceef. Test Plan: revert-hammer Reviewers: Subscribers:

2021-11-26

Implement GradientAccumulatorAddWithScale Summary: Replace GradientAccumulatorAdd with GradientAccumulatorAddWithScale, which takes in an additional scale parameter for scaling the accumulator value prior to accumulation. The objective is to enable accumulation of the type `acc <- acc * acc_scale + grad * grad_scale`, which will enable us to support a running mean. In this implementation, `grad * grad_scale` will be computed in Python and passed to the new op along with `acc_scale` REF T46005 TF2.4 Only TF1 diff: D53583 Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Maniphest Tasks: T46005 Differential Revision: https://phabricator.sourcevertex.net/D55699

Implement GradientAccumulatorAddWithScale Summary: Replace GradientAccumulatorAdd with GradientAccumulatorAddWithScale, which takes in an additional scale parameter for scaling the accumulator value prior to accumulation. The objective is to enable accumulation of the type `acc <- acc * acc_scale + grad * grad_scale`, which will enable us to support a running mean. In this implementation, `grad * grad_scale` will be computed in Python and passed to the new op along with `acc_scale` REF T46005 TF1.15 Only TF2 diff: D55699 Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Maniphest Tasks: T46005 Differential Revision: https://phabricator.sourcevertex.net/D53583

Fuse scale with reduction Summary: Fixes T42432 popops supports fusing single element, f32 scale with reduction having one of the following types ADD, LOG_ADD, SQUARE_ADD Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: georgep Maniphest Tasks: T42432 Differential Revision: https://phabricator.sourcevertex.net/D55667

Fuse scale with reduction Summary: Fixes T42432 popops supports fusing single element, f32 scale with reduction having one of the following types ADD, LOG_ADD, SQUARE_ADD Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm Subscribers: georgep Maniphest Tasks: T42432 Differential Revision: https://phabricator.sourcevertex.net/D55667

2021-11-24

Revert "Implement GradientAccumulatorAddWithScale" This reverts commit 4e7bb3206235438527ea5d2fd11cf14ca5288936.

Revert "Implement GradientAccumulatorAddWithScale" This reverts commit f6eb0436d06329d8e4267a238e0f2bc728a581ff.

2021-11-23

Implement GradientAccumulatorAddWithScale Summary: Replace GradientAccumulatorAdd with GradientAccumulatorAddWithScale, which takes in an additional scale parameter for scaling the accumulator value prior to accumulation. The objective is to enable accumulation of the type `acc <- acc * acc_scale + grad * grad_scale`, which will enable us to support a running mean. In this implementation, `grad * grad_scale` will be computed in Python and passed to the new op along with `acc_scale` REF T46005 TF2.4 Only TF1 diff: D53583 Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Maniphest Tasks: T46005 Differential Revision: https://phabricator.sourcevertex.net/D55699

Implement GradientAccumulatorAddWithScale Summary: Replace GradientAccumulatorAdd with GradientAccumulatorAddWithScale, which takes in an additional scale parameter for scaling the accumulator value prior to accumulation. The objective is to enable accumulation of the type `acc <- acc * acc_scale + grad * grad_scale`, which will enable us to support a running mean. In this implementation, `grad * grad_scale` will be computed in Python and passed to the new op along with `acc_scale` REF T46005 TF1.15 Only TF2 diff: D55699 Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, vladimirm, samuelh Maniphest Tasks: T46005 Differential Revision: https://phabricator.sourcevertex.net/D53583

2021-11-12

Adding missing return statements. Summary: Precursor to setting `-Werror=return-type` compiler flag which makes missing return statements an error. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T49252 Differential Revision: https://phabricator.sourcevertex.net/D55485

Adding missing return statements. Summary: Precursor to setting `-Werror=return-type` compiler flag which makes missing return statements an error. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T49252 Differential Revision: https://phabricator.sourcevertex.net/D55485

2021-11-04

Combine multiple gather operations in to AllGather Summary: - Create a colocator to merge multiple gather operations in to a single AllGather operation where possible. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, alfiee, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee, babakk Subscribers: georgep Maniphest Tasks: T48296 Differential Revision: https://phabricator.sourcevertex.net/D54111

Combine multiple gather operations in to AllGather Summary: - Create a colocator to merge multiple gather operations in to a single AllGather operation where possible. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, alfiee, babakk Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, alfiee, babakk Subscribers: georgep Maniphest Tasks: T48296 Differential Revision: https://phabricator.sourcevertex.net/D54111

2021-11-02

Remove _profiling from IPUConfig Summary: Removes the hidden _profiling category from IPUConfig that was added to temporarily support tests that still used profiling features. Also remove auto_assign_report_subdirectories from the internal config protobuf and executor as it's now redundant. Depends on D54505 Fixes T39600 Applies to both branches. Test Plan: CI. Removed relevant IPUConfig tests. Reviewers: alfiee, georgew, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T39600 Differential Revision: https://phabricator.sourcevertex.net/D54432

Remove _profiling from IPUConfig Summary: Removes the hidden _profiling category from IPUConfig that was added to temporarily support tests that still used profiling features. Also remove auto_assign_report_subdirectories from the internal config protobuf and executor as it's now redundant. Depends on D54505 Fixes T39600 Applies to both branches. Test Plan: CI. Removed relevant IPUConfig tests. Reviewers: alfiee, georgew, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T39600 Differential Revision: https://phabricator.sourcevertex.net/D54432

2021-10-29

Update PrngSeedState to call poplar::setStochasticRounding Summary: Adding StochasticRoundingMethod_None option so stochastic rounding can be disabled/enabled via the PrngSeedState class. This makes it easier to keep calls to poplar::setStochasticRounding in sync with the order of poplar program execution, as we already do that for the other stochastic rounding modes. Test Plan: CI + New C++ tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T48700 Differential Revision: https://phabricator.sourcevertex.net/D54429

Update PrngSeedState to call poplar::setStochasticRounding Summary: Adding StochasticRoundingMethod_None option so stochastic rounding can be disabled/enabled via the PrngSeedState class. This makes it easier to keep calls to poplar::setStochasticRounding in sync with the order of poplar program execution, as we already do that for the other stochastic rounding modes. Test Plan: CI + New C++ tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T48700 Differential Revision: https://phabricator.sourcevertex.net/D54429

2021-10-27

Removing deterministicWorkers backend option Summary: Removing deterministicWorkers backend option since it's a global setting and so cant be set per instruction, which was the original intention. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T48677 Differential Revision: https://phabricator.sourcevertex.net/D54293

Removing deterministicWorkers backend option Summary: Removing deterministicWorkers backend option since it's a global setting and so cant be set per instruction, which was the original intention. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T48677 Differential Revision: https://phabricator.sourcevertex.net/D54293

Add Poplar checks into embedded runtime Summary: Ref T48682 Test Plan: CI Reviewers: jakeh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: jakeh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T48682 Differential Revision: https://phabricator.sourcevertex.net/D54281

Add Poplar checks into embedded runtime Summary: Ref T48682 Test Plan: CI Reviewers: jakeh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: jakeh, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T48682 Differential Revision: https://phabricator.sourcevertex.net/D54281

Remove HloReplicationIndexInstruction Summary: Fix T47046 Test Plan: CI Reviewers: babakk, vladimirm, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: babakk, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T47046 Differential Revision: https://phabricator.sourcevertex.net/D54265

Remove HloReplicationIndexInstruction Summary: Fix T47046 Test Plan: CI Reviewers: babakk, vladimirm, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: babakk, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T47046 Differential Revision: https://phabricator.sourcevertex.net/D54265

2021-10-21

Add reduce-mean support in reduce-scatter Summary: REF T47313 Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm, jackh, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm, jackh, hakons Subscribers: hakons Maniphest Tasks: T47313 Differential Revision: https://phabricator.sourcevertex.net/D53827

Add reduce-mean support in reduce-scatter Summary: REF T47313 Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm, jackh, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, vladimirm, jackh, hakons Subscribers: hakons Maniphest Tasks: T47313 Differential Revision: https://phabricator.sourcevertex.net/D53827

2021-09-23

Adding enable_experimental_prng stability flag. Summary: Adding enable_experimental_prng stability flag for conditionally enabling work related to sr/prng seed management. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, markf Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T46295 Differential Revision: https://phabricator.sourcevertex.net/D51859

Adding enable_experimental_prng stability flag. Summary: Adding enable_experimental_prng stability flag for conditionally enabling work related to sr/prng seed management. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, markf Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T46295 Differential Revision: https://phabricator.sourcevertex.net/D51859

2021-09-17

Adding StochasticRoundingMethod backend option (TF1) Summary: Fixing up D51361 for TF1 - Removed TF2 specific optypes from NeedsSpecificSeedType This change adds the StochasticRoundingMethod option to the backend config. This is intended to be used as an explicit way of describing how we want to perform stochastic rounding (with an identical seed/differing seed or either). By having an extra backend option we avoid having to overload the meaning of being replica identical and having to add an extra category to the replica dataflow analysis, which will further complicate it. StochasticRoundingMethod gets set by the AddStochasticRoundingOptions so only instructions which require a specific type of seed will cause the seeds to be changed. It's currently setup so that instructions which read/restructure data don't change the seed. TF1.15 Only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T45895 Differential Revision: https://phabricator.sourcevertex.net/D51636

Adding StochasticRoundingMethod backend option Summary: This change adds the `StochasticRoundingMethod` option to the backend config. This is intended to be used as an explicit way of describing how we want to perform stochastic rounding (with an identical seed/differing seed or either). By having an extra backend option we avoid having to overload the meaning of being replica identical and having to add an extra category to the replica dataflow analysis, which will further complicate it. `StochasticRoundingMethod` gets set by the `AddStochasticRoundingOptions` so only instructions which require a specific type of seed will cause the seeds to be changed. It's currently setup so that instructions which read/restructure data don't change the seed. (TF1 - D51636) TF2.4 Only Test Plan: C++ Tests Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T45895 Differential Revision: https://phabricator.sourcevertex.net/D51361

2021-09-15

Allow merging of the remote buffers across identical clusters Summary: Follow the logic in subcomputation graph caching and compare elementwise cluster computations. Allow merging buffers across identical clusters so they could be reused. Propagate all merged indices so new size will be changes in all remote buffer info structures. Fix T45972. Test Plan: CI, fixed HW test Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, hakons Maniphest Tasks: T45972 Differential Revision: https://phabricator.sourcevertex.net/D51403

Allow merging of the remote buffers across identical clusters Summary: Follow the logic in subcomputation graph caching and compare elementwise cluster computations. Allow merging buffers across identical clusters so they could be reused. Propagate all merged indices so new size will be changes in all remote buffer info structures. Fix T45972. Test Plan: CI, fixed HW test Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, hakons Maniphest Tasks: T45972 Differential Revision: https://phabricator.sourcevertex.net/D51403

2021-09-14

Move `enable_fast_math` to algebraic simplifier config Summary: TF2.4 Only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf Subscribers: georgep, vladimirm Maniphest Tasks: T45300 Differential Revision: https://phabricator.sourcevertex.net/D51300

Move `enable_fast_math` to algebraic simplifier config Summary: TF1.15 Only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf Subscribers: jamiep, vladimirm, georgep Maniphest Tasks: T45300 Differential Revision: https://phabricator.sourcevertex.net/D51010

Make accumulation count of resource update runtime input Summary: Provide gradient accumulation op inside the resource update, and use this when finding the gradient accumulation count later of resource updates. Also adapt passes to handle this as an optional. TF2.4 only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, markf Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, markf Subscribers: markf, jakeh Maniphest Tasks: T41151 Differential Revision: https://phabricator.sourcevertex.net/D50184

Make accumulation count of resource update runtime input Summary: Provide gradient accumulation op inside the resource update, and use this when finding the gradient accumulation count later of resource updates. Also adapt passes to handle this as an optional. TF1.15 only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, markf Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: markf, jakeh Maniphest Tasks: T41151 Differential Revision: https://phabricator.sourcevertex.net/D51231

2021-09-13

Merge branch 'poplar/r2.4/release' into poplar/r2.4/merge

2021-09-06

Use CollectiveBalancedReorder for replicated tensor sharding clusters Summary: This commit provides both host and runtime rearrangements for the clusters with reduce-scatter/all-gather to ensure the minimal exchange is required. Fix T35351. Overview of the changes: Replicated resource update elementwise clustering pass: - Marks elementwise clusters with 'partitioned_elementwise_cluster' attribute to allow custom visitor for those clusters later. - Insert new custom instructions: collective-rearrange and undo-collective-rearrange before reduce-scatter and after all-gather for unpartitioned remote buffers. Add replicated elementwise cluster visitor, and add additional validation rules in it. GCL collective balance reorder may return any particular shape depending on the input layout, so validate it not only against XLA shape, but also replica slice and collectives tensor. Add host rearrangement for remote buffers in poplar executor class. This is host-side equivalent of the collective-reorder/undo-collective-reorder instructions. Test Plan: CI, additional host rearrangement code in replicated_resource_update_elementwise_clustering_hw_test. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, hakons Subscribers: hakons Maniphest Tasks: T35351 Differential Revision: https://phabricator.sourcevertex.net/D44634

Use CollectiveBalancedReorder for replicated tensor sharding clusters Summary: This commit provides both host and runtime rearrangements for the clusters with reduce-scatter/all-gather to ensure the minimal exchange is required. Fix T35351. Overview of the changes: Replicated resource update elementwise clustering pass: - Marks elementwise clusters with 'partitioned_elementwise_cluster' attribute to allow custom visitor for those clusters later. - Insert new custom instructions: collective-rearrange and undo-collective-rearrange before reduce-scatter and after all-gather for unpartitioned remote buffers. Add replicated elementwise cluster visitor, and add additional validation rules in it. GCL collective balance reorder may return any particular shape depending on the input layout, so validate it not only against XLA shape, but also replica slice and collectives tensor. Add host rearrangement for remote buffers in poplar executor class. This is host-side equivalent of the collective-reorder/undo-collective-reorder instructions. Test Plan: CI, additional host rearrangement code in replicated_resource_update_elementwise_clustering_hw_test. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh, georgep, hakons Subscribers: hakons Maniphest Tasks: T35351 Differential Revision: https://phabricator.sourcevertex.net/D44634

Add dot strengh reduction optimisation Summary: TF2.4 Only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, hakons Subscribers: hakons, vladimirm, markf Maniphest Tasks: T44870, T45300 Differential Revision: https://phabricator.sourcevertex.net/D50808

2021-09-03

Add dot strengh reduction optimisation Summary: TF1.15 Only Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, hakons Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, hakons Subscribers: markf, vladimirm, hakons Maniphest Tasks: T44870, T45300 Differential Revision: https://phabricator.sourcevertex.net/D50520

2021-08-30

Add gradient accumulation count op Summary: To be used to track dynamic counts for resource update op Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf, georgep Subscribers: markf Maniphest Tasks: T41151 Differential Revision: https://phabricator.sourcevertex.net/D50437

Add gradient accumulation count op Summary: To be used to track dynamic counts for resource update op Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, markf, georgep Subscribers: markf Maniphest Tasks: T41151 Differential Revision: https://phabricator.sourcevertex.net/D50437

2021-08-18

Track whether an executable can stall on lack of inputs Summary: Track whether the compiled module can stall without more data. Ref T41143 Test Plan: CI Pipeline already tested Added a test for IO tiles which stalled. Reviewers: jakeh, gauthamg, hakons, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: jakeh, hakons, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T41143 Differential Revision: https://phabricator.sourcevertex.net/D49943

Track whether an executable can stall on lack of inputs Summary: Track whether the compiled module can stall without more data. Ref T41143 Test Plan: CI Pipeline already tested Added a test for IO tiles which stalled. Reviewers: jakeh, gauthamg, hakons, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Reviewed By: jakeh, hakons, #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved Maniphest Tasks: T41143 Differential Revision: https://phabricator.sourcevertex.net/D49943

Adding ReduceMany op and colocation helper Summary: TF2.4 Only Adding hlo-only ReduceMany op. Reductions (including fusion reductions) can be combined into these ReduceMany ops. The interface to control this is this is a new ipu config option optimizations.maximum_reduce_many_buffer_size. This also has a change to the clustering scheduler to prevent a memory regression in a test. The change makes it so that ops with a valid colocator helper will not be put into their own cluster if the buffer size for the colocator is zero. This will prevent colocator helpers added in future from affecting the schedule in unrelated models. V1 Diff: D49202 Test Plan: Tests check that that simple reduces and reduce fusions get combined in hlo according to the specified optimizations.maximum_reduce_many_buffer_size. There is also a test which executes a graph with a ReduceMany and checks the output values. Reviewers: #tensorflow, simonl, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: georgep Maniphest Tasks: T40084 Differential Revision: https://phabricator.sourcevertex.net/D48529

Adding ReduceMany op and colocation helper Summary: TF1.15 Only Adding hlo-only ReduceMany op. Reductions (including fusion reductions) can be combined into these ReduceMany ops. The interface to control this is this is a new ipu config option optimizations.maximum_reduce_many_buffer_size. This also has a change to the clustering scheduler to prevent a memory regression in a test. The change makes it so that ops with a valid colocator helper will not be put into their own cluster if the buffer size for the colocator is zero. This will prevent colocator helpers added in future from affecting the schedule in unrelated models. V2 Diff: D48529 Test Plan: Tests check that that simple reduces and reduce fusions get combined in hlo according to the specified optimizations.maximum_reduce_many_buffer_size. There is also a test which executes a graph with a ReduceMany and checks the output values. Reviewers: #tensorflow, simonl, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: zhenyingl, mariok, georgep Maniphest Tasks: T40084 Differential Revision: https://phabricator.sourcevertex.net/D49202

2021-08-16

Remove verified streams Summary: Fixes T43482. TF2.4 Only. Test Plan: Tested with Poplar with the public API removed as in D49012. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh Maniphest Tasks: T43482 Differential Revision: https://phabricator.sourcevertex.net/D49852

Remove verified streams Summary: Fixes T43482. TF1.15 Only. Test Plan: Tested with Poplar with the public API removed as in D49012. Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, anthonyb, jakeh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, jakeh Maniphest Tasks: T43482 Differential Revision: https://phabricator.sourcevertex.net/D49757

2021-08-13

Populate gcl::CollectiveBalancedHostRearrangement objects in PoplarExecutableCore Summary: This commit creates all host rearrangement objects in advance and speeds up run preparations. Fix T44246. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, hakons, jakeh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T44246 Differential Revision: https://phabricator.sourcevertex.net/D49526

Populate gcl::CollectiveBalancedHostRearrangement objects in PoplarExecutableCore Summary: This commit creates all host rearrangement objects in advance and speeds up run preparations. Fix T44246. Test Plan: CI Reviewers: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep, hakons, jakeh Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Maniphest Tasks: T44246 Differential Revision: https://phabricator.sourcevertex.net/D49526

2021-08-05

Adding poplar options flags for slice operations Summary: Adds a slices.poplar_options dictionary to the config similar to matmuls.poplar_options. The options specified get passed into calls to popops::multiSlice, popops::multiUpdate, popops::multiUpdateAdd, and popops::embedding::plan. Slice options can also be specified per-pipeline-stage as part of PipelineStageOptions. Reviewers: #tensorflow, simonl, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Reviewed By: #tensorflow, #framework_ip_review_-_any_oss_or_third-party_code_use_has_been_approved, georgep Subscribers: georgep Maniphest Tasks: T42623 Differential Revision: https://phabricator.sourcevertex.net/D49190