Proto commits in tomasreimers/tensorflow-emscripten

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2017-02-08

Rename the DEBUG logging level to DEBUGGING. Some software projects compile C/C++ code with -DDEBUG (equivalent to -DDEBUG=1) in debug builds. Notably, this is done by default in debug builds on iOS. Therefore, we should avoid using DEBUG as an identifier in C/C++ code. Change: 146840049

The documentation is generated from this commit.

2017-02-07

Addition of Outfeed HLO op. Change: 145772331

[XLA] Move fast-math flags into HLO module config. Previously, XLA controlled the presence/absence of fast-math flags (FMF) via a command-line flag. This patch changes things so we use a new CompileOptions proto instead. This proto lives in HloModuleConfig, and is passed to the service via ExecuteRequest. This change lets us entirely remove llvm_backend_flags.{h,cc}. In addition, this change takes us from two to one fast-math flags. Previously we tried to control "unsafe FP transformations" separately from "full fast math". It turns out that LLVM is misleadingly inconsistent in how it handles these. In the backend, they are indeed two separate options that can be enabled/disabled independently. In the frontend, however, unsafe-fp-math implies all the other FMFs. As a result, it doesn't really make sense for XLA to attempt to split out these two flags, at least not until LLVM changes how it handles them. Change: 146183994

[XLA] Update comment on RNG_UNIFORM in xla_data.proto Change: 145810242

[XLA] Move shape_with_output_layout and seed fields into ExecutionOptions. This simplifies the execution API by getting rid of two default params. Also change HloModuleConfig so it stores each of the fields of ExecutionOptions individually, instead of keeping an instance of the ExecutionOptions proto. This is necessary because HloModuleConfig already has a field derived from shape_with_output_layout -- if we stored the ExecutionOptions proto in HloModuleConfig, its shape wouldn't necessarily match the shape we already have. Change: 146477669

[XLA] Rename CompilationOptions proto to ExecutionOptions. We want to put fields in this proto that aren't strictly related to compilation. Change: 146477500

Added optional string argument to infeed HLO op. Change: 145188452

2017-02-03

Add a mechanism for hiding, skipping, and modifying the generated op functions for C++. A souped-up version of the hidden_ops mechanism in Python, the intent is to use this for most or all of the client languages, with a common list of changes to make in a common file and per-language overrides. Also: * include the documentation for outputs in the generated comments * several updates to C++ API to match Python * fix C++ shape function for ConcatV2 now that we use it by default * split op_gen_lib out of core:framework, since it is only used by the op generators, and I don't want to add another proto to mobile builds Change: 146267344

2017-01-12

Delete the FunctionDef.Node code paths, now that we have switched to the NodeDef representation. Change: 144281952

2017-01-11

Add support for list(func) AttrValues. Change: 144211572

2017-01-09

Initial open-source release of XLA: Accelerated Linear Algebra. XLA is a compiler-based linear algebra execution engine that targets CPUs, GPUs and custom accelerators. XLA is still experimental; we are releasing it early to get the community involved. Change: 143990941

2017-01-06

Allow empty tf.Features in the FeatureLists for sparse sequence examples. Change: 143802382

2016-12-23

Combine NamedTensorProto and NamedTensor into a single proto. Also use `Tensor::AsProtoTensorContent()` when populating the fetched values from a gRPC worker service, as this is more efficient for larger values. This should improve #6256 slightly. Change: 142813084

2016-12-15

Merge changes from github. Change: 142074581

2016-12-14

Merge BUILD file changes

Optimize the case of a client communicating with an in-process master. This change introduces a new `LocalMaster` class, which enables distributed TensorFlow sessions to communicate directly with a master in the same process, without making a loopback RPC call. This improves the performance on some distributed session microbenchmarks: * The median step time for a minimal step decreases from 392us to 273us. * The median step time for a step that feeds 4MB of data decreases from 9.26ms to 5.26ms. This change is a first step towards addressing issue #6256. The code contains TODO comments with several further potential improvements: * TODO(mrry): Add methods that avoid protobuf encoding the request/response objects where this affects performance. * TODO(mrry): Avoid closure creation/context switch overhead for synchronous invocation of Master methods. * TODO(mrry): Apply a similar change to the WorkerInterface. Change: 141950977

2016-12-13

Update comments for Tensor proto tensor_content field (#6151)

2016-12-12

Fix typo in master_service.proto documentation Change: 141803185

2016-12-08

tfdbg core: remove unnecessary construction of DebuggerState when there is no watch Hitchhiking change: * A typo fix Change: 141386349

2016-12-07

Add a :debug BUILD target which, when linked into a binary, enables DirectSession support for TensorFlow Debugger (tfdbg). Binaries that do not want debugging support can avoid this dependency and its transitive deps. This replaces the previous approach that was based on a preprocessor flag (-DNOTFDBG). Change: 141321165

2016-12-02

Change proto package: tensorflow -> tensorflow.contrib.tensorboard to avoid conflict with existing tensorflow.TensorInfo defined in tensorflow/core/protobuf/meta_graph.proto Change: 140890467

2016-11-30

Add a trace annotator for python files. Change: 140606458

2016-11-29

Add experimental option for compilation. Change: 140517341

2016-11-28

Adding person detection/tracking sample Activity to Android TensorFlow demo. Change: 140411483

2016-11-23

BigQuery table partition accessor to be used in BigQuery Reader. This code implements a component to read table partitions from BigQuery using the HTTP API for BigQuery. It specifically uses tabledata.list mehtod. This implementation uses Application Default Credentials (similar to GCS Filesystem). The current implementation does not support nested types. Change: 140058933

2016-11-22

A proto that applications other than TensorFlow can use to send data to GraphExplorer. Change: 139937019

2016-11-17

Adding __version__ and __git_version__ information into saved meta graphs. Change: 139471934

Uses 64-bit integers for storing variable dimensions in SaveSliceInfoDef. 32-bit integers are not enough for representing large scale variables with more than ~2 billion elements. The change is backward compatible as the proto serialization of int32s is the same as int64s. Change: 139393418

2016-11-16

Automated rollback of change 137740850 Change: 137747341

Automated rollback of change 137731142 Change: 137740850

2016-11-15

In call_cpp_shape_fn, use whether the shape inference fn requested input tensor values to rerun the shape inference function. This removes the need to pass input_tensors_needed and input_tensor_as_shapes_needed. Remove calls to call_cpp_shape_fn that pass input_tensors_needed or inptu_tensor_as_shapes_needed. Change: 139228180

2016-11-09

Internal change. Change: 138451572

2016-11-07

Reload the checkpoint reader when the latest checkpoint has changed. The embedding projector plugin caches the checkpoint reader, which is pointing to the latest checkpoint when TensorBoard was started. But after some time, the saver will remove that old checkpoint file (keeps only N latest checkpoints), while the reader is still pointing to it. Also add more tests regarding the V1 and V2 checkpoint versions. Change: 138416977

2016-11-04

Replace usages all_variables -> global_variables, GraphKeys.VARIABLES -> GraphKeys.GLOBAL_VARIABLES Change: 138212111

2016-11-02

Distributed partial run support. This change completes support for distributed partial run. RPC/Proto changes: - PartialRunSetup RPC is added to the Master interface. - A is_partial bool is added to the RunStepRequest. MasterSession implementation: - When the MasterSession gets a PartialRunSetupRequest, it constructs the graph and registers the partitions. - When a is_partial RunStepRequest is received, the feeds and fetches are validated, and corresponding is_partial RunGraphRequests are sent to the workers. Change: 137989990

2016-10-31

Automated rollback of change 137740850 Change: 137747341

Automated rollback of change 137731142 Change: 137740850

Adds functionality for shape inference and dtype checking of variable-based resources. Does not implement these resources yet as this change is big enough already. Change: 137731142

2016-10-29

Add is_last_partial_run to partial RunGraph Request. Change: 137587827

2016-10-28

Add sprite image support to the backend. Change: 137540630

Add sprite information to the projector config. Also serve the projector_config.proto from the python server as json, instead of introducing a custom mapping. Change: 137460915

2016-10-26

Populate cost graph in workers, aggregate and return them in master. Change: 137245840

2016-10-25

Add tfprof python API to tf.contrib and move tfprof CLI to tensorflow/tools. Change: 137207286

2016-10-20

Replace enum BusAdjacency with protobuf DeviceLocality for describing the topological neighborhood of a device. Change: 136663586

2016-10-17

Add experimental partial run to worker interface. Change: 136373434

2016-10-15

Added the data type to the cost graph Change: 136219453

2016-10-14

Adds a resource reference dtype to TF. Change: 136199499

2016-10-13

Simplify api for tf.summary.tensor_summary and tf.summary.scalar tf.summary.tensor_summary and tf.summary.scalar now use "name" rather than "tag" to organize what data they contain. This re-uses the existing TF name mechanism rather than inventing a new one. Additional metadata (e.g. a type hint for the data) is now communicated via a SummaryDescription protocol buffer rather than attr fields. There are some unused fields on the TensorSummary op that are being left for backwards compatibility. Change: 136095678

2016-10-12

Build the cost model every k steps after the first l steps: this makes it possible to skip the first iterations that are much slower than the rest, and also make it possible to subsample the collection of statistics to avoid impacting the performance of the model too much. Change: 135961158

2016-10-10

Add an option to the projector in server mode to start the projector with bookmarks pre-loaded. Also add a count to the bookmark tray. Change: 135718868

2016-10-08

Add tensor output shapes to CostGraphDef. Update StepStatsCollector::BuildCostModel() to populate these fields. Misc. cleanup and optimizations in step_stats_collector.cc and costmodel.cc. Change: 135541791

2016-10-07

Allow writing the MetaGraphDef to an event summary. Supervisor and graph_actions now do this by default if saving a graph is requested. TensorBoard can now fall back to using the MetaGraphDef if the GraphDef is not available in the Event. Other changes include: * factor out metagraph code from training.saver to framework.meta_graph * update the summary writer, session hooks, and supervisor to support these new summaries Change: 135424442

2016-09-26

Automated rollback of change 133191013 Change: 134297651

2016-09-23

Tensor bundle: a module to efficiently serialize/deserialize tensors. This module will serve as the backing store of TensorFlow's V2 checkpoint format. Highlights of the benefits: * Peak memory usage of both write and read paths are just 1x (plus a small constant-sized buffer). * Stores tensors in a non-proto format, circumventing the size limit on proto messages. * Flexibility: potentially allows by-row loading of a large tensor, or mmap. Change: 134028119

2016-09-22

Pass benchmark name (path:target_name) to TestResults proto. Change: 134000963

tfprof: "Swiss Army Knife Tool" To Explore Your Model. Change: 133968335

Integrate the embedding projector backend into TensorBoard. * Introduce a simple plugin architecture for plugins to add their HTTP handlers and data to TensorBoard. Each plugin implements a simple interface defined in base_plugin.py. Each plugin P exports its own API methods to tf.tensorboard.plugins.P * The goal of this plugin system is minimal coupling between tensorboard and plugins so we can remove tensorboard/plugins/... later and replace it with the new plugin architecture - only 10 lines of code added to tensorboard/backend/handler.py will need to be removed. * Add and register the embedding projector under the name `projector`. Also export tf.contrib.tensorboard.plugins.projector.visualize_embeddings(summary_writer,config) to the tensorflow API. * The embedding projector writes its own data in a separate file but in the same logdir as the summary writer. Change: 133908053

2016-09-21

Add timeline_step to GraphOptions. Change: 133882443

Merge changes from github. Change: 133874452

Add AssetFile proto to meta graph def. Change: 133852423

2016-09-16

Merge commit for internal changes

2016-09-15

Automated rollback of change 133191013 Change: 133258312

Merge commit for internal changes

fix comment grammar (#4381)

2016-09-14

tfdbg: 1st step toward GRPC debug targets: Protobuf service definition This CL defines the basic protocol buffer message and service that will be used in the communication between the debug ops and a debug server. The debug server is designed to run, e.g., inside the TensorBoard backend. It listens to intermediate tensor values (and potentially other information) encapsulated as Event protos published by the debug ops. The response message from the debug server is currently just an empty acknowledgment, but can hold fields if so desired in the future. Change: 133191013

Added the ability to remap the cost model node ids: this is needed since these ids are only unique inside a partition. Added a test to check the partitioning of the inception model with 8 towers Change: 133160985

2016-09-13

Distributed runtime bfloat16 casting. Change: 133062008

Added to_proto and from_proto for CondContext and WhileContext. Change: 133013495

2016-09-08

Cosmetic: Consistently avoid using a namespace specifier when the 'using' statement in the file obviates it. Change: 132517534

2016-09-01

Added the node compute costs to the cost graph Change: 131997704

2016-08-31

Add new QueueRunner optional argument: queue_closed_exception_types. * This is a backwards compatible change. * Includes some extra helper functions in the tf.errors module. * Includes extension to the QueueRunner proto * Includes tests that QueueRunner.{from,to}_proto are backwards compatible. Change: 131791267

2016-08-26

Automated rollback of change 131356339 Change: 131410487

Automated rollback of change 131340536 Change: 131356339

Add a 'visible to virtual' GPU device remapping to ConfigProto. Allows one to remap the visible GPUs to virtual GPUs on a per-session basis. For example, if the visible devices are 0,1,2,3,4,5,6,8, setting visible_device_list='5,3' means that visible device 5 gets mapped to /gpu:0 and visible device 3 maps to /gpu:1. Tested manually on my local machine: our tests are single GPU only so there's no good way to test this ongoing. Fixes #1888. Change: 131340536

2016-08-25

graph.proto no longer needs to export NodeDef, clients should use node_def.proto instead. Change: 131290995

2016-08-23

Initial version of the Go API. The API is subject to change. Remaining work to do: - Generated ops. - Generated protocol buffers. - A few calls requiring protocol buffers aren't in this change. Change: 131066649

Begin transition to use NodeDef in FunctionDef instead of FunctionDef.Node. Change: 131009401

2016-08-22

Split NodeDef out of graph.proto into node_def.proto. Needed so we can use NodeDef in FunctionDef. Change: 130982373

2016-08-17

Introduce a CheckpointFormatVersion to SaverDef. This unlocks the opportunity for experimenting w/ different checkpoint formats. Change: 130473327

2016-08-15

Enable output of partition graphs via RunMetadata for DirectSession::Run() calls Change: 130271393

2016-08-01

Replace a minomer: s/SaveRestoreHelper/Saver. Change: 129002390

2016-07-30

Debugger: File IO utils + File dumping from DebugOps * Add debug_urls to existing Debug Ops to allow specification of debug URLs. * Support file debug URLs, such as "file:///tmp/tfdbg_dump_1", where the path specifies a directory in which the dump files will be generated for intermediate tensors during a Session::Run() call. For example, given that the node name is "foo/bar/node_a", the output slot index of the dumped tensor is 0, and the debug Op is "DebugIdentity", the full path of the dump file will be: "/tmp/tfdbg_dump_1/foo/bar/node_a_0_DebugIdentity_${WALL_TIME_US}", where WALL_TIME_US is the wall timestamp for when the dumped tensor is generated, in microseconds. * The debug_urls list of strings can be contain multiple individual URLs, e.g., {"file:///tmp/dump_1", "file:///tmp/dump_2"}, in which case, each debug signal tensor will be dumped to all specified paths. * Add C++ and Python unit tests. Other related changes: * Move debugger- (tfdbg-)related build targets from core/BUILD to core/debug/BUILD Future change lists will implement GRPC debug URL targets. Change: 128873549

2016-07-25

Change for internal compatibility.

2016-07-22

Initial submit of SavedModel proto and related additions to MetaGraph proto. Change: 128193561

2016-07-20

Automated rollback of change 127562075 Change: 127906463

2016-07-18

Automated rollback of change 127562075 Change: 127709092

2016-07-15

tfdb: Debug nodes inserter EXPERIMENTAL: Insert special debug ops (e.g., DebugIdentity) to graph for debugging. Currently, debug ops need to take exactly one input and has the string attribute "tensor_name" to indicate what tensor it watches. For example, before the node insertion, the graph may look like: A:0 -----------1----------> B | ---------2-----------> C wherein the output slot 0 of node A feeds as the input to nodes B through edge 1 and to node C through edge 2. After the node insertion, assuming both B and C have non-Ref input, the graph becomes: A:0 ---3---> Copy -----------4----------> B | ---------5--------> C | ---------6--------> X If a node (e.g., B) has Ref input, the graph becomes: ----------------4---------------> B | A:0 ---3-----> Copy -----------5----------> C | -----------6--------> X In other words, we do not feed Refs to deep-copies to downstream nodes. The Copy node is the inserted deep-copy node that copies the input tensor on-device (e.g., CPU-to-CPU or GPU-to-GPU deep copy) that reduces the likelihood of racy updates during debug tensor-watching. X is the newly created debug node that transforms the input (copy of the watched tensor) into a debug signal. DebugIdentity is the simplest debugging paradigm, in which the debug signal (i.e., X:0) equals the tensor itself. More sophisticated debug ops can be used to transform the tensor into other useful debug signals. An example is the added DebugNanCounter op. If the nodes (A, B and C) are located on GPU and the edges from A to B or C is HOST_MEMORY, the CopyHost op will be used instead of the Copy op. A reserved string attribute "debug_url" is created for the debug ops to make it possible to send debug signals to files or RPC calls in the future. Other points worth noting: * The debug ops have control-edge connections to the original destination node, in order to ensure that the debug signals are deterministically generated before the destination node executes. * More than one debug ops can be added to watch a tensor. * A new field called "DebugTensorWatch" is added to RunOptions to support debug node insertion. * A new method GPUUtil::CopyGPUTensorToSameGPU has been added to make GPU-to-GPU deep-copy of tensors possible. * The two test files (debug_gateway_test.cc and debug_gateway_gpu_test.cc) have been consolidated to the former, by using the GOOGLE_CUDA macro. Change: 127562075

2016-07-11

Change tensorflow/contrib/session_bundle namespace from tensorflow::contrib to tensorflow::serving. Update all Python users of tensorflow_serving/session_bundle/manifest.proto to use the equivalent tensorflow/contrib/session_bundle/manifest.proto since there are otherwise namespace collisions. Change: 127129383

2016-07-06

Parse tf.Example using arena allocation. Dense string benchmark shows 10-20% speedup. Change: 126726018

2016-07-01

First step for adding the new TensorSummary op: Add fields to summary.proto Change: 126453617

2016-06-29

Python tensorflow.Example parser configuration extractor - Proto definition for configuration - Utility for converting from proto - Visibility change Change: 126145305

2016-06-15

Extend documenation for SequenceExample. Change: 124962631

2016-06-11

Drop unused import of tensor_description.proto. Change: 124625745

2016-06-03

Phase 1 of moving TensorFlow Serving SessionBundle to tensorflow/contrib. Change: 123975418

2016-06-02

Update copyright for 3p/tf/core. Change: 123900938

2016-05-31

Add experimental qualification in the comment on the inter-op session thread pools. Change: 123690916

2016-05-27

Add serial_number field to MachineConfiguration message in test_log.proto Change: 123455485

Include control edges in cost model. Change: 123452806

2016-05-23

Merge changes from github. Change: 123026122

Enables GPU Tracer in DirectSession. Disables GPU tracing when building a CostModel. Fixes some Timeline parsing issues related to CSE and GPUTracing. Turns off memory visualization by default. Change: 123014754