Proto commits in ValveSoftware/GameNetworkingSockets

These 36 commits are when the Protocol Buffers files have changed:

2024-03-13

Delete a bunch of transmit and receive speed distribution stats stuff. It is unused, and when I actually do hook something up, I think I will rework it. I decided to comment it out instad of delete it, because it will make it easier to find all the places in the code I need to touch again. P4:8762717

The documentation is generated from this commit.

Delete a bunch of transmit and receive speed distribution stats stuff. It is unused, and when I actually do hook something up, I think I will rework it. I decided to comment it out instad of delete it, because it will make it easier to find all the places in the code I need to touch again. P4:8762717

The documentation is generated from this commit.

2023-05-28

Groundwork for automatically fixing up out-of-order packets. If they arrive in quick succession. This has shipped on the relay, but the implementation for flows received by the end hosts will be different and is not implemented yet. P4:8088074

Added a field so certs can be bound to particular IP address. This is useful for short-term automatically issued certs, for example the ones that relays use. Not currently supported in the opensource code, for certs issues to hosts. (But it could be, I guess!) P4:8088054

2022-04-08

Move shared stuff to CConnectionTransportP2PICE::LocalCandidateGathered. Also added some notes on ping times and locking that need to be investigated. p4:7194271

2021-10-08

Two changes to the design of lanes - Added a system for strict priority classes, in addition to the "weight" system. The combination of these two makes for a very flexible system, while also hopefully being easy to understand. - Each lane has its own message number sequence. This has several advantages: - It makes it clear that the order of messages between lanes is not guaranteed - It greatly simplifies packet encoding/decoding. (Going to make some more progress on that next.) (Based on discussions with jeffreyh@valvesoftware.com. Thanks!) P4:6818689,6818692

2021-10-06

Support for using multiple wifi bands. Context: https://twitter.com/ZPostFacto/status/1445748202926342144 NOTE: This is *not* currently useful in the opensource code, so STEAMNETWORKINGSOCKETS_ENABLE_DUALWIFI is never defined. It depends on the sender negotiating with the receiver that a secondary source IP:port address will be associated with the same connection, and that has not been implemented for ordinary UDP connections at this time. (On Steam it is part of the SDR protocol.) When I do that work, I probably will do it in conjunction with some other planned work to support doing all UDP connections on the same socket, and using the connection ID for multiplexing instead of IP:port tuples. This strategy not only is more efficient, due to fewer sockets being used, but it also makes it very easy to support the peer roaming to a different IP:port without the connection dropping, which can happen in mobile environments and is a characteristic of QUIC.

2021-09-06

Can now send application messages via signaling Will be used by the "Fake UDP port" system I am adding to Steam. P4:6753643

2021-05-29

[SteaM only] Added "FakeIP" system. This won't work without Steam (STEAMNETWORKINGSOCKETS_ENABLE_FAKEIP won't ever be defined here). But to make sure we can always share the headers between this code and Steamworks SDK, the interface functions will still exist. P4:6571466,6571469

2021-05-28

Tweak P2P stats P4:6570979

Fix rogue em-dash (en-dash?) that's confusing protoc

2020-12-17

Plumb through more P2P analytics fields

2020-10-29

P2P refactor for hosted dedicated server connections This change has a bunch of refactoring to allow connections to be made on Steam to servers in SDR known data centers. Previously, the app needed to have a central matchmaking authority (what we call the "game coordinator") issue a ticket. Now, the client may initiate the connection as an ordinary P2P connect, not knowing that the server is in a known data center. The server will issue a ticket and sign it with its own cert and then send back the ticket via signaling. Then the client will switch to use the optimized routing for this case.

2020-07-04

Add "symmetric connect" mode This is a P2P feature for a common use case: - The two peers are "equal" to each other. (Neither is clearly the "client" or "server".) - Either peer may initiate the connection, and indeed they may do this at the same time - The peers only desire a single connection to each other, and if both peers initiate connections simultaneously, a protocol is needed for them to resolve the conflict, so that we end up with a single connection. Also added remote virtual port to ConnectP2PCustomSignaling

2020-05-28

Several P2P improvements Add a connection time to CMsgSteamDatagramLinkLifetimeStats. Connection will track when it entered and left theconnected state, so that when it fills out this record, it can populate this field. Change default value of P2P_Transport_ICE_Enable. We don't have a "platform" to ask for user preference in this opernsource lib. Adjust some asserts so that if they fire because some transport is trying to process packets in the wrong state, we won't crash. TRack how long each P2P transport was selected. Added some SDR transport session stats (in Steam). Fix a compiler warning. Remove an instance where RTTI is used. (Many people prefer to compile without RTTI.)

Improvements to ICE error handling Change how CSteamNetworkConnectionBase::GuessTimeoutReason defaults are set. Just set them directly in CSteamNetworkConnectionBase::ConnectionTimedOut. What we were doing previously was a bit convoluted. Add logic to actually attempt to classify ICE failure.

2020-05-20

P2P / ICE imprvoements This change includes many changes to improve P2P connections. I didn't bother replaying them one at a time from perforce. Changed the plumbing of how data packets are sent, using context structures more. The main goal of these changes is to remove the assumption that data packets are always sent/received on the currently selected transport. Also, I removed two specialized end-to-end messages for ICE transport and SDR P2P transport. Now all end-to-end packets are ordinary data packets (just without any data). This is good because, for UDP, they are encrypted. (For SDR connections we don't encrytpt them, because we want the relays to be able to observe them and perhaps mutate them.) And is reduces the number of different packet types and codepaths that need to be supported. This does have one unfortunate downside: the receiver does not know which transport is used to send dropped packets. So, any out-of-band checks on alternate transports, which do not contain data, and are much more likely to drop (since the transport is not selectged) will be observed as a gap in the end-to-end sequence number. Oddly, the sender actually will have the more complete picture (albeit after some delay), since he knows the outcome of every single packet, and he knows what transport was used to send it. Added a flag that can be used to indicate to the peer, "I am not using this transport as my primary transport." My protocol does not have an explicit nomination framework. Instead, any data packet that doesn't have this flag is considered a nomination. SNP now needs to remember what transport was used for each outgoing packet. Added some plumbing so that I can collect data about how often ICE is successful, how the routes compare to SDR, and if it fails, what progress it made. The ICE session interface offers more control over what kind of candidtes to gather, and returns more detailed info about the candidates and route. The goal here is to allow a semi-private mode. If you don't want to share your public address with random Internet peers (to avoid getting DDoSed), but you do want to get a LAN connection to any peers in your office / dorm (where LAN broadcasts may not discover that you are on the same LAN), you can choose to share only local addresses. This seems oddd to only share the "private" address while keeping your "public" address a secret -- but in this case the "private" address is actually more sensitive, and the public address is the one that script kiddies would need to boot you. We are dealing with a differet kind of thread here. The option to go totally private and disable ICE completely, or only share relayed candidates, or only share public addressses, are all still available, if the threat model is different.

2020-05-17

Refactor rendezvous messages. Promoted concept of a stream of reliable rendezvous messages to the connection layer. That is not a thing that only ICE might find useful. (Although right now, only ICE is using it.) Manually base-64 encode the ICE credential values, and only use the lower 30 bits, so that we get exactly 5 characters, without any '=' padding. Added IsControllingAgent(), which has been promoted to a connection layer The terminology is from ICE, but the concept is generally useful. I will use that concept when deciding which transport to use (ICE or SDR), to avoid asymmetric routing. Fix bug where we might try to send a queued reliable rendezvous message before we know who we are sending it to. Added code to take appropriate action when ICE tells us the writable state has changed. Previously we were relying on polling, but now we definitely need something to kick-start the thinking loop. Added some hack code to try to load up the steamwebrtc library manually, if we have no CreateICESessionFunc function already. Refactored P2P rendezvous handling a bit, moving more of it into a member function. This is a wire-protocol-breaking change. I didn't bother doing any versioning because that is not necessary yet.

2020-05-12

Plumb through proper STUN credentials. The "user fragment" is the connection ID, which is already being shared. And the password is a random number. Both of these values are Base64 encoded. Changed the server to be the controlling agent. This is opposite of the ICE default convention, but ICE worksjust fine either way. The reason I want to have the clientbe the controlling agent is that this same role assignmehnt will apply to the higher level routing decision of whether to use the ICE route or relay over SDR. And in themost immediate use case I am working on (steam remote play), the server will have some convars available that might notbe accessible on the client (e.g. a phone). This is kind of arbitrary and maybe not optimal for future use cases. I could plumb thorugh some options to control this later, but for now this is an OK default. Changed the way that initial ICE settings were plumbed through. Pass them in a struct to the factory func, instead of making them callbacks on the delegate. Changed the way that candidates were sent back and forth in signals. Used a more generic reliable stream of messages instead. Also made the receiving a bit more robust, if a message contains some retransmission and also new data. (Previously it would ignore everything if the first message was not the next thing expected.) Set webrtc network thread priority.

Make ICE connections wire equivalent with plain UDP. (Once NAT is pierced, and assuming we are not relaying through TURN, and ignoring the actual STUN packets.) Refactored code so that almost all of the UDP code is shared. Having this exact wire compatibility is really important for the future. I had assumed that the WebRTC code was putting some connection IDs, etc in the envelope, but I was wrong. I believe that is happening at a higher layer in their code (the DTLS layer), and I'm below that. I am pretty sur ethat I could get rid of the special "ping check" messages, and in fact they probably are just delaying the connection and I should trust the WebRTC code. Will probably revisit this later.

2020-04-26

Rename "WebRTC" to "ICE" in more places

2020-04-25

ICE connections thorugh WebRTC working. Not happy with this yet. We need to use a lower level interface in WebRTC, since all we want is ICE, not DTLS, SIP, and bandwidth estuimation and pacing. But it basically works.

2020-04-16

Delete old, unused stuff from UDP messages

More progress on WebRTC connections. Need to pass the transport to SNP_SendPacket. We cannot assume that the current transport is the one who filled out the context and is sending the packet. Got some signaling messages working to send answer/offer and candidates. This is probably more complicated than neccessary, but it is because I want to assumevery little from the signaling framework. Specifically, it is *not* assumed to be reliable, and I assume that really large messages might not work. I do not currently assume that messages will gitin a UDP packet, although limiting signals to <2K or so seems like a good idea. The WebRTC connections are managing to get connected. Now I need to have logic to update the SteamNetworkingSockets state machine, and if we have SDR, I need to intelligently decide when to use the NAT-punched route versus the relayed route. I'm expectig that the relayed route could be faster in some cases, although I don't know how often. I am pretty sure that I am using too much WebRTC, and all I really want is the ICE NAT punching bit, and this will eventually get rewritten.

2020-04-14

First steps for P2P via WebRTC - Got WebRTC transport class started. - Added steamwebrtc interface header. This is some work that SamL did for Steamlink, to isolate all the webrtc build requirements from our code. I'm not sure what the final version will look loike, but this is a good place to start, since this library already exists and I can get a proof of concept working. After we're working, I'll probably peel back the layers a bit and refactor this, and then eventually figure out how to opensource that work. - Added config values to set the STUN server list

2020-04-12

Working on P2P connections Take most of the guts of SDR relayed connections and promote them to base classes. Work in progress, the P2P connection code does not yet compile, and so it is notin the makefiles yet. There are still a few places where we assume we are SDR relayed.

2019-09-18

Added cipher agility. The only other currently supported cipher is the NULL cipher, which we want for remote streaming because we don't want to pay cost to encrypt/decrypt on mobile. But eventually we might want to add other ciphers, and this is the mechanism to do it. This required shuffling around the crypto handshake quite a bit, because we want the app to be able to set configuration options on incoming cnonections before accepting them.

2019-09-17

Refactor to split connection from transport. The connection will be responsible for end to end stuff including encryption, message fragmentation and reliability, etc. Transports deliver datagrams. A transport is always associated with a single connection, but a connection may have more than one transport, and may switch between transports over the course of the connection lifetime. The goal here is to have conenction types that can switch transport. Specifically, we need to support a connection that can detect when peers are on the same LAN and use ordinary UDP in that case, attempt NAT piercing using STUN, and if that fails, then relay (either using TURN or SDR). There is more work to be done here. For example, each transport almost certainly has its own ping time and quality characeristics, so we probably ought to track some stats there. (But our stats situation is already getting pretty crazy, so maybe just track ping seperately). This change also includes some changes relevant for Steam Remote Play streaming, which is going to use this protocol for relayed connections. (And eventualy, hopefully, for all connections.) For example, steam remote play needs really high bandwidth and packet rate, and the tolerance for sending the timing data for jitter tracking was too tight. I also need to pass a "certificate" to a subprocess in a single blob, which actually includes the private key.

2019-09-02

Always serialize identity using text. This is a bit more wasteful, but it facilitates forward and backward compatibility. If one peer is using an identity type that an old peer doesnt' parse, they can still often communicate. The old peer often doesn't need to really "understand" what the identity is. Also fixed some parsing bugs in SteamAPI_SteamNetworkingIdentity_ParseString.

2019-08-29

Added interface for app to provision cert. Also: - STEAMNETWORKINGSOCKETS_STEAM now mens "running on steam", not "running using the steam client". STEAMNETWORKINGSOCKETS_STEAMCLIENT is for that. - Refactored stats stuff, moved it into the namespace. At one point I thought I might expose some stuff in a public interface. For now, keeping it internal. - Removed concept of Steam "universe" from this branch of the code. - Don't use OVERRIDE, override works.

2019-04-04

Implement a rudimentary PKI. Store a list of public keys and certs we have for them, and implement logic to evaluate their trust. New cert checking flow in the connection code. I found a *tiny* bug where we were not actually checking the signature of the connection crypt info using the signature in the user cert (!). We were checking to make sure that the user cert was valid, but anybody in the middle could have substituted their own ephemeral keypair for key exchange, and we wouldn't have known. Certs can now be issued to multiple AppIDs. Cert tool can now emit json output and bind certs to particular AppIDs.

2019-01-23

Use 96-bit AES-GCM IV's. Secure the protocol number. Switched AES-GCM IV's to be 96-bit instead of 128-bit. - This is what TLS uses (RFC5288) - This is what NIST recommends (https://dl.acm.org/citation.cfm?id=2206251) - This is more optimal. (Certainly for OpenSSL's implementation; I believe it's true in general for GCM.) - This is the only value that libsodium supports. Note that we're still using a 32-bit tag, and libsodium only supports the full 128-bit tag (to my knowledge, see #48). But we might decide to use the full 128-bit tag, and that that point libsodium could be used. Moved the protocol number out of transport-specific handshake packets to be a "crypt parameter", where it will be signed. In the future we might need to allow some clients to connect to older clients that have a known security vulnerability (perhaps only during some temporary transition period). We don't want an attacker to be able to forge the protocol version and force the weaker protocol to be negotiated. Bumped the protocol version because these are wire-compatibility-breaking changes. Added a specific error code if a connection fails due to protocol incompatibilities.

2019-01-03

Sync up with work internally in the Steam code. * Interface changes: Replaced all CSteamIDs with SteamNetworkingIdentity, which is an abstract concept. Added IPv6 support. This has not been tested at this time, but all the basics are there and any issues should be small bugfixes. Improved and standardized connection debug names. You can set a debug name for a connection, and the full debug string, which includes the Connection ID, transport-specific data, and your name (if any) are printed consistently in logging messages and this name is also present in SteamNetConnectionInfo_t. Connection handles are now the same as the local ConnectionID on the wire. This makes debugging a lot easier. GetListenSocketInfo renamed to GetListenSocketAddress and now actually does something. Split out P2P and ordinary UDP listen sockets into two seperate objects. This simplifies the opensource code (since P2P is not support) * Misc internal code cleanup: Broke up CCrypto monolith, so that we can more easily and cleanly: - Share what we really want to share with Steam. Mainly some text parsing of keys, etc. - Replace the actual crypto primitives with another provider (e.g. libsodium) Delete a whole bunch of unused and messy cruft that was left when we ripped this out from Steam, as well as code and messages specific to relayed connections (supported only on Steam) Switch a bunch of maps to hashmaps instead of red/black tree maps. The CUtlHashMap class comes from Steam and is designed to scale to very large size (N in the millions), and if the table needs to be rehashed, the cost is amortized instead of triggering a large cost all at once. This functionality is not needed. Nor do we need stable handles, so it would be better to use a table based on open addressing. That would be a small optimization and also allow us to get rid of bitstring.*, which unfortunately got added as a result of this change. Added standalone IPv6 address parsing files: ipv6text.*. These are plain C and do not have any dependencies, and could be split out, since they are independenctly useful and, unlike all of our other utility routes (e.g. tier0, tier1, etc), actually worthy of being used outside of Valve.

2018-09-10

Integrate end-to-end stats tracking acks with SNP data acks. This not only removes some redundancy, but also is a step towards rejecting duplicate sequence numbers earlier. (#20) Also rework how hosted dedicated servers are configured, but that is not relevant to the opensource version of this code.

2018-04-23

Always send stats inline, never standalone. Previously there were some cases where the connection code would need to send stats, and it wasn't able to piggy-back them on a data packet. A seperate standalone stats message was used for this. Changed to plumbing so that the connection code could initiate sending of an SNP data packet, but ask SNP to not use the whole packet, and pass through some data, back to the connection-type-specific code. This way all stats can be sent inline in a data packet. This is a step towards getting rid of the LinkStatsTracker's acks. Those are still useful for stats on the relay leg of relayed connections, but for end-to-end, SNP already does acks, and so it's silly and complicated to have two different mechanisms for this. Tweaked ack delay values and timeouts. If the ping was realy low, then if a receiver waited until the timeout and was forced to flush the acks (no piggy-back opportunity presented itself), then the sender's timeout was exactly the same, and so there was no room for error and the ack would just barely timeout. Also when a packet is NACKed, spew a bit more detail about what actually caused the nack (either an actual nack from the other side, or ack timeout). Changed some constants related to PingLocation stuff. That isn't really applicable to this lib, but I've made some improvements to that system in the Steam code.

2018-03-30

Initial checkin. There's a lot of work that needs to be done to make this more open-source friendly, especially on Windows. The README has notes on the known shortcomings and what should be coming soon.