package scrape_repos

Get desktop application:
View/edit binary Protocol Buffers messages

A single content file record.

• optional string clone_from_url = 1

• optional string relpath = 2

• optional int32 artifact_index = 3

• optional bytes sha256 = 4

• optional int32 charcount = 5

• optional int32 linecount = 6

• optional string text = 7

The configuration for the contentfiles importer.

Used in: LanguageToClone

• optional string source_code_pattern = 1

  A regular expression to match repo-relative file paths. E.g. the pattern "src/.*\.c" would match any files in a repository's 'src/' directory ending in suffix '.c'. This pattern must match the entire path. E.g. the pattern '\.c$' would match only a single file called '.c'. The start and end of match characters '^' and '$' are permitted but redundant.

• repeated string preprocessor = 2

  A list of preprocessor passes to run on each imported source code, in the order in which they should be executed. A preprocesor is a Python function, decorated with the @datasets.github.scrape_repos.preprocessors.public.dataset_preprocessor decorator. THe name of the decorator is the fully qualified python module, followed by ':' and the name of the Python function, e.g. datasets.github.scrape_repos.preprocessors.extractors:JavaMethods See //datasets/github/scrape_repos/preprocessors/... for definitions.

The "metafile" schema. Each repository which is scraped produces one of these files, recording various attributes about the repository.

• optional int64 scraped_unix_epoch_ms = 7

  The number of milliseconds since the Unix epoch that the repository was scraped or cloned.

• optional int64 cloned_unix_epoch_ms = 8

• optional string owner = 1

  The GitHub username of the repository owner.

• optional string name = 2

  The name of the repository as it appears on GitHub.

• optional string clone_from_url = 3

  The git URL to clone the repo.

• optional int32 num_stars = 4

  The number of stargazers, forks, and watchers of the repository.

• optional int32 num_forks = 5

• optional int32 num_watchers = 6

A GitHub query to search for repositories.

Used in: LanguageToClone

• optional string string = 1

  The query string.

• optional int32 max_results = 2

  The maximum number of results to process for this query. Less results will be processed if the query returns less than this number of results. Note that the GitHub API may limit the maximum number of results which are returned (currently this limit is 1,000 results per query).

• optional string clone_from_url = 1

• optional string clone_dir = 2

• optional string abspath = 3

• repeated string all_files_relpaths = 4

• repeated string preprocessors = 5

• optional string index_dir = 6

The schema for "clone lists", which are used to determine the GitHub queries to run and repos to clone.

• repeated LanguageToClone language = 1

A single programming language to clone repositories of.

Used in: LanguageCloneList

• optional string language = 1

• repeated GitHubRepositoryQuery query = 2

  A list of queries to search for repositories on GitHub. Queries are searched in the order they appear, until num_repos_to_clone repositories have been cloned. For parameters, see the GitHub API docs: https://developer.github.com/v3/search/#search-repositories

• optional string destination_directory = 3

  The base directory to clone GitHub repositories to.

• repeated ContentFilesImporterConfig importer = 6

  The configuration for the program which imports source code from cloned repositories into contentfile databases.

• repeated string clone_from_url_blacklist = 7

  An optional list of HTTPS repo URLs to ignore, e.g. "https://github.com/ChrisCummins/phd.git". Before a repo is scraped, the URL is checked against this list, and ignored if it matches. Black listing is case sentive - only lower case letters should be used.

• repeated ListOfStrings list_of_strings = 1

Used in: ListOfListOfStrings

• repeated string string = 1

Used by JavaMethodsExtractor to extract a list of methods from an input source.

• repeated string method = 1