MapleStoryDetectionSampleGenerator

Generate Machine Learning Samples Object Detection In MapleStory

Performance

This generator can generate arbitrarily many annotated samples. All bounding boxes are precisely annotated based on rendering coordinates.

With RTMDet and ~10000 samples, it can achieve 97.3%mAP in test set.

Requirement

Build

  1. Clone this repository with submodules by
    git clone --recursive git@github.com:charlescao460/MapleStoryDetectionSampleGenerator.git.
    Note that --recursive is necessary.
  2. Build MapleStoryDetectionSampleGenerator.sln
  3. Run MapleStory.MachineLearningSampleGenerator\bin\Release\net10.0-windows7.0\WzComparerR2.exe and open MapRender once. Running WzComparerR2.exe will generate Setting.config, which is required for our MapRender invoker.

Run

(Assuming assemblies are built with Release configuration. Debug configuration is similar)

  1. Use WzComparerR2.exe to find the desired map you want to sample. Assuming 993134200.img is the map you want in Limina.
  2. From the solution root, prepare a YAML config file. A checked-in example is available at Examples\sample-generator.yml.
  3. If you want synthetic players, configure avatar WZ part IDs in the player post-processor. The generator renders player frames on the fly through MapleStory.Avatar.
  4. Run dotnet run --project .\MapleStory.MachineLearningSampleGenerator -- --config ".\Examples\sample-generator.yml"
    You can run .\MapleStory.MachineLearningSampleGenerator.exe --help for usage hint, or execute the built binary directly with --config <path>. The config file is the single source of truth for maps, rendering, output, and post-processors.

Example YAML:

mode: character
concurrency: 2

output:
  format: coco
  path: .
  name: sample-generator

render:
  width: 1366
  height: 768

sampling:
  count: 1000
  intervalMs: 0

postProcessors:
  - type: player
    count: 3
    actions: [stand1, walk1, jump]
    emotions: [default]
    avatars:
      - parts: [2000, 12003, 20000, 30000, 1040036, 1060026]
      - parts: [2000, 12003, 20000, 30000, 1040036, 1060026, 1703598]

maps:
  - id: 993134200
  - id: 450007010
    sampling:
      count: 2000
    postProcessors: []

Notes about the YAML format:

Rune mode uses the same render and sampling sections, but requires output.format: coco and does not support postProcessors. Each output image is a center-square crop with four generated rune_arrow annotations and two COCO keypoints per arrow.

Rune mode can also randomly choose maps from all numeric *.img map nodes in the MapleStory data. Explicit entries are always included first; random.count adds that many additional maps and excludes duplicate explicit IDs. Add seed when you need repeatable selection.

Use maps.allMaps: true to sample every numeric *.img map node. Explicit entries are still included first and are not duplicated. allMaps cannot be combined with maps.random.

mode: rune
output:
  format: coco
  path: ./rune-output
  name: rune-sample
render:
  width: 1366
  height: 768
sampling:
  count: 1000
  intervalMs: 0
maps:
  entries:
    - id: 410013660
  random:
    count: 50
    seed: 12345

Note

Output Formats

Tensorflow TFRecord

According to Tensorflow official document, the output .tfrecord contains multiple tf.train.Example in single file. With each example store in the following formats:

uint64 length
uint32 masked_crc32_of_length
byte   data[length]
uint32 masked_crc32_of_data

And

masked_crc = ((crc >> 15) | (crc << 17)) + 0xa282ead8ul

Each tf.train.Example is generated by protobuf-net according to Tensorflow example.proto

Darknet

Output directory structure:

data/
|---obj/
|   |---1.jpg
|   |---1.txt
|   |---......
|---obj.data
|---obj.names
|---test.txt
|---train.txt

obj.data contains

classes=2
train=data/train.txt
valid=data/test.txt
names=data/obj.names
backup = backup/

And obj.names contains the class name for object. test.txt and train.txt contains samples for testing/training with ratio of 5:95 (5% of images in obj/ are used for testing).

COCO

Output directory structure:

coco/
|---train2017/
|   |---1.jpg
|   |---2.jpg
|   |---......
|---val2017/
|   |---1000.jpg
|   |---1001.jpg
|   |---......
|---annotations/
|   |---instances_train2017.json
|   |---instances_val2017.json

The COCO json is defined as following:

{
  "info": {
    "description": "MapleStory 993134100.img Object Detection Samples - Training",
    "url": "https://github.com/charlescao460/MapleStoryDetectionSampleGenerator",
    "version": "1.0",
    "year": 2021,
    "contributor": "CSR"
  },
  "licenses": [
    {
      "url": "https://github.com/charlescao460/MapleStoryDetectionSampleGenerator/blob/main/LICENSE",
      "id": 1,
      "name": "MIT License"
    }
  ],
  "images": [
    {
      "license": 1,
      "file_name": "30a892e1-7f3d-4c65-bdd1-9d28f1ae5187.jpg",
      "coco_url": "",
      "height": 768,
      "width": 1366,
      "flickr_url": "",
      "id": 1
    },
    ...],
  "categories": [
    {
      "supercategory": "element",
      "id": 1,
      "name": "Mob"
    },
    {
      "supercategory": "element",
      "id": 2,
      "name": "Player"
    }
  ],
  "annotations": [
    {
      "segmentation": [
        [
          524,
          429,
          664,
          429,
          664,
          578,
          524,
          578
        ]
      ],
      "area": 20860,
      "iscrowd": 0,
      "image_id": 1,
      "bbox": [
        524,
        429,
        140,
        149
      ],
      "category_id": 1,
      "id": 1
    },
    ...]

Note that segmentation covers the area as the same as bbox does. No segmentation or masked implemented .