Weekly update from Gabbarland

[bkowshik]

17th Apr - 23rd Apr, 2017

Datasets for training and testing the model are now on S3.

• Labelled/unlabelled changesets from osmcha.
• Geojson version of changesets from real-changesets.
• User details from osm-comments user api.
• Datasets documentation: https://github.com/mapbox/gabbar/blob/master/docs/data.rst

Workflow

• Model training and testing is done in one Jupyter notebook.
• https://github.com/mapbox/gabbar/blob/master/notebooks/workflow.ipynb

Command line API

• Package is now wired up to take a changeset ID and output predictions.
• python gabbar/scripts/cli.py --changeset 47734592

Model performance metrics

• Performance on both labelled and unlabelled changesets is tracked in metrics.csv
• https://github.com/mapbox/gabbar/blob/master/metrics.csv
• We have a work in progrss PR with a hit rate around 30%

NOTE: This is our very first weekly update! 🎉

---

cc: @anandthakker @geohacker @batpad

[bkowshik]

24th Apr - 30th Apr, 2017

API

• Changesets after 28th April having predictions from Gabbar on S3
• Simple frontend to get predictions for a changeset ID
• https://mapbox.github.io/gabbar/

Aaron from Wikimedia

• Insightful conversation with Aaron from Wikimedia AI
• https://phabricator.wikimedia.org/T163928

Performance

• Workflow to measure performance in a Jupyter notebook
• Real-world performance #29
• Hit rate for harmful changesets at: 21.7 % and not harmful changesets at: 90.0 %

Prediction

• Add changeset ID in prediction
• Have attributes and values used for prediction in the result.

Python 2 and 3 compatibility

• Using protocol 2 in joblib to export trained model and scaler.

[bkowshik]

1st May - 7th May, 2017

Say hello to GradientBoostingClassifier

• Performance gain is looking very promising

Feature importance scores from model training

• scikit has a couple of packages that give out score on feature importance

Code review ML models from wiki-ai

• Created a bunch of tickets on setup revscoring

Talk proposal about Gabbar at The Fifth Elephant, Bengaluru

• Guarding OSM from invalid edits with Gabbar

[bkowshik]

Confusion matrix on validation dataset

	Predicted good	Predicted harmful
Labelled good	46	49
Labelled harmful	3	1

NOTE: Numbers are based on team review from 26th May, 2017 (Fri)

Workflow

• Personally, the best week so far working on Gabbar. 😃
• We completed a complete iteration in one week with the feature level classifier
  1. What is the experiment idea
  2. Collect and clean required datasets
  3. Feature engineering
  4. Model training and fine tuning
  5. Model predictions on testing dataset
  6. Review results, collect learnings and feedback

Feature level classifier

• The Jupyter notebook is looking pretty interesting!
• Detailed feature engineering with both new and old version of features resulted in 150+ attributes per feature
• Add profanity checks to a couple of attributes
• Using user experience of mapper who previously touched feature

Communications

• My first YouTube video: Two minute preview video for The Fifth Elephant conference 🎉

[bkowshik]

NOTE: Sorry to miss out a couple of weekly updates. Putting all of them here.

29th May - 18th June, 2017

Datasets

• Using reverted changesets for model training
• Modified scripts to download labels from new osmcha API
• Regression test suite that Gabbar should 💯 catch
• Increase training size for feature level classifier
• Using results from osm-compare as attributes

Analysis

• Effect of attributes on the feature level classifier
• Reviewing labelled and predicted changesets from feature classifier
• Script to get similar samples based on attributes

Models

• Bag of Tags: An experiment for property based validation
• Feature level classifier is a great idea! 🎉
• Prototyping Gabbar for detection if a highway is good or problematic
• Detect changesets that are very likely to have problems