Same topics, different languages

Aleks sent along this announcement:

Call for participation

The AAAI-2007 Workshop on Evaluation Methods for Machine Learning II

Description:
============
The purpose of this workshop is to continue the lively and interesting debate started last year at the AAAI 2006 workshop on Evaluation Methods for Machine learning. The previous workshop was successful on the following points:

– It established that the current means of evaluating learning algorithms has some serious drawbacks.
– It established that there are many important properties of algorithms that should be measured, requiring more than a single evaluation metric.
– It established that algorithms must be tested under many different conditions.
– It established that the UCI data sets do not reflect the variety of domains to which algorithms are applied in practice.

For this year’s workshop, we are intending to address, in a more specific fashion, some of the topics that were raised at last year’s workshop and some new ones. Last year’s participants are invited to submit papers reflecting the evolution of their views within the year, or new ideas. Researchers, or practitioners, interested in this issue who did not get a chance to submit a paper last year are invited to do so this year.

Topics:
=======
We invite position papers and technical papers addressing three main topics, and their subtopics:

Evaluation Metrics:
* The efficacy of existing evaluation metrics
* The need for new metrics
* Useful metrics from the other fields
* Single number summaries versus curves
* New performance visualization methods

Statistical Issues:
* Cross-validation vs. bootstrap
* Bias vs. variance
* Parametric vs. non-parametric
* The power of tests
* Sampling methods
* Multiple comparisons

Data Sets:
* The (over) use of community repositories such as UCI
* Concept Drift
* Synthetic, or semi-synthetic, data sets

Format and Attendance:
======================
The workshop will consist of:

Invited talks: We are planning to have several invited speakers. Some, from outside of our research community, will be able to criticize our accepted practices from an external point of view. Some, from inside our community, will discuss how we could improve on our current practices.

Panel Discussion: Our invited speakers will be asked to engage each other on the various issues surrounding the problem of evaluation in Machine Learning at the end of the workshop. The audience will be strongly encouraged to participate in the discussion.

Presentations: The papers accepted to the workshop will be presented throughout the day between the various invited talks. Papers will be grouped by theme, in order to facilitate discussion at the end of each session.

Workshop attendance is open to the public and is estimated at 20-25 attendees. Priority will be given to those active participants in the workshop (paper authors or speakers).

Submission:
===========
Authors are invited to submit papers on the topics outlined above or other related issues. Both position and technical reports will be considered for this workshop. To promote a lively event, with plenty of discussion, the organizers are very interested in papers taking strong positions on the issues listed above. Workshop papers should not exceed 6 pages using the AAAI Style. Submissions should be made electronically in PDF or Postscript format and should be sent (no later than April 1, 2007) by email to William Elazmeh:

[email protected]

Resubmissions and position papers (of at least 1 page) are welcome.

Important Dates:
================
* Submission deadline: April 1, 2007
* Notification date: April 25, 2007
* Camera ready deadline: May 15, 2007.
* The workshop date: July 22 or 23, 2007.
* AAAI-07 conference is to be held on July 22-26, 2007, in Vancouver, British Columbia, Canada.

Organizers:
===========
Chris Drummond
NRC Institute for Information Technology, Canada
Email: [email protected]

William Elazmeh (main contact)
School of Information Technology and Engineering, University of Ottawa,
800 King Edward Ave., room 5-105, Ottawa, Ontario, K1N 6N5 Canada
Telephone: +1 (613) 562 5800 Extension 6699
Fax.: +1 (613) 562 5664
Email: [email protected]

Nathalie Japkowicz
School of Information Technology and Engineering
University of Ottawa, Canada
Email: [email protected]

Sofus A. Macskassy
Fetch Technologies
United States of America
Email: [email protected]

Program Committee:
==================
Nitesh Chawla, University of Notre Dame, USA
Peter Flach, University of Bristol, UK
Alek Kotcz, Microsoft Research, USA
Nicolas Lachiche, Louis Pasteur University, France
Sylvain Letourneau, National Research Council, Canada
Dragos Margineantu, The Boeing Company, Phantom Works, USA
Stan Matwin, University of Ottawa, Canada
Kiri Wagstaff, NASA Jet Propulsion Laboratory, USA
Bianca Zadrozny, Fluminense Federal University, Brazil

Workshop Website:
=================
http://www.site.uottawa.ca/~welazmeh/conferences/AAAI-07/workshop/

1 thought on “Same topics, different languages

Comments are closed.