annotate problem.py @ 76:0e5ae28e0b2b

Points are now weighted on a test context basis In particular, this has allowed for simple extensions to the format of testconf to award points to whole test groups without at the same time compromising the future ability of giving partial score for correct but slow solutions. Specifically, the groupweight configuration variable has been added and normally has the format {groupindex: points} where groupindex is the group's index in the tests configuration variable. The backwards incompatible change is that test contexts are no longer guaranteed to learn the score awarded or the maximum possible score for every test case and may instead be notified about them in batches. In other news, the pointmap and groupweight configuration variables can (now) be given as sequences in addition to mappings. (Technically, the distinction currently made is dict versus everything else.) Items of a sequence pointmap/groupweight correspond directly to the test cases/ groups defined in the tests configuration variable; in particular, when groups are used, tests=[1],[2,3];pointmap={1:1,2:2,3:3} can now be written as pointmap=tests=[1],[2,3]. Missing items are handled in the same way in which they are handled when the variable is a mapping. Note that the items of groupweight correspond to whole test groups rather than individual test cases. In other news again, the wording of problem total lines has been changed from '<unweighted> points; weighted score: <weighted>' to '<weighted> points (<unweighted> before weighting)', and group total lines now properly report fractional numbers of points (this is a bug fix).
author Oleg Oshmyan <chortos@inbox.lv>
date Sat, 08 Jan 2011 16:03:35 +0200
parents 007f7eb6fb2b
children 69eadc60f4e2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
1 #! /usr/bin/env python
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
2 # Copyright (c) 2010 Chortos-2 <chortos@inbox.lv>
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
3
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
4 from __future__ import division, with_statement
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
5
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
6 try:
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
7 from compat import *
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
8 import config, testcases
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
9 except ImportError:
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
10 import __main__
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
11 __main__.import_error(sys.exc_info()[1])
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
12 else:
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
13 from __main__ import clock, options
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
14
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
15 import os, re, sys
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
16
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
17 try:
75
007f7eb6fb2b The test context stack is now a deque.
Oleg Oshmyan <chortos@inbox.lv>
parents: 74
diff changeset
18 from collections import deque
007f7eb6fb2b The test context stack is now a deque.
Oleg Oshmyan <chortos@inbox.lv>
parents: 74
diff changeset
19 except ImportError:
007f7eb6fb2b The test context stack is now a deque.
Oleg Oshmyan <chortos@inbox.lv>
parents: 74
diff changeset
20 deque = list
007f7eb6fb2b The test context stack is now a deque.
Oleg Oshmyan <chortos@inbox.lv>
parents: 74
diff changeset
21
007f7eb6fb2b The test context stack is now a deque.
Oleg Oshmyan <chortos@inbox.lv>
parents: 74
diff changeset
22 try:
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
23 import signal
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
24 except ImportError:
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
25 signalnames = ()
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
26 else:
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
27 # Construct a cache of all signal names available on the current
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
28 # platform. Prefer names from the UNIX standards over other versions.
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
29 unixnames = frozenset(('HUP', 'INT', 'QUIT', 'ILL', 'ABRT', 'FPE', 'KILL', 'SEGV', 'PIPE', 'ALRM', 'TERM', 'USR1', 'USR2', 'CHLD', 'CONT', 'STOP', 'TSTP', 'TTIN', 'TTOU', 'BUS', 'POLL', 'PROF', 'SYS', 'TRAP', 'URG', 'VTALRM', 'XCPU', 'XFSZ'))
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
30 signalnames = {}
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
31 for name in dir(signal):
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
32 if re.match('SIG[A-Z]+$', name):
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
33 value = signal.__dict__[name]
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
34 if isinstance(value, int) and (value not in signalnames or name[3:] in unixnames):
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
35 signalnames[value] = name
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
36 del unixnames
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
37
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
38 __all__ = 'Problem', 'TestContext', 'test_context_end', 'TestGroup'
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
39
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
40 def strerror(e):
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
41 s = getattr(e, 'strerror')
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
42 if not s: s = str(e)
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
43 return ' (%s%s)' % (s[0].lower(), s[1:]) if s else ''
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
44
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
45 class Cache(object):
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
46 def __init__(self, mydict):
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
47 self.__dict__ = mydict
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
48
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
49 class TestContext(object):
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
50 pass
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
51
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
52 test_context_end = object()
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
53
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
54 class TestGroup(TestContext):
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
55 __slots__ = 'points', 'case', 'log', 'correct', 'allcorrect', 'real', 'max', 'ntotal', 'nvalued', 'ncorrect', 'ncorrectvalued'
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
56
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
57 def __init__(self, points=None):
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
58 self.points = points
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
59 self.real = self.max = self.ntotal = self.nvalued = self.ncorrect = self.ncorrectvalued = 0
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
60 self.allcorrect = True
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
61 self.log = []
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
62
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
63 def case_start(self, case):
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
64 self.case = case
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
65 self.correct = False
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
66 self.ntotal += 1
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
67 if case.points:
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
68 self.nvalued += 1
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
69
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
70 def case_correct(self):
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
71 self.correct = True
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
72 self.ncorrect += 1
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
73 if self.case.points:
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
74 self.ncorrectvalued += 1
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
75
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
76 def case_end(self):
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
77 self.log.append((self.case, self.correct))
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
78 del self.case
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
79 if not self.correct:
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
80 self.allcorrect = False
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
81
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
82 def score(self, real, max):
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
83 self.real += real
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
84 self.max += max
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
85
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
86 def end(self):
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
87 if not self.allcorrect:
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
88 self.real = 0
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
89 if self.points is not None and self.points != self.max:
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
90 max, weighted = self.points, self.real * self.points / self.max if self.max else 0
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
91 before_weighting = ' (%g/%g before weighting)' % (self.real, self.max)
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
92 else:
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
93 max, weighted = self.max, self.real
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
94 before_weighting = ''
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
95 say('Group total: %d/%d tests, %g/%g points%s' % (self.ncorrect, self.ntotal, weighted, max, before_weighting))
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
96 # No real need to flush stdout, as it will anyway be flushed in a moment,
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
97 # when either the problem total or the next test case's ID is printed
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
98 return weighted, max, self.log
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
99
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
100 class Problem(object):
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
101 __slots__ = 'name', 'config', 'cache', 'testcases'
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
102
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
103 def __init__(prob, name):
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
104 if not isinstance(name, basestring):
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
105 # This shouldn't happen, of course
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
106 raise TypeError('Problem() argument 1 must be string, not ' + type(name).__name__)
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
107 prob.name = name
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
108 prob.config = config.load_problem(name)
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
109 prob.cache = Cache({'padoutput': 0})
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
110 prob.testcases = testcases.load_problem(prob)
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
111
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
112 # TODO
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
113 def build(prob):
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
114 raise NotImplementedError
16
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
115
f2279b7602d3 Initial 2.00 commit
Oleg Oshmyan <chortos@inbox.lv>
parents:
diff changeset
116 def test(prob):
23
c1f52b5d80d6 Compatibility and bug fixes
Oleg Oshmyan <chortos@inbox.lv>
parents: 22
diff changeset
117 case = None
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
118 try:
75
007f7eb6fb2b The test context stack is now a deque.
Oleg Oshmyan <chortos@inbox.lv>
parents: 74
diff changeset
119 contexts = deque((TestGroup(),))
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
120 for case in prob.testcases:
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
121 if case is test_context_end:
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
122 real, max, log = contexts.pop().end()
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
123 for case, correct in log:
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
124 contexts[-1].case_start(case)
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
125 if correct:
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
126 contexts[-1].case_correct()
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
127 contexts[-1].case_end()
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
128 contexts[-1].score(real, max)
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
129 continue
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
130 elif isinstance(case, TestContext):
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
131 contexts.append(case)
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
132 continue
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
133 contexts[-1].case_start(case)
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
134 granted = 0
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
135 id = str(case.id)
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
136 if case.isdummy:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
137 id = 'sample ' + id
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
138 say('%*s: ' % (prob.cache.padoutput, id), end='')
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
139 sys.stdout.flush()
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
140 try:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
141 granted = case(lambda: (say('%7.3f%s s, ' % (case.time_stopped - case.time_started, case.time_limit_string), end=''), sys.stdout.flush()))
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
142 except testcases.CanceledByUser:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
143 verdict = 'canceled by the user'
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
144 except testcases.TimeLimitExceeded:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
145 verdict = 'time limit exceeded'
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
146 except testcases.WrongAnswer:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
147 e = sys.exc_info()[1]
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
148 if e.comment:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
149 verdict = 'wrong answer (%s)' % e.comment
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
150 else:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
151 verdict = 'wrong answer'
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
152 except testcases.NonZeroExitCode:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
153 e = sys.exc_info()[1]
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
154 if e.exitcode < 0:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
155 if sys.platform == 'win32':
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
156 verdict = 'terminated with error 0x%X' % (e.exitcode + 0x100000000)
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
157 elif -e.exitcode in signalnames:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
158 verdict = 'terminated by signal %d (%s)' % (-e.exitcode, signalnames[-e.exitcode])
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
159 else:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
160 verdict = 'terminated by signal %d' % -e.exitcode
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
161 else:
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
162 verdict = 'non-zero return code %d' % e.exitcode
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
163 except testcases.CannotStartTestee:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
164 verdict = 'cannot launch the program to test%s' % strerror(sys.exc_info()[1].upstream)
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
165 except testcases.CannotStartValidator:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
166 verdict = 'cannot launch the validator%s' % strerror(sys.exc_info()[1].upstream)
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
167 except testcases.CannotReadOutputFile:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
168 verdict = 'cannot read the output file%s' % strerror(sys.exc_info()[1].upstream)
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
169 except testcases.CannotReadInputFile:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
170 verdict = 'cannot read the input file%s' % strerror(sys.exc_info()[1].upstream)
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
171 except testcases.CannotReadAnswerFile:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
172 verdict = 'cannot read the reference output file%s' % strerror(sys.exc_info()[1].upstream)
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
173 except testcases.TestCaseNotPassed:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
174 verdict = 'unspecified reason [this may be a bug in test.py]%s' % strerror(sys.exc_info()[1])
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
175 #except Exception:
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
176 # verdict = 'unknown error [this may be a bug in test.py]%s' % strerror(sys.exc_info()[1])
21
ec6f1a132109 A pretty usable version
Oleg Oshmyan <chortos@inbox.lv>
parents: 16
diff changeset
177 else:
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
178 try:
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
179 granted, comment = granted
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
180 except TypeError:
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
181 comment = ''
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
182 else:
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
183 if comment:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
184 comment = ' (%s)' % comment
24
c23d81f4a1a3 Score returned by TestCase.__call__() is now normalized to 0..1
Oleg Oshmyan <chortos@inbox.lv>
parents: 23
diff changeset
185 if granted >= 1:
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
186 contexts[-1].case_correct()
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
187 verdict = 'OK' + comment
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
188 elif not granted:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
189 verdict = 'wrong answer' + comment
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
190 else:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
191 verdict = 'partly correct' + comment
26
5bbb68833868 Output text improvements
Oleg Oshmyan <chortos@inbox.lv>
parents: 24
diff changeset
192 granted *= case.points
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
193 say('%g/%g, %s' % (granted, case.points, verdict))
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
194 contexts[-1].case_end()
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
195 contexts[-1].score(granted, case.points)
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
196 weighted = contexts[0].real * prob.config.taskweight / contexts[0].max if contexts[0].max else 0
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
197 before_weighting = valued = ''
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
198 if prob.config.taskweight != contexts[0].max:
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
199 before_weighting = ' (%g/%g before weighting)' % (contexts[0].real, contexts[0].max)
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
200 if contexts[0].nvalued != contexts[0].ntotal:
76
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
201 valued = ' (%d/%d valued)' % (contexts[0].ncorrectvalued, contexts[0].nvalued)
0e5ae28e0b2b Points are now weighted on a test context basis
Oleg Oshmyan <chortos@inbox.lv>
parents: 75
diff changeset
202 say('Problem total: %d/%d tests%s, %g/%g points%s' % (contexts[0].ncorrect, contexts[0].ntotal, valued, weighted, prob.config.taskweight, before_weighting))
39
2b459f9743b4 Test groups are now supported
Oleg Oshmyan <chortos@inbox.lv>
parents: 26
diff changeset
203 sys.stdout.flush()
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
204 return weighted, prob.config.taskweight
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
205 finally:
23
c1f52b5d80d6 Compatibility and bug fixes
Oleg Oshmyan <chortos@inbox.lv>
parents: 22
diff changeset
206 if options.erase and (not prob.config.stdio or case and case.validator):
22
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
207 for var in 'in', 'out':
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
208 name = getattr(prob.config, var + 'name')
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
209 if name:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
210 try:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
211 os.remove(name)
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
212 except Exception:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
213 pass
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
214 if case.validator and not callable(case.validator):
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
215 if prob.config.ansname:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
216 try:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
217 os.remove(prob.config.ansname)
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
218 except Exception:
f07b7a431ea6 Further 2.00 work
Oleg Oshmyan <chortos@inbox.lv>
parents: 21
diff changeset
219 pass