Skip to content

Commit edd92f0

Browse files
committed
Add a section on fixing tests
Fixes #19.
2 parents ce91b74 + 6c5a448 commit edd92f0

File tree

1 file changed

+41
-0
lines changed

1 file changed

+41
-0
lines changed

rules/0120-Tests.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,44 @@ If your number is guaranteed to be an integral number, use `int`, `long`
137137
or your language's equivalent instead of floating point numbers. However, keep
138138
in mind that all values, either input, output, or intermediate should fit in
139139
this type, otherwise you might encounter overflow or underflow issues.
140+
141+
### Fix broken tests early and fix them right
142+
143+
While [#163](https://github.com/Codewars/codewars.com/issues/163) will give you
144+
the tools to fix a kata even late, you should take every reported test issue by
145+
heart. If the tests are failing for some only, check whether your current
146+
example tests contain enough corner cases. Ask them for their code
147+
(in spoilers).
148+
149+
There are several ways your tests can be wrong. For examples, your tests can be
150+
too strict. This is usually the case with floating points, where one expects the
151+
user to do things in a certain way.
152+
153+
However, sometimes your reference solution is wrong. People have know adjusted
154+
their submitted functions to yours, so that they still solve the kata, even
155+
though their logic is broken. If you now fix your reference solution, the
156+
results of many users might get rejected.
157+
158+
So what can you do?
159+
160+
#### Change description to address the bug
161+
You should definitely tell your users if your kata is broken. Especially if your
162+
tests have a defect in only one particular language. It's not the best you can
163+
do, but as long as #163 is still there, it might be the only thing you can do.
164+
165+
#### Reject old solutions, they didn't follow the description
166+
This is fine as long as there haven't been many solutions. If your kata has been
167+
solved by hundreds, it might seem a little bit unfair, but correct tests are
168+
a little bit more important.
169+
170+
#### Experimental solutions
171+
There are also other alternatives:
172+
173+
- check the time: if the kata solution is submitted after a certain point in
174+
time, use the new tests, otherwise the old. This can lead to problems with
175+
the anti-cheat measurements.
176+
- test against both results: this basically duplicates all your tests, which
177+
can be hard to maintain if you do it wrong.
178+
179+
Either way, fix them, otherwise it will lead to submitted issues and therefore
180+
to negative honour.

0 commit comments

Comments
 (0)