Chapter 1
Introduction

This report gives the result of running the computer algebra independent integration problems.

The listing of the problems used by this report are

1. MIT_bee_integration_problems.zip
2. handbook_integration_problems.zip
3. CAS_integration_tests_2023_Mathematica_format.m
4. CAS_integration_tests_2023_Maple_and_Mupad_format.zip
5. CAS_integration_tests_2023_SAGE_format.zip
6. CAS_integration_tests_2023_Sympy_format.zip

The Mathematica/Rubi format file above can be read into Mathematica using the following commands

SetDirectory[NotebookDirectory[]] (*where the above .m file was save*) 
lst=First@ReadList["CAS_integration_tests_2023_Mathematica_format.m",Expression]; 
Length[lst]
 

lst[[1]] will be the first integrand,var and lst[[2]] will be the second one and so on.

The Rubi test suite files were downloaded from rulebasedintegration.org.

The current number of problems in this test suite is [85977].

1.1 Listing of CAS systems tested

The following are the CAS systems tested:

1. Mathematica 13.3.1 (August 16, 2023) on windows 10.
2. Rubi 4.16.1 (Dec 19, 2018) on Mathematica 13.3 on windows 10
3. Maple 2023.1 (July, 12, 2023) on windows 10.
4. Maxima 5.47 (June 1, 2023) using Lisp SBCL 2.3.0 on Linux via sagemath 10.1 (Aug 20, 2023).
5. FriCAS 1.3.9 (July 8, 2023) based on sbcl 2.3.0 on Linux via sagemath 10.1 (Aug 20, 2023).
6. Giac/Xcas 1.9.0-57 (June 26, 2023) on Linux via sagemath 10.1 (Aug 20, 2023).
7. Sympy 1.12 (May 10, 2023) Using Python 3.11.3 on Linux.
8. Mupad using Matlab 2021a with Symbolic Math Toolbox Version 8.7 on windows 10.

Maxima and Fricas and Giac are called using Sagemath. This was done using Sagemath integrate command by changing the name of the algorithm to use the different CAS systems.

Sympy was run directly in Python not via sagemath.

1.2 Results

Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions.

If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.

However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.

If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.

The results given in in the table below reflects the above.

Table 1.1: Percentage solved for each CAS

System	solved	Failed
Mathematica	% 98.23 ( 84455 )	% 1.77 ( 1522 )
Rubi	% 94.214 ( 81002 )	% 5.786 ( 4975 )
Maple	% 86.612 ( 74466 )	% 13.388 ( 11511 )
Fricas	% 80.894 ( 69550 )	% 19.106 ( 16427 )
Giac	% 59.024 ( 50747 )	% 40.976 ( 35230 )
Maxima (*)	% 57.273 ( 49242 )	% 42.727 ( 36735 )
Mupad	% 57.082 ( 49077 )	% 42.918 ( 36900 )
Sympy	% 43.557 ( 37449 )	% 56.443 ( 48528 )

The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.

Table 1.2: Description of grading applied to integration result

grade	description
A	Integral was solved and antiderivative is optimal in quality and leaf size.
B	Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size.
C	Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons antiderivative contains a hypergeometric function and the optimal antiderivative does not. antiderivative contains a special function and the optimal antiderivative does not. antiderivative contains the imaginary unit and the optimal antiderivative does not.
F	Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised.

Grading is implemented for all CAS systems in this version except for CAS Mupad where a grade of B is automatically assigned as a place holder for all integrals it completes on time.

The following table summarizes the grading results.

Table 1.3: Antiderivative Grade distribution for each CAS

System	% A grade	% B grade	% C grade	% F grade
Rubi	87.35	1.96	0.76	5.79
Mathematica	75.34	5.81	13.07	1.77
Maple	60.52	13.7	7.49	13.39
Fricas	49.95	18.99	8.45	19.11
Maxima	39.7	11.7	1.69	42.73
Giac	38.63	15.07	1.07	40.98
Sympy	25.87	10.24	3.46	56.44
Mupad	0.	52.15	0.	42.92

The following Bar chart is an illustration of the data in the above table.

The figure below compares the CAS systems for each grade level.

1.3 Time and leaf size Performance

The table below summarizes the performance of each CAS system in terms of time used and leaf size of results.

Mean size is the average leaf size produced by the CAS (before any normalization). The Normalized mean is relative to the mean size of the optimal anti-derivative given in the input files.

For example, if CAS has Normalized mean of \(3\), then the mean size of its leaf size is 3 times as large as the mean size of the optimal leaf size.

Median size is value of leaf size where half the values are larger than this and half are smaller (before any normalization). i.e. The Middle value.

Similarly the Normalized median is relative to the median leaf size of the optimal.

For example, if a CAS has Normalized median of \(1.2\), then its median is \(1.2\) as large as the median leaf size of the optimal.

Table 1.4: Time and leaf size performance for each CAS

System	Mean time (sec)	Mean size	Normalized mean	Median size	Normalized median
Rubi	0.28	154.14	1.2	98.	1.
Maxima	0.44	587.31	4.65	65.	1.08
Fricas	1.18	1161.39	6.29	99.	1.34
Giac	2.03	632.64	4.85	70.	1.14
Mathematica	2.31	311.02	2.	80.	0.99
Sympy	5.36	373.46	4.35	44.	1.1
Maple (*)	5.73	64591.4	753.3	80.	1.
Mupad	6.82	664.71	4.14	63	1.04

1.4 Performance based on number of rules Rubi used

This section shows how each CAS performed based on the number of rules Rubi needed to solve the same integral. One diagram is given for each CAS.

On the \(y\) axis is the percentage solved which Rubi itself needed the number of rules given the \(x\) axis. These plots show that as more rules are needed then most CAS system percentage of solving decreases which indicates the integral is becoming more complicated to solve.

Figure 1.1: Solving statistics per number of Rubi rules used

1.5 Performance based on number of steps Rubi used

This section shows how each CAS performed based on the number of steps Rubi needed to solve the same integral. Note that the number of steps Rubi needed can be much higher than the number of rules, as the same rule could be used more than once.

Figure 1.2: Solving statistics per number of Rubi steps used

The above diagram show that the precentage of solved intergals decreases for most CAS systems as the number of steps increases. As expected, for integrals that required less steps by Rubi, CAS systems had more success which indicates the integral was not as hard to solve. As Rubi needed more steps to solve the integral, the solved percentage decreased for most CAS systems which indicates the integral is becoming harder to solve.

1.6 Solved integrals histogram based on leaf size of result

The following shows the distribution of solved integrals for each CAS system based on leaf size of the antiderivatives produced by each CAS. It shows that most integrals solved produced leaf size less than about 100 to 150. The bin size used is \(40\).

Figure 1.3: Solved integrals based on leaf size distribution

1.7 Solved integrals histogram based on CPU time used

The following shows the distribution of solved integrals for each CAS system based on CPU time used in seconds. The bin size used is \(0.1\) second.

Figure 1.4: Solved integrals histogram based on CPU time used

1.8 Leaf size vs. CPU time used

The following shows the relation between the CPU time used to solve an integral and the leaf size of the antiderivative.

The result for Fricas, Maxima and Giac is shifted more to the right than the other CAS system due to the use of sagemath to call them, which causes an initial slight delay in the timing to start the integration due to overhead of starting a new process each time. This should also be taken into account when looking at the timing of these three CAS systems. Direct calls not using sagemath would result in faster timings, but current implementation uses sagemath as this makes testing much easier to do.

Figure 1.5: Leaf size vs. CPU time. Full range

1.9 Performance per integrand type

The following are the different integrand types the test suite contains.

1. Independent tests.
2. Algebraic Binomial problems (products involving powers of binomials and monomials).
3. Algebraic Trinomial problems (products involving powers of trinomials, binomials and monomials).
4. Miscellaneous Algebraic functions.
5. Exponentials.
6. Logarithms.
7. Trigonometric.
8. Inverse Trigonometric.
9. Hyperbolic functions.
10. Inverse Hyperbolic functions.
11. Special functions.
12. Sam Blake input file.
13. Waldek Hebisch input file.
14. MIT Bee integration.
15. Few problems from Ryzhik and Gradshteyn table of integrals handbook.

The following table gives percentage solved of each CAS per integrand type.

Table 1.5: Percentage solved per integrand type

Integrand type	problems	Rubi	Mathematica	Maple	Maxima	Fricas	Sympy	Giac	Mupad
Independent tests	1892	99.	99.21	94.24	82.14	95.51	75.74	86.73	82.03
Algebraic Binomial	14276	99.99	99.8	85.41	60.16	82.4	63.43	65.63	60.65
Algebraic Trinomial	10187	99.99	99.36	90.4	51.7	87.47	42.51	72.99	56.66
Algebraic Miscellaneous	1519	99.41	98.95	87.76	50.49	85.19	49.9	62.21	61.88
Exponentials	961	99.79	97.29	82.52	66.7	90.84	47.97	49.84	71.49
Logarithms	3085	99.81	97.34	66.52	53.97	58.12	35.43	47.23	43.18
Trigonometric	22551	99.91	97.88	85.98	48.32	79.52	16.36	46.65	49.38
Inverse Trigonometric	4585	99.96	98.28	83.99	35.66	50.36	37.4	43.1	38.43
Hyperbolic	5166	100.	98.84	83.1	62.04	91.17	24.56	62.45	54.72
Inverse Hyperbolic	6626	99.97	98.43	81.09	46.54	63.57	26.32	36.88	39.6
Special functions	999	100.	95.6	71.67	47.85	71.27	46.75	31.23	40.14
Sam Blake file	3154	64.84	94.23	83.51	40.27	73.72	35.67	41.38	47.59
Waldek Hebisch file	10335	63.43	96.75	99.22	93.2	99.76	94.75	87.74	90.1
MIT Bee integration	321	94.08	99.38	95.33	92.52	96.26	82.55	91.9	90.03
Table of integrals	163	100.	100.	97.55	92.64	100.	90.8	100.	92.64

In addition to the above table, for each type of integrand listed above, 3D chart is made which shows how each CAS performed on that specific integrand type.

These charts and the table above can be used to show where each CAS relative strength or weakness in the area of integration.

1.10 Maximum leaf size ratio for each CAS against the optimal result

The following table gives the largest ratio found in each test file, between each CAS antiderivative and the optimal antiderivative.

For each test input file, the problem with the largest ratio \(\frac {\text {CAS leaf size}}{\text {Optimal leaf size}}\) is recorded with the corresponding problem number.

In each column in the table below, the first number is the maximum leaf size ratio, and the number that follows inside the parentheses is the problem number in that specific file where this maximum ratio was found. This ratio is determined only when CAS solved the the problem and also when an optimal antiderivative is known.

If it happens that a CAS was not able to solve all the integrals in the input test file, or if it was not possible to obtain leaf size for the CAS result for all the problems in the file, then a zero is used for the ratio and -1 is used for the problem number.

This makes it easier to locate the problem. In the future, a direct link will be added as well.

#	Rubi	Mathematica	Maple	Maxima	FriCAS	Sympy	Giac	Mupad
1	1. (1)	3.9 (50)	4.5 (170)	3.8 (169)	4. (45)	4789.3 (145)	4.2 (164)	0. (-1)
2	7.3 (21)	5. (26)	3.6 (17)	113.1 (21)	14.3 (13)	16.8 (5)	4.6 (2)	0. (-1)
3	1. (1)	2. (7)	2. (6)	11.1 (7)	2. (8)	1.9 (5)	1.9 (5)	0. (-1)
4	6.4 (5)	14.3 (13)	11.7 (8)	29.7 (8)	5.5 (43)	4.8 (40)	5.3 (1)	0. (-1)
5	1. (1)	54.7 (278)	11.9 (280)	8.1 (280)	7.7 (280)	39.8 (123)	19.5 (141)	0. (-1)
6	1. (1)	1.4 (3)	2.2 (4)	1.9 (1)	1.4 (7)	0.8 (4)	2.3 (5)	0. (-1)
7	2.2 (3)	5.6 (7)	1.8 (3)	2.8 (3)	6.7 (9)	45.4 (9)	1.9 (3)	0. (-1)
8	1.6 (50)	5.3 (31)	4.5 (57)	6.5 (11)	5. (42)	26.4 (71)	5.8 (40)	0. (-1)
9	1.2 (365)	6.8 (316)	3.5 (323)	12.1 (328)	4.2 (341)	4789.3 (251)	15. (328)	0. (-1)
10	3.2 (335)	10.9 (446)	367.9 (417)	36.9 (399)	93.4 (137)	124.9 (217)	18.8 (537)	0. (-1)
11	7.5 (82)	2.8 (24)	24.7 (55)	2.7 (2)	14.9 (77)	43. (17)	6.6 (50)	0. (-1)
12	1.8 (6)	2.3 (4)	1.2 (8)	1.5 (2)	3.3 (3)	3.4 (3)	1.6 (2)	0. (-1)
13	7.1 (369)	23.8 (1323)	27.7 (1323)	32.9 (1323)	32.9 (1323)	136.1 (671)	34. (1323)	0. (-1)
14	2. (870)	16.5 (1101)	22. (1101)	22.2 (1716)	21.8 (1101)	185.4 (2046)	46.7 (827)	0. (-1)
15	3.3 (97)	9.6 (110)	12.1 (100)	2.8 (119)	10.8 (21)	49.2 (119)	9.7 (119)	0. (-1)
16	1. (1)	1.3 (24)	8.6 (25)	4. (25)	9. (25)	143.6 (25)	19.9 (25)	0. (-1)
17	2.6 (35)	3.1 (47)	23.3 (53)	1.7 (35)	9.6 (59)	5.1 (17)	37. (53)	0. (-1)
18	1. (3)	28. (31)	3.7 (9)	0. (-1)	2.4 (3)	0. (-1)	0. (-1)	0. (-1)
19	8.2 (664)	6.9 (663)	5.3 (196)	10. (196)	10. (196)	55.3 (528)	8. (434)	0. (-1)
20	1.6 (254)	8.5 (51)	16.3 (149)	4.4 (73)	16.3 (159)	10.2 (24)	5.9 (69)	0. (-1)
21	1. (655)	17.9 (740)	8.3 (1016)	3.1 (313)	15.1 (781)	33.1 (324)	8.6 (553)	0. (-1)
22	1.3 (64)	2.6 (75)	22.5 (55)	1.1 (18)	10.7 (62)	3. (21)	2.9 (98)	0. (-1)
23	1. (1)	1.1 (50)	10.4 (15)	2. (15)	7. (15)	53. (15)	13.8 (15)	0. (-1)
24	1.2 (173)	1.9 (45)	2.2 (48)	3.6 (161)	5.2 (26)	51.1 (57)	3.9 (157)	0. (-1)
25	8.4 (2686)	13.4 (2913)	13.7 (2591)	13.2 (2285)	3754.9 (1276)	146.8 (2266)	28.4 (2813)	0. (-1)
26	1.4 (342)	48.2 (336)	7.2 (321)	4. (40)	15.1 (265)	35.2 (379)	6.1 (292)	0. (-1)
27	1.3 (816)	13.6 (1007)	15.7 (498)	29.1 (1063)	41.3 (457)	36.6 (124)	9.8 (1052)	0. (-1)
28	1.2 (46)	0.9 (45)	51. (15)	2.5 (15)	28.4 (15)	890.6 (15)	349.3 (15)	0. (-1)
29	1.2 (552)	3.8 (45)	5. (20)	10. (43)	4003.6 (171)	16. (577)	8.1 (591)	0. (-1)
30	1.3 (278)	10. (328)	12.5 (331)	11.2 (348)	16.3 (197)	23.8 (348)	1757.6 (348)	0. (-1)
31	1. (1)	6.4 (283)	4.9 (269)	3.2 (114)	4. (269)	21.6 (269)	6.3 (269)	0. (-1)
32	2.8 (83)	6.2 (127)	2.5 (74)	2.2 (83)	7.2 (127)	16.4 (63)	6.9 (17)	0. (-1)
33	2. (2419)	13.8 (1256)	96.2 (557)	18.9 (1089)	67.3 (2300)	239.3 (2549)	45. (2354)	0. (-1)
34	1.3 (1471)	7.4 (659)	18.8 (2223)	50.9 (2170)	61.6 (1627)	1537.6 (1013)	136.1 (1464)	0. (-1)
35	2.1 (833)	36.3 (915)	116.1 (801)	6. (579)	68.7 (852)	141.7 (925)	41.5 (751)	0. (-1)
36	1. (1)	10.3 (4)	24.3 (108)	2.7 (95)	30.2 (112)	10.5 (104)	92.3 (6)	0. (-1)
37	1. (129)	9.5 (35)	14196.2 (12)	6.6 (27)	66. (74)	8.6 (14)	122.1 (25)	0. (-1)
38	1.8 (76)	17.3 (259)	102.6 (278)	89. (278)	89. (278)	232.5 (367)	123.4 (278)	0. (-1)
39	1.7 (636)	8.8 (109)	7. (109)	5.4 (515)	36.3 (1087)	27.8 (1105)	13.7 (885)	0. (-1)
40	1.7 (212)	9.5 (88)	5. (408)	6.5 (88)	63. (268)	55.3 (285)	60.5 (275)	0. (-1)
41	1.9 (327)	30.1 (394)	8.5 (55)	5.6 (70)	71.8 (309)	80.3 (55)	35.3 (309)	0. (-1)
42	1. (59)	3.6 (103)	1.7 (103)	1.4 (111)	8719.8 (22)	43. (11)	26.2 (41)	0. (-1)
43	1.6 (135)	1.8 (51)	13.8 (37)	1.6 (131)	4782.7 (22)	119.4 (37)	20.4 (60)	0. (-1)
44	1.9 (1)	2.4 (22)	6.4 (29)	0. (-1)	4.2 (35)	0.8 (1)	2.3 (42)	0. (-1)
45	1. (1)	4.9 (4)	0.9 (4)	0. (-1)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
46	2.1 (154)	20. (601)	20.7 (596)	6.3 (609)	46.7 (637)	376.3 (596)	141. (596)	0. (-1)
47	1. (1)	25.5 (83)	1.1 (35)	1.8 (68)	29.9 (41)	42.2 (68)	15.3 (37)	0. (-1)
48	1. (67)	25.1 (143)	88.8 (96)	88.7 (96)	77.1 (93)	82.9 (93)	73.6 (96)	0. (-1)
49	1. (1)	11. (17)	3.1 (16)	2.1 (16)	2.2 (16)	3.2 (11)	3.3 (16)	0. (-1)
50	1. (1)	1.7 (99)	4. (72)	1.1 (72)	9.5 (102)	18.1 (72)	12.1 (79)	0. (-1)
51	6.2 (424)	11.6 (162)	23.1 (194)	42.3 (63)	15310.8 (134)	90.8 (255)	66. (122)	0. (-1)
52	4.1 (1017)	21. (871)	146.6 (202)	5.1 (612)	40.5 (871)	163.8 (182)	41.7 (717)	0. (-1)
53	1. (1)	1.2 (82)	3.1 (64)	2.2 (2)	2. (81)	2.5 (2)	62.4 (2)	0. (-1)
54	1. (1)	1.6 (4)	16. (46)	2.5 (46)	4.6 (58)	3.2 (25)	37.7 (39)	0. (-1)
55	1.2 (655)	6. (648)	39. (267)	125.2 (267)	28.5 (292)	11.9 (563)	53.7 (563)	0. (-1)
56	1. (1)	1.3 (133)	10.6 (99)	4.9 (149)	5. (150)	11. (150)	10.2 (81)	0. (-1)
57	1.5 (120)	3.9 (363)	35.1 (440)	5.1 (348)	21.1 (440)	29.5 (318)	11. (392)	0. (-1)
58	1.5 (176)	12.4 (64)	94.2 (87)	2.9 (166)	10.1 (237)	2.4 (246)	45.6 (187)	0. (-1)
59	1.6 (178)	33.5 (308)	32.9 (156)	7.4 (10)	7.3 (171)	14.2 (7)	56. (66)	0. (-1)
60	1. (1)	16.3 (81)	12.5 (213)	79.5 (81)	9.2 (212)	15.3 (114)	17.7 (108)	0. (-1)
61	1.4 (39)	51.5 (68)	5.7 (13)	14.2 (44)	4.5 (15)	4.3 (27)	12.4 (34)	0. (-1)
62	1.2 (367)	9.6 (341)	31.3 (88)	9.1 (340)	8.8 (404)	38.1 (427)	35. (456)	0. (-1)
63	1.5 (390)	38.5 (131)	13.2 (148)	7.4 (390)	33.9 (197)	34.4 (183)	13.8 (45)	0. (-1)
64	1.2 (284)	13.1 (44)	39.2 (172)	10.6 (23)	11.2 (91)	15.9 (189)	15.3 (28)	0. (-1)
65	1. (1)	5. (360)	20.1 (387)	3.9 (111)	4.3 (414)	137.4 (62)	5.7 (105)	0. (-1)
66	1. (1)	12.7 (250)	10.3 (343)	21.4 (209)	13.1 (209)	29.5 (193)	633.3 (22)	0. (-1)
67	1. (1)	3.3 (8)	4.3 (51)	2.4 (21)	5.9 (53)	17.1 (49)	2.4 (5)	0. (-1)
68	1. (1)	1.8 (113)	3.4 (27)	21.3 (45)	1.7 (38)	2.2 (12)	69.6 (38)	0. (-1)
69	1. (1)	3.3 (203)	7.8 (201)	168.3 (37)	4.7 (44)	23.3 (45)	14. (217)	0. (-1)
70	2. (615)	47.7 (617)	9.1 (606)	9. (151)	24.4 (509)	68.3 (344)	248.2 (367)	0. (-1)
71	1. (1)	1.1 (10)	1.4 (29)	8.1 (33)	1.1 (10)	3.9 (12)	1.6 (13)	0. (-1)
72	1.6 (103)	35. (118)	3.1 (17)	4. (53)	7. (201)	2.6 (40)	657.3 (36)	0. (-1)
73	1.9 (621)	315.7 (767)	2501.4 (795)	30.7 (256)	17. (711)	83.8 (470)	32.7 (633)	0. (-1)
74	1.6 (1108)	24.9 (1562)	56.7 (174)	8.6 (46)	14.7 (937)	223.9 (697)	424.5 (175)	0. (-1)
75	1.3 (12)	90.6 (42)	619.3 (48)	7.2 (16)	28.8 (35)	3.4 (1)	2.4 (6)	0. (-1)
76	1.2 (206)	9.3 (100)	1717.5 (353)	35.1 (48)	16.5 (327)	66.6 (265)	290.5 (220)	0. (-1)
77	1. (1)	10.5 (13)	2.7 (2)	12.4 (1)	2.3 (2)	412.4 (8)	21.3 (1)	0. (-1)
78	1.4 (32)	2.4 (18)	4.3 (16)	3.3 (20)	2.2 (18)	2.3 (32)	1.3 (16)	0. (-1)
79	1.8 (236)	11.6 (574)	3987.5 (593)	17.6 (487)	3020.2 (254)	9937.7 (81)	544.2 (365)	0. (-1)
80	1. (1)	2.2 (2)	1.3 (6)	1.3 (2)	4.6 (1)	11.7 (4)	2.3 (2)	0. (-1)
81	1. (1)	1.5 (8)	1.3 (13)	1. (19)	51.7 (13)	2.8 (11)	1.8 (14)	0. (-1)
82	1. (1)	3.7 (284)	8.3 (12)	16.5 (170)	4.1 (42)	2.5 (64)	1015. (141)	0. (-1)
83	1. (1)	3.5 (187)	8.3 (76)	12.1 (133)	6.9 (33)	4.1 (9)	627.4 (22)	0. (-1)
84	1. (1)	2.4 (61)	3.4 (50)	788.2 (7)	6.2 (52)	6. (41)	2. (5)	0. (-1)
85	1. (1)	1.3 (94)	4.2 (26)	4.2 (86)	1.4 (87)	6. (61)	4.3 (35)	0. (-1)
86	4.3 (11)	4.1 (60)	5.5 (80)	3.2 (3)	4.2 (32)	35.5 (25)	3.7 (11)	0. (-1)
87	1. (1)	1. (10)	1.4 (29)	8.1 (32)	1.1 (10)	3.8 (12)	1.6 (13)	0. (-1)
88	1. (1)	3.2 (1)	4.3 (20)	4.1 (3)	4.1 (20)	0. (-1)	3. (3)	0. (-1)
89	1.4 (370)	34.4 (773)	11.4 (763)	6489.4 (123)	7.2 (484)	28.7 (464)	609.3 (219)	0. (-1)
90	1. (1)	2.8 (2)	2.4 (2)	0. (-1)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
91	1. (1)	3. (1)	1.3 (1)	3.5 (1)	1.7 (1)	0. (-1)	1.5 (1)	0. (-1)
92	1.1 (40)	36.7 (454)	15.7 (436)	7944.2 (100)	8.1 (279)	77.7 (251)	934. (188)	0. (-1)
93	1. (1)	53.3 (393)	10.3 (31)	20.3 (115)	3.4 (319)	9. (35)	1080.9 (91)	0. (-1)
94	1.4 (940)	41.8 (1158)	18.9 (1155)	7808.5 (402)	7.5 (1007)	74.1 (564)	1093.2 (490)	0. (-1)
95	1.2 (81)	2.6 (6)	5.9 (69)	9.4 (53)	3020.2 (79)	1914.1 (31)	3.7 (91)	0. (-1)
96	1. (1)	2.1 (9)	2. (17)	1. (2)	15.2 (14)	13.9 (4)	7. (4)	0. (-1)
97	1. (1)	1.9 (5)	1.4 (20)	0.8 (11)	51.4 (8)	3.2 (12)	66.7 (8)	0. (-1)
98	1. (1)	16.8 (373)	186.8 (52)	1.3 (7)	5. (317)	3. (376)	24.8 (8)	0. (-1)
99	1. (1)	4.5 (44)	9.4 (61)	11.2 (49)	4.9 (54)	2.5 (24)	6. (22)	0. (-1)
100	1. (1)	2.5 (44)	1.1 (21)	7.9 (52)	4.5 (39)	16.9 (21)	1.7 (21)	0. (-1)
101	1.5 (562)	26.3 (638)	146.8 (620)	19. (393)	8.6 (80)	40. (172)	718. (543)	0. (-1)
102	1. (1)	7. (46)	1.8 (42)	2.9 (67)	7.5 (75)	1.3 (2)	324.8 (32)	0. (-1)
103	1.4 (891)	45.1 (676)	10357.7 (510)	141. (1121)	226.9 (1297)	68.3 (1213)	24.2 (1203)	0. (-1)
104	1. (1)	3. (178)	15275. (454)	144. (373)	197.8 (463)	42.4 (280)	23. (257)	0. (-1)
105	1. (130)	26.8 (109)	123.6 (126)	3. (83)	541.4 (138)	145.8 (74)	35.4 (64)	0. (-1)
106	1. (1)	44.6 (159)	1800.7 (350)	18.1 (272)	20.7 (379)	62.9 (245)	776.9 (30)	0. (-1)
107	1. (1)	3.4 (21)	31766. (14)	0. (-1)	104.6 (24)	0. (-1)	0. (-1)	0. (-1)
108	1. (1)	9.9 (48)	1.6 (2)	1.3 (4)	4.9 (20)	2.6 (1)	4.2 (3)	0. (-1)
109	1. (1)	5.6 (42)	9.4 (59)	18.7 (47)	4.9 (59)	2.4 (22)	33.1 (8)	0. (-1)
110	1. (1)	2.5 (11)	2.1 (9)	3.3 (11)	4. (7)	1.3 (2)	2.5 (7)	0. (-1)
111	1. (1)	2.4 (5)	2.2 (7)	4.3 (7)	3.3 (7)	1.2 (2)	2.7 (6)	0. (-1)
112	1. (1)	5.1 (20)	26.7 (102)	1.9 (94)	40.1 (103)	35.7 (93)	2.4 (94)	0. (-1)
113	1. (1)	13.4 (37)	8. (30)	22.4 (42)	13. (57)	59.8 (7)	17.1 (37)	0. (-1)
114	1. (1)	3.8 (22)	31765.8 (3)	0. (-1)	98.2 (12)	0. (-1)	0. (-1)	0. (-1)
115	1. (1)	4.7 (42)	13.1 (269)	25.9 (47)	5.7 (42)	3.3 (1)	11.7 (42)	0. (-1)
116	1. (1)	13.2 (39)	4.1 (29)	14.9 (16)	5.1 (6)	0. (-1)	5.7 (18)	0. (-1)
117	1. (1)	3.2 (18)	5.6 (73)	120.4 (20)	4.5 (68)	2.2 (53)	4.2 (20)	0. (-1)
118	1.4 (423)	251.5 (874)	17.6 (578)	1460.2 (263)	7. (515)	2.6 (5)	5.8 (513)	0. (-1)
119	1. (1)	45.2 (153)	6.5 (290)	2.9 (65)	5.8 (227)	0. (-1)	7. (196)	0. (-1)
120	1.7 (340)	19.3 (232)	27. (339)	3.7 (67)	34.3 (339)	13.1 (90)	7.1 (286)	0. (-1)
121	1.3 (115)	104.2 (207)	265.8 (153)	37.5 (109)	8.8 (159)	0. (-1)	5.1 (197)	0. (-1)
122	2.2 (197)	227.9 (265)	7. (238)	43.2 (130)	15.3 (263)	3. (170)	4.3 (256)	0. (-1)
123	1.3 (265)	462.7 (634)	19.2 (391)	1629.1 (267)	8.2 (336)	2.2 (47)	6.9 (335)	0. (-1)
124	1. (1)	5.1 (25)	11.2 (19)	13.5 (25)	2.6 (58)	2.9 (33)	2.7 (41)	0. (-1)
125	1.2 (870)	582.4 (1351)	24. (971)	2062.5 (624)	7.8 (923)	3. (930)	7.5 (922)	0. (-1)
126	1. (1)	66.8 (138)	367.6 (432)	58. (256)	27.3 (461)	6. (459)	15. (389)	0. (-1)
127	1. (1)	5.6 (42)	6.9 (21)	33.4 (39)	3.8 (42)	3.1 (1)	3.1 (41)	0. (-1)
128	1. (1)	4.2 (25)	5. (83)	39.4 (15)	4.6 (69)	2.3 (53)	2.9 (61)	0. (-1)
129	1. (1)	5.3 (36)	11. (18)	6.4 (13)	7.8 (20)	0. (-1)	13.6 (15)	0. (-1)
130	1. (1)	2.5 (8)	2. (9)	4.9 (8)	3.7 (14)	0. (-1)	2.2 (8)	0. (-1)
131	1.3 (20)	3.3 (10)	1.9 (5)	3.5 (1)	5. (22)	0. (-1)	2.2 (10)	0. (-1)
132	1. (1)	2.7 (3)	2.1 (8)	2.5 (8)	2.3 (9)	4.9 (18)	3.3 (12)	0. (-1)
133	1. (1)	1.2 (1)	1.8 (1)	0. (-1)	1.4 (1)	0. (-1)	0. (-1)	0. (-1)
134	1. (12)	3.1 (18)	26.9 (15)	39.7 (11)	16.6 (11)	0. (-1)	24.5 (11)	0. (-1)
135	1. (1)	29.1 (187)	5281024.1 (170)	85. (57)	7.2 (231)	6948.3 (39)	558.9 (93)	0. (-1)
136	3.3 (23)	25.2 (272)	4.6 (211)	9.5 (209)	8.5 (143)	35.6 (18)	406.5 (236)	0. (-1)
137	1.1 (281)	9.5 (164)	9.4 (331)	58.8 (171)	13.9 (273)	10.3 (396)	6002.2 (153)	0. (-1)
138	1. (1)	2.7 (1)	6.9 (9)	0.4 (5)	12. (4)	1.1 (5)	0.7 (5)	0. (-1)
139	4.3 (259)	7.9 (276)	8.4 (1)	90.9 (225)	4. (236)	5.6 (18)	2097.9 (70)	0. (-1)
140	19.2 (34)	9.1 (133)	1.8 (70)	81.3 (34)	4.2 (63)	16.2 (43)	264.4 (31)	0. (-1)
141	10.8 (759)	718.9 (434)	638.5 (860)	418.5 (198)	27.7 (503)	4789.3 (480)	5737. (605)	0. (-1)
142	1.4 (107)	2.5 (95)	4.3 (156)	1.7 (155)	1.8 (7)	2.3 (11)	9.9 (145)	0. (-1)
143	1. (237)	3.9 (33)	19.9 (90)	3.3 (195)	6. (642)	2.9 (413)	56.7 (620)	0. (-1)
144	1.9 (147)	11.3 (469)	14. (55)	12.1 (177)	9.2 (104)	8.1 (206)	15.5 (255)	0. (-1)
145	1. (1)	4.9 (41)	2.6 (156)	1.8 (155)	3. (7)	2.3 (11)	26.4 (147)	0. (-1)
146	1. (1)	2. (5)	2.8 (5)	2.4 (11)	5.1 (33)	2. (23)	36.3 (23)	0. (-1)
147	1. (1)	11. (114)	3.3 (18)	2.4 (24)	5.7 (29)	2. (58)	4.6 (80)	0. (-1)
148	1. (1)	4. (83)	114.6 (118)	1.5 (165)	2.3 (105)	7.5 (105)	1.9 (134)	0. (-1)
149	1. (1)	3.6 (25)	43. (20)	1.8 (8)	7555.1 (24)	44.7 (8)	1.2 (21)	0. (-1)
150	1.3 (152)	6.4 (429)	80.4 (146)	4.8 (218)	9.9 (1223)	4.3 (197)	2.4 (1279)	0. (-1)
151	1. (1)	3.3 (36)	24.5 (37)	26.3 (61)	3. (30)	9.5 (12)	1. (27)	0. (-1)
152	1.9 (344)	2.7 (248)	12.5 (56)	9.5 (180)	10. (375)	11.6 (375)	7.9 (375)	0. (-1)
153	1.1 (117)	11.4 (54)	27.1 (147)	5.4 (67)	5.6 (50)	13.1 (131)	5.8 (125)	0. (-1)
154	1.3 (109)	11.4 (164)	28.2 (31)	13.3 (107)	6.8 (64)	5.9 (106)	27.1 (135)	0. (-1)
155	1. (1)	1.2 (7)	1. (2)	1. (2)	1.1 (5)	2.7 (4)	1.1 (2)	0. (-1)
156	1.2 (68)	2.7 (107)	6. (105)	3.4 (31)	8.7 (151)	2.5 (12)	84.8 (69)	0. (-1)
157	1. (1)	12.7 (22)	2.9 (26)	1.7 (14)	4. (24)	2.7 (8)	2.6 (2)	0. (-1)
158	1.4 (51)	3. (114)	6. (112)	1.9 (22)	8.7 (156)	2.6 (12)	27.3 (91)	0. (-1)
159	1. (1)	13. (21)	3.3 (26)	1.6 (13)	4. (23)	2.7 (8)	3.5 (26)	0. (-1)
160	1. (1)	10.9 (193)	7.5 (379)	3.7 (327)	23.9 (496)	24.6 (231)	8.7 (6)	0. (-1)
161	1. (1)	2.1 (3)	3.4 (98)	12.9 (90)	6.6 (20)	1.9 (10)	6.9 (29)	0. (-1)
162	1. (1)	1.5 (24)	1.8 (28)	6. (7)	5.4 (21)	0. (-1)	1.1 (7)	0. (-1)
163	1. (271)	8.7 (365)	7.3 (126)	21.3 (134)	32.5 (87)	26.4 (102)	25.9 (273)	0. (-1)
164	1.3 (16)	9.9 (394)	4.6 (90)	21.9 (315)	2699.3 (269)	9185.8 (34)	23.5 (81)	0. (-1)
165	1. (1)	13.6 (173)	3.4 (15)	3.6 (1)	16.2 (36)	4.1 (8)	8.7 (6)	0. (-1)
166	1. (1)	1.8 (38)	8. (107)	3.5 (5)	4.3 (108)	2.3 (12)	18. (32)	0. (-1)
167	1. (1)	2.1 (3)	3.4 (64)	12.9 (56)	6.6 (20)	1.9 (10)	5.2 (25)	0. (-1)
168	1. (1)	1.5 (12)	1.8 (28)	6. (7)	5.4 (21)	0. (-1)	1.1 (7)	0. (-1)
169	1. (244)	8.7 (328)	6.8 (10)	11.4 (196)	40.1 (177)	53.6 (55)	25.9 (246)	0. (-1)
170	1.3 (60)	2.5 (11)	3.4 (42)	7.4 (13)	2966.3 (69)	1913.7 (26)	16.8 (61)	0. (-1)
171	1. (1)	3.6 (63)	5.5 (68)	3.2 (8)	23.5 (11)	1.7 (8)	3.2 (8)	0. (-1)
172	1.2 (109)	4.1 (73)	4.1 (144)	12.6 (188)	65.9 (200)	98.5 (64)	4.2 (102)	0. (-1)
173	1.3 (257)	10.5 (254)	7.5 (221)	23.3 (190)	178.8 (253)	7.3 (171)	13. (219)	0. (-1)
174	1. (1)	3.7 (47)	5.8 (37)	3.7 (8)	20. (47)	11.7 (27)	3.2 (8)	0. (-1)
175	1. (1)	6.4 (113)	6.2 (35)	13. (193)	66.3 (205)	11.6 (148)	5.9 (29)	0. (-1)
176	1. (1)	10.3 (47)	7.5 (24)	6.3 (10)	177.8 (46)	5.5 (5)	13. (45)	0. (-1)
177	1. (1)	1.9 (1)	2.9 (5)	2.5 (7)	16.2 (9)	0. (-1)	2.7 (7)	0. (-1)
178	1. (1)	2.8 (80)	3.6 (79)	2. (15)	22.6 (82)	1.5 (5)	2. (31)	0. (-1)
179	3.5 (186)	10.9 (185)	11.8 (27)	8.6 (59)	168.4 (146)	2.3 (191)	5.4 (186)	0. (-1)
180	1.4 (54)	14.4 (168)	8.8 (48)	21.9 (158)	170.3 (218)	5. (142)	6.4 (219)	0. (-1)
181	1. (1)	8.6 (26)	4.6 (5)	3.1 (7)	15.7 (9)	0. (-1)	2.8 (7)	0. (-1)
182	1. (1)	4.7 (82)	3.9 (78)	2.3 (15)	28.8 (15)	1.4 (5)	1.9 (5)	0. (-1)
183	3.3 (160)	10. (24)	22.3 (24)	6.3 (24)	33.8 (124)	2.1 (165)	9. (24)	0. (-1)
184	1.1 (12)	3.4 (24)	5.6 (18)	6.2 (1)	96.8 (15)	0. (-1)	8.8 (22)	0. (-1)
185	1.9 (192)	515.8 (777)	140.9 (767)	26. (100)	105. (745)	138.5 (810)	12.2 (11)	0. (-1)
186	1. (1)	1.9 (141)	2.7 (38)	1.4 (15)	3.3 (7)	1. (22)	2.3 (19)	0. (-1)
187	1. (1)	3.5 (230)	9.1 (104)	3.5 (219)	6. (651)	3.6 (255)	2.6 (118)	0. (-1)
188	1. (1)	7.3 (368)	4.9 (357)	12.3 (115)	5.3 (11)	8.2 (147)	8.4 (115)	0. (-1)
189	1. (1)	3.2 (39)	2.3 (18)	1.2 (135)	2. (7)	0. (-1)	2. (19)	0. (-1)
190	1.4 (20)	11.8 (176)	17.2 (93)	2.6 (22)	7. (508)	0. (-1)	2.2 (528)	0. (-1)
191	1.3 (73)	5.7 (49)	15. (291)	9.2 (93)	9.7 (20)	2.8 (278)	6.3 (93)	0. (-1)
192	1. (1)	3. (220)	43.8 (156)	5.3 (202)	11.3 (216)	30.9 (63)	7.5 (1)	0. (-1)
193	1.6 (21)	7.9 (18)	54.2 (20)	2.3 (40)	31.4 (35)	62.6 (8)	10.6 (1)	0. (-1)
194	1.6 (538)	5.6 (156)	71.1 (235)	16.1 (244)	7.1 (516)	4.5 (307)	6.7 (15)	0. (-1)
195	1. (43)	17.8 (42)	31.6 (46)	5.2 (15)	5. (37)	118.9 (37)	15.4 (37)	0. (-1)
196	2. (172)	3.7 (868)	14.5 (220)	18.3 (1152)	12.2 (1368)	31. (997)	9.7 (1368)	0. (-1)
197	1.7 (81)	8.2 (323)	22.2 (318)	4.3 (72)	6.3 (315)	7.8 (276)	7.6 (133)	0. (-1)
198	1.2 (78)	8.3 (112)	1614.5 (159)	3.9 (95)	7.1 (47)	118.9 (108)	15.3 (108)	0. (-1)
199	1.9 (172)	4.9 (430)	8. (54)	3.5 (37)	3.8 (467)	15.5 (446)	5.1 (235)	0. (-1)
200	1. (1)	16.3 (85)	7.9 (124)	1.4 (47)	8.9 (168)	1.5 (35)	0. (-1)	0. (-1)
201	2.8 (38)	25.8 (18)	40.2 (80)	1. (34)	9.3 (6)	2. (38)	3.3 (47)	0. (-1)
202	1.2 (75)	3.2 (114)	5.8 (75)	2.1 (10)	9.7 (156)	1.2 (9)	0. (-1)	0. (-1)
203	1.6 (55)	8.2 (13)	5.2 (66)	2.7 (31)	7.3 (71)	2.3 (54)	3.1 (54)	0. (-1)
204	1. (1)	1.6 (138)	2.1 (190)	1.1 (31)	2. (140)	2.7 (221)	1.6 (18)	0. (-1)
205	1. (1)	2.5 (57)	1.5 (92)	3.3 (136)	2.5 (60)	2. (179)	0. (-1)	0. (-1)
206	1. (1)	1.8 (35)	3.4 (134)	4.4 (88)	5.7 (117)	7.8 (69)	557.1 (66)	0. (-1)
207	1. (1)	1.8 (35)	1.2 (7)	0. (-1)	0. (-1)	9.1 (69)	0. (-1)	0. (-1)
208	1. (1)	1.3 (153)	2.4 (41)	4.1 (155)	2.7 (28)	4.9 (30)	0. (-1)	0. (-1)
209	380.1 (628)	650.2 (1384)	183783.9 (2420)	6.9 (1028)	14782.7 (2646)	20.4 (315)	31. (1760)	0. (-1)
210	643. (6841)	21182. (7738)	892063.8 (10190)	1205. (6099)	145.9 (2571)	363.2 (8409)	39084.8 (5246)	0. (-1)
211	66.8 (46)	527.7 (105)	439.4 (227)	439.4 (227)	21.8 (64)	522.8 (105)	573.4 (258)	0. (-1)
212	2.9 (77)	10. (17)	2.5 (9)	6.5 (16)	10. (23)	23.3 (162)	12.5 (18)	0. (-1)
213	2.9 (10)	1. (1)	10.2 (9)	1.4 (1)	1.2 (8)	0.7 (2)	0. (-1)	0. (-1)

1.11 Pass/Fail per test file for each CAS system

The following table gives the number of passed integrals and number of failed integrals per test number. There are 210 tests. Each tests corresponds to one input file.

#	Rubi		MMA		Maple		Maxima		FriCAS		Sympy		Giac		Mupad
	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail
1	175	0	175	0	173	2	166	9	174	1	165	10	170	5	169	6
2	33	2	34	1	28	7	16	19	25	10	9	26	17	18	9	26
3	13	1	14	0	12	2	8	6	13	1	9	5	10	4	11	3
4	48	2	50	0	33	17	26	24	50	0	19	31	41	9	12	38
5	279	5	284	0	282	2	251	33	281	3	254	30	269	15	270	14
6	3	4	7	0	5	2	3	4	7	0	5	2	5	2	7	0
7	7	2	9	0	9	0	7	2	9	0	5	4	9	0	9	0
8	113	0	113	0	113	0	111	2	112	1	107	6	111	2	106	7
9	376	0	376	0	376	0	374	2	376	0	363	13	375	1	372	4
10	705	0	705	0	656	49	565	140	662	43	460	245	590	115	542	163
11	113	3	102	14	88	28	20	96	90	26	29	87	36	80	37	79
12	8	0	8	0	8	0	7	1	8	0	8	0	8	0	8	0
13	1917	0	1917	0	1565	352	1328	589	1603	314	1225	692	1380	537	1241	676
14	3201	0	3201	0	2870	331	2042	1159	2932	269	1657	1544	2442	759	1884	1317
15	158	1	154	5	128	31	39	120	69	90	30	129	45	114	49	110
16	34	0	34	0	28	6	16	18	28	6	19	15	28	6	4	30
17	78	0	78	0	78	0	27	51	64	14	4	74	48	30	39	39
18	35	0	35	0	35	0	0	35	9	26	0	35	0	35	0	35
19	1071	0	1071	0	785	286	632	439	732	339	1024	47	616	455	695	376
20	349	0	349	0	264	85	79	270	277	72	103	246	106	243	66	283
21	1156	0	1156	0	1048	108	682	474	984	172	631	525	822	334	730	426
22	115	0	114	1	107	8	15	100	68	47	26	89	31	84	27	88
23	51	0	51	0	14	37	14	37	14	37	28	23	14	37	14	37
24	174	0	174	0	170	4	170	4	170	4	154	20	166	8	129	45
25	3078	0	3059	19	2685	393	2196	882	2630	448	2888	190	2051	1027	2228	850
26	385	0	383	2	255	130	166	219	218	167	144	241	130	255	170	215
27	1081	0	1081	0	911	170	391	690	813	268	406	675	571	510	531	550
28	46	0	46	0	12	34	12	34	12	34	23	23	12	34	12	34
29	594	0	594	0	577	17	415	179	538	56	430	164	420	174	449	145
30	454	0	454	0	386	68	153	301	325	129	125	329	261	193	193	261
31	298	0	296	2	275	23	212	86	277	21	138	160	227	71	197	101
32	143	0	143	0	113	30	108	35	113	30	101	42	111	32	132	11
33	2590	0	2580	10	2324	266	1392	1198	2325	265	1120	1470	1980	610	1589	1001
34	2646	0	2646	0	2584	62	1715	931	2563	83	1365	1281	2272	374	1685	961
35	958	0	942	16	728	230	312	646	659	299	265	693	475	483	276	682
36	123	0	123	0	121	2	67	56	111	12	67	56	90	33	53	70
37	143	0	143	0	140	3	15	128	83	60	12	131	53	90	19	124
38	400	0	394	6	357	43	291	109	352	48	189	211	351	49	195	205
39	1126	0	1126	0	1061	65	687	439	1059	67	441	685	822	304	695	431
40	412	1	405	8	399	14	77	336	331	82	187	226	182	231	184	229
41	413	0	406	7	376	37	144	269	329	84	131	282	259	154	218	195
42	111	0	111	0	111	0	83	28	93	18	45	66	106	5	106	5
43	145	0	145	0	143	2	73	72	125	20	80	65	143	2	143	2
44	42	0	40	2	40	2	0	42	26	16	6	36	6	36	1	41
45	4	0	4	0	4	0	0	4	0	4	0	4	0	4	0	4
46	664	0	662	2	496	168	302	362	535	129	254	410	436	228	360	304
47	96	0	92	4	49	47	17	79	49	47	42	54	37	59	49	47
48	156	0	147	9	137	19	66	90	129	27	78	78	110	46	122	34
49	17	0	17	0	3	14	2	15	7	10	2	15	4	13	5	12
50	140	0	139	1	136	4	24	116	135	5	47	93	109	31	72	68
51	491	3	494	0	489	5	408	86	456	38	436	58	426	68	485	9
52	1019	6	1009	16	844	181	359	666	838	187	322	703	519	506	455	570
53	98	0	98	0	78	20	64	34	93	5	38	60	56	42	58	40
54	93	0	85	8	81	12	78	15	93	0	51	42	54	39	53	40
55	768	2	752	18	634	136	499	271	687	83	372	398	369	401	576	194
56	193	0	193	0	121	72	106	87	123	70	79	114	102	91	60	133
57	456	0	449	7	333	123	225	231	280	176	283	173	190	266	146	310
58	249	0	243	6	139	110	68	181	90	159	47	202	58	191	46	203
59	314	0	301	13	224	90	238	76	210	104	114	200	191	123	200	114
60	263	0	249	14	161	102	180	83	156	107	54	209	125	138	127	136
61	106	2	107	1	45	63	68	40	41	67	20	88	36	72	35	73
62	543	4	543	4	344	203	222	325	221	326	166	381	216	331	209	338
63	641	0	604	37	418	223	338	303	393	248	197	444	351	290	326	315
64	314	0	314	0	267	47	220	94	279	35	133	181	188	126	183	131
65	538	0	538	0	446	92	242	296	439	99	102	436	213	325	248	290
66	348	0	347	1	264	84	203	145	314	34	116	232	183	165	143	205
67	72	0	72	0	47	25	32	40	47	25	32	40	39	33	36	36
68	113	0	113	0	113	0	53	60	113	0	26	87	71	42	20	93
69	357	0	349	8	245	112	270	87	305	52	112	245	183	174	129	228
70	653	0	638	15	569	84	287	366	533	120	108	545	314	339	258	395
71	36	0	36	0	34	2	34	2	36	0	20	16	34	2	16	20
72	206	2	201	7	178	30	141	67	178	30	5	203	156	52	154	54
73	837	0	795	42	640	197	217	620	580	257	165	672	482	355	344	493
74	1560	3	1512	51	1416	147	982	581	1295	268	242	1321	1217	346	1131	432
75	51	0	47	4	50	1	16	35	31	20	4	47	20	31	13	38
76	358	0	333	25	296	62	133	225	275	83	102	256	288	70	178	180
77	19	0	15	4	13	6	13	6	13	6	8	11	13	6	13	6
78	34	0	18	16	5	29	7	27	9	25	1	33	3	31	9	25
79	592	2	580	14	523	71	332	262	481	113	78	516	311	283	334	260
80	9	0	9	0	9	0	2	7	9	0	5	4	9	0	9	0
81	19	0	19	0	19	0	5	14	18	1	6	13	9	10	19	0
82	294	0	294	0	196	98	92	202	197	97	18	276	39	255	80	214
83	189	0	189	0	135	54	140	49	137	52	55	134	112	77	74	115
84	62	0	62	0	45	17	39	23	45	17	32	30	39	23	35	27
85	99	0	99	0	87	12	69	30	91	8	34	65	52	47	30	69
86	88	0	88	0	88	0	27	61	57	31	23	65	32	56	34	54
87	34	0	34	0	32	2	32	2	34	0	18	16	32	2	15	19
88	22	0	22	0	22	0	17	5	21	1	1	21	21	1	18	4
89	932	0	929	3	854	78	325	607	675	257	101	831	281	651	310	622
90	4	0	4	0	4	0	0	4	0	4	0	4	0	4	0	4
91	1	0	1	0	1	0	1	0	1	0	0	1	1	0	1	0
92	644	0	635	9	629	15	209	435	470	174	69	575	210	434	231	413
93	393	0	389	4	238	155	119	274	238	155	15	378	26	367	75	318
94	1541	0	1534	7	1508	33	499	1042	1161	380	126	1415	540	1001	629	912
95	98	0	98	0	96	2	70	28	85	13	19	79	76	22	67	31
96	21	0	21	0	21	0	2	19	18	3	6	15	19	2	19	2
97	20	0	20	0	20	0	4	16	19	1	5	15	20	0	20	0
98	387	0	387	0	267	120	137	250	241	146	18	369	82	305	122	265
99	62	1	63	0	58	5	49	14	63	0	28	35	35	28	32	31
100	66	0	66	0	36	30	61	5	48	18	35	31	36	30	38	28
101	700	0	700	0	571	129	405	295	573	127	124	576	258	442	369	331
102	91	0	90	1	83	8	79	12	83	8	8	83	82	9	83	8
103	1328	0	1253	75	1103	225	575	753	1164	164	295	1033	457	871	835	493
104	855	0	819	36	783	72	428	427	783	72	209	646	271	584	528	327
105	171	0	169	2	122	49	84	87	113	58	63	108	84	87	103	68
106	499	0	497	2	412	87	269	230	414	85	99	400	280	219	283	216
107	51	0	51	0	33	18	0	51	42	9	0	51	0	51	0	51
108	52	0	52	0	37	15	37	15	37	15	8	44	15	37	26	26
109	61	0	61	0	58	3	49	12	61	0	28	33	35	26	28	33
110	23	0	23	0	23	0	19	4	23	0	6	17	23	0	23	0
111	19	0	19	0	19	0	15	4	19	0	4	15	19	0	19	0
112	106	0	105	1	103	3	3	103	103	3	2	104	3	103	103	3
113	64	0	64	0	63	1	23	41	64	0	11	53	54	10	39	25
114	32	0	32	0	20	12	0	32	32	0	0	32	0	32	0	32
115	299	0	299	0	227	72	93	206	218	81	29	270	39	260	78	221
116	46	0	46	0	42	4	36	10	46	0	20	26	24	22	24	22
117	83	0	79	4	51	32	48	35	63	20	37	46	43	40	47	36
118	879	0	867	12	735	144	323	556	609	270	51	828	243	636	323	556
119	305	1	306	0	267	39	175	131	243	63	8	298	191	115	193	113
120	364	1	345	20	331	34	189	176	260	105	40	325	220	145	181	184
121	240	1	234	7	218	23	98	143	147	94	6	235	76	165	56	185
122	286	0	282	4	267	19	166	120	238	48	1	285	202	84	191	95
123	634	0	634	0	582	52	214	420	458	176	8	626	183	451	195	439
124	70	0	70	0	70	0	48	22	70	0	3	67	46	24	49	21
125	1373	0	1341	32	1250	123	511	862	1036	337	11	1362	462	911	552	821
126	468	2	438	32	431	39	290	180	416	54	22	448	250	220	243	227
127	70	0	70	0	53	17	28	42	53	17	9	61	28	42	16	54
128	84	0	80	4	52	32	51	33	64	20	37	47	44	40	47	37
129	59	0	53	6	41	18	25	34	41	18	3	56	40	19	33	26
130	16	0	16	0	16	0	12	4	16	0	0	16	16	0	16	0
131	23	0	23	0	23	0	18	5	23	0	0	23	23	0	23	0
132	24	0	24	0	24	0	24	0	24	0	9	15	24	0	24	0
133	1	0	1	0	1	0	0	1	1	0	0	1	0	1	0	1
134	27	0	27	0	27	0	18	9	27	0	0	27	21	6	8	19
135	254	0	252	2	226	28	159	95	221	33	68	186	161	93	169	85
136	294	0	294	0	290	4	271	23	290	4	68	226	281	13	290	4
137	397	0	397	0	359	38	341	56	362	35	121	276	245	152	155	242
138	9	0	9	0	9	0	1	8	9	0	1	8	1	8	1	8
139	330	0	305	25	148	182	141	189	183	147	69	261	90	240	149	181
140	140	2	142	0	114	28	114	28	115	27	43	99	63	79	50	92
141	944	6	946	4	902	48	656	294	912	38	434	516	725	225	700	250
142	227	0	227	0	217	10	75	152	85	142	101	126	163	64	75	152
143	703	0	698	5	555	148	253	450	265	438	209	494	238	465	146	557
144	472	2	466	8	378	96	115	359	206	268	161	313	252	222	89	385
145	227	0	223	4	215	12	75	152	85	142	101	126	163	64	73	154
146	33	0	33	0	30	3	12	21	15	18	11	22	15	18	3	30
147	118	0	113	5	78	40	31	87	51	67	34	84	57	61	22	96
148	166	0	163	3	151	15	93	73	92	74	96	70	85	81	108	58
149	31	0	27	4	30	1	14	17	12	19	11	20	6	25	14	17
150	1301	0	1280	21	1198	103	407	894	565	736	580	721	457	844	754	547
151	70	0	67	3	69	1	37	33	28	42	23	47	9	61	30	40
152	385	0	368	17	203	182	134	251	286	99	91	294	117	268	147	238
153	153	0	151	2	134	19	87	66	143	10	51	102	59	94	55	98
154	234	0	228	6	228	6	143	91	168	66	81	153	111	123	108	126
155	12	0	12	0	6	6	1	11	6	6	3	9	1	11	6	6
156	174	0	174	0	139	35	62	112	124	50	70	104	97	77	53	121
157	50	0	49	1	37	13	18	32	28	22	13	37	27	23	10	40
158	178	0	178	0	147	31	63	115	123	55	67	111	94	84	57	121
159	49	0	49	0	36	13	15	34	27	22	12	37	25	24	12	37
160	502	0	496	6	340	162	289	213	465	37	120	382	198	304	209	293
161	102	0	102	0	80	22	84	18	78	24	32	70	54	48	29	73
162	33	0	33	0	31	2	31	2	33	0	9	24	31	2	9	24
163	369	0	369	0	330	39	266	103	353	16	122	247	279	90	221	148
164	525	0	501	24	486	39	196	329	445	80	77	448	232	293	247	278
165	183	0	182	1	111	72	143	40	150	33	62	121	103	80	70	113
166	111	0	111	0	111	0	64	47	111	0	26	85	71	40	20	91
167	68	0	68	0	58	10	62	6	60	8	24	44	43	25	21	47
168	33	0	33	0	31	2	31	2	33	0	9	24	31	2	9	24
169	336	0	336	0	308	28	208	128	325	11	109	227	259	77	190	146
170	85	0	83	2	83	2	34	51	76	9	18	67	41	44	55	30
171	77	0	72	5	69	8	63	14	64	13	30	47	45	32	39	38
172	247	0	247	0	205	42	151	96	209	38	72	175	186	61	175	72
173	263	0	263	0	249	14	177	86	263	0	40	223	200	63	185	78
174	61	0	61	0	58	3	55	6	61	0	28	33	35	26	28	33
175	224	0	224	0	165	59	105	119	179	45	34	190	131	93	131	93
176	53	0	53	0	43	10	16	37	53	0	7	46	35	18	32	21
177	16	0	16	0	8	8	5	11	12	4	3	13	4	12	4	12
178	84	0	80	4	50	34	39	45	62	22	36	48	44	40	47	37
179	201	0	192	9	140	61	90	111	183	18	14	187	116	85	94	107
180	220	0	220	0	180	40	147	73	220	0	10	210	102	118	121	99
181	29	0	29	0	19	10	13	16	25	4	4	25	8	21	8	21
182	83	0	78	5	49	34	55	28	62	21	36	47	43	40	47	36
183	175	0	175	0	136	39	109	66	169	6	3	172	107	68	91	84
184	27	0	27	0	14	13	10	17	27	0	0	27	16	11	5	22
185	1059	0	1055	4	939	120	762	297	992	67	344	715	812	247	740	319
186	156	0	156	0	109	47	51	105	43	113	48	108	36	120	30	126
187	663	0	656	7	506	157	255	408	241	422	199	464	95	568	133	530
188	370	1	370	1	249	122	100	271	158	213	101	270	98	273	78	293
189	166	0	160	6	120	46	57	109	52	114	25	141	39	127	32	134
190	569	0	559	10	476	93	239	330	237	332	93	476	106	463	144	425
191	295	1	290	6	197	99	69	227	130	166	40	256	81	215	64	232
192	243	0	231	12	206	37	154	89	147	96	84	159	127	116	128	115
193	49	0	48	1	48	1	29	20	17	32	10	39	17	32	17	32
194	538	0	535	3	509	29	270	268	259	279	145	393	177	361	175	363
195	62	0	60	2	61	1	34	28	17	45	16	46	17	45	17	45
196	1378	0	1353	25	1100	278	620	758	1118	260	456	922	667	711	698	680
197	361	0	359	2	342	19	268	93	350	11	99	262	261	100	239	122
198	300	0	294	6	273	27	246	54	223	77	108	192	152	148	153	147
199	935	0	914	21	784	151	505	430	841	94	194	741	444	491	518	417
200	190	0	190	0	154	36	63	127	130	60	49	141	45	145	52	138
201	100	0	99	1	83	17	21	79	73	27	3	97	4	96	56	44
202	178	0	178	0	103	75	61	117	123	55	38	140	46	132	49	129
203	71	0	70	1	53	18	42	29	53	18	36	35	32	39	41	30
204	311	0	300	11	188	123	140	171	258	53	198	113	131	180	203	108
205	218	0	190	28	154	64	120	98	190	28	118	100	60	158	60	158
206	136	0	134	2	118	18	57	79	126	10	52	84	71	65	34	102
207	136	0	136	0	104	32	34	102	34	102	52	84	34	102	34	102
208	198	0	195	3	152	46	127	71	104	94	47	151	16	182	70	128
209	2045	1109	2972	182	2634	520	1270	1884	2325	829	1125	2029	1305	1849	1501	1653
210	6555	3780	9999	336	10254	81	9632	703	10310	25	9792	543	9068	1267	9312	1023
211	302	19	319	2	306	15	297	24	309	12	265	56	295	26	289	32
212	163	0	163	0	159	4	151	12	163	0	148	15	163	0	151	12
213	10	4	11	3	14	0	2	12	13	1	2	12	0	14	1	13

1.12 Timing

The command AbsoluteTiming[] was used in Mathematica to obtain the elapsed time for each integrate call. In Maple, the command Usage was used as in the following example

cpu_time := Usage(assign ('result_of_int',int(expr,x)),output='realtime'

For all other CAS systems, the elapsed time to complete each integral was found by taking the difference between the time after the call completed from the time before the call was made. This was done using Python’s time.time() call.

All elapsed times shown are in seconds. A time limit of 3 CPU minutes was used for each integral. If the integrate command did not complete within this time limit, the integral was aborted and considered to have failed and assigned an F grade. The time used by failed integrals due to time out was not counted in the final statistics.

1.13 Verification

A verification phase was applied on the result of integration for Rubi and Mathematica.

Future version of this report will implement verification for the other CAS systems. For the integrals whose result was not run through a verification phase, it is assumed that the antiderivative was correct.

Verification phase also had 3 minutes time out. An integral whose result was not verified could still be correct, but further investigation is needed on those integrals. These integrals were marked in the summary table below and also in each integral separate section so they are easy to identify and locate.

1.14 Important notes about some of the results

Important note about Maxima results

Since tests were run in a batch mode, and using an automated script, then any integral where Maxima needed an interactive response from the user to answer a question during the evaluation of the integral will fail.

The exception raised is ValueError. Therefore Maxima results is lower than what would result if Maxima was run directly and each question was answered correctly.

The percentage of such failures were not counted for each test file, but for an example, for the Timofeev test file, there were about 14 such integrals out of total 705, or about 2 percent. This percentage can be higher or lower depending on the specific input test file.

Such integrals can be identified by looking at the output of the integration in each section for Maxima. The exception message will indicate the cause of error.

Maxima integrate was run using SageMath with the following settings set by default

'besselexpand : true' 
'display2d : false' 
'domain : complex' 
'keepfloat : true' 
'load(to_poly_solve)' 
'load(simplify_sum)' 
'load(abs_integrate)' 'load(diag)'
 

SageMath automatic loading of Maxima abs_integrate was found to cause some problems. So the following code was added to disable this effect.

 from sage.interfaces.maxima_lib import maxima_lib 
 maxima_lib.set('extra_definite_integration_methods', '[]') 
 maxima_lib.set('extra_integration_methods', '[]')
 

See https://ask.sagemath.org/question/43088/integrate-results-that-are-different-from-using-maxima/ for reference.

Important note about FriCAS result

There were few integrals which failed due to SageMath interface and not because FriCAS system could not do the integration.

These will fail With error Exception raised: NotImplementedError.

The number of such cases seems to be very small. About 1 or 2 percent of all integrals. These can be identified by looking at the exception message given in the result.

Important note about finding leaf size of antiderivative

For Mathematica, Rubi, and Maple, the builtin system function LeafSize was used to find the leaf size of each antiderivative.

The other CAS systems (SageMath and Sympy) do not have special builtin function for this purpose at this time. Therefore the leaf size for Fricas and Sympy antiderivative was determined using the following function, thanks to user slelievre at https://ask.sagemath.org/question/57123/could-we-have-a-leaf_count-function-in-base-sagemath/

def tree_size(expr): 
    r""" 
    Return the tree size of this expression. 
    """ 
    if expr not in SR: 
        # deal with lists, tuples, vectors 
        return 1 + sum(tree_size(a) for a in expr) 
    expr = SR(expr) 
    x, aa = expr.operator(), expr.operands() 
    if x is None: 
        return 1 
    else: 
        return 1 + sum(tree_size(a) for a in aa)
 

For Sympy, which was called directly from Python, the following code was used to obtain the leafsize of its result

try: 
  # 1.7 is a fudge factor since it is low side from actual leaf count 
  leafCount = round(1.7*count_ops(anti)) 
 
  except Exception as ee: 
         leafCount =1
 

Important note about Mupad results

Matlab’s symbolic toolbox does not have a leaf count function to measure the size of the antiderivative. Maple was used to determine the leaf size of Mupad output by post processing Mupad result.

Currently no grading of the antiderivative for Mupad is implemented. If it can integrate the problem, it was assigned a B grade automatically as a placeholder. In the future, when grading function is implemented for Mupad, the tests will be rerun again.

The following is an example of using Matlab’s symbolic toolbox (Mupad) to solve an integral

integrand = evalin(symengine,'cos(x)*sin(x)') 
the_variable = evalin(symengine,'x') 
anti = int(integrand,the_variable)
 

Which gives sin(x)^2/2

1.15 Design of the test system

The following diagram gives a high level view of the current test build system.